JP7786902B2 - Ophthalmic device, method for controlling an ophthalmic device, program, and recording medium - Google Patents

Description

The present invention relates to an ophthalmic apparatus, a method for controlling an ophthalmic apparatus , a program , and a recording medium.

Diagnostic imaging plays an important role in the field of ophthalmology. Various types of ophthalmic devices are used for diagnostic imaging. These devices include slit lamp microscopes, fundus cameras, scanning laser ophthalmoscopes (SLOs), and optical coherence tomography (OCT). Additionally, various examination and measurement devices, such as refractometers, keratometers, tonometers, specular microscopes, wavefront analyzers, and microperimeters, are also equipped with the ability to photograph the anterior segment and fundus.

Among these various ophthalmic devices, one of the most widely and frequently used is the slit lamp microscope, also known as the stethoscope for ophthalmologists. A slit lamp microscope is an ophthalmic device that illuminates the subject's eye with a slit of light and observes and photographs the illuminated cross-section from the side using a microscope (see, for example, Patent Documents 1 and 2). Also known is a slit lamp microscope that can quickly scan a three-dimensional area of the subject's eye by using an optical system configured to satisfy the Scheimpflug condition (see, for example, Patent Document 3). In addition to slit lamp microscopes, there are also imaging methods such as rolling shutter cameras that scan an object with a slit of light.

One example of an ophthalmic device that uses slit light for measurements is a flare cell meter for evaluating the inflammatory state of the subject's eye (see, for example, Patent Documents 4 and 5). A flare cell meter is an ophthalmic device that measures the number of inflammatory cells floating in the anterior chamber and the protein concentration (flare concentration) within the anterior chamber. For example, the slit light is generated by one-dimensionally scanning a laser beam or by restricting LED light with a slit aperture.

Japanese Patent Application Laid-Open No. 2016-159073 Japanese Patent Application Laid-Open No. 2016-179004 Japanese Patent Application Laid-Open No. 2019-213733 Japanese Patent Application Laid-Open No. 2005-102938 International Publication No. 2018/003906

One object of the present invention is to automate the assessment of inflammatory conditions based on eye images.

An ophthalmologic apparatus according to an exemplary embodiment includes an image acquisition unit and a data processing unit. The image acquisition unit acquires a Scheimpflug image of the subject's eye. The data processing unit performs processing to generate inflammation state information indicating the inflammatory state of the subject's eye from the Scheimpflug image acquired by the image acquisition unit.

According to exemplary embodiments, it is possible to automate the assessment of inflammatory conditions based on eye images.

1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment.

Some exemplary aspects of the embodiments will be described in detail with reference to the drawings. It should be noted that any known technology can be combined with the embodiments. For example, any known technology in the relevant technical field, such as any matter disclosed in the documents cited herein, can be combined with any of the embodiments. In particular, the entire disclosure of Patent Document 3 (JP 2019-213733 A) is incorporated by reference into this disclosure. Similarly, any technical matter disclosed by the applicant of the present application regarding technology related to this disclosure (matters disclosed in patent applications, papers, etc.) can be combined with any of the embodiments. It is also possible to at least partially combine any two or more of the various aspects of the present disclosure.

At least a portion of the functionality of the elements described in this disclosure is implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general-purpose processor, a special-purpose processor, an integrated circuit, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), a Field Programmable Gate Array (FPGA)), or a combination of these devices configured and/or programmed to perform at least some of the disclosed functions. Array), conventional circuitry, and any combination thereof. A processor is considered to be processing circuitry or circuitry, including transistors and/or other circuitry. In this disclosure, circuitry, unit, means, or similar terms refers to hardware that performs at least some of the disclosed functions or that is programmed to perform at least some of the disclosed functions. The hardware may be the hardware disclosed herein or known hardware that is programmed and/or configured to perform at least some of the described functions. In the case of a processor, where the hardware can be considered to be a type of circuitry, circuitry, unit, means, or similar terms refers to a combination of hardware and software, and the software is used to configure the hardware and/or processor.

<Overview of the embodiment>
The embodiment relates to a technology for generating information indicating the inflammatory state of a test eye (referred to as inflammatory state information) from a digital image (referred to as a Scheimpflug image) generated by photographing the test eye using an optical system that satisfies the Scheimpflug condition.

The number of Scheimpflug images processed by an embodiment may be any number; some embodiments process one Scheimpflug image, and some embodiments process multiple images. These multiple images are collected, for example, by scanning with a slit of light (referred to as slit scanning). Slit scanning is an ophthalmic imaging technique developed by the applicant of the present application that collects a series of images by scanning a three-dimensional region of the subject's eye with a slit of light, and is described in Patent Document 3 (JP 2019-213733 A), among others.

The Scheimpflug image processed by some embodiments may be an image created by processing a Scheimpflug image. Examples of images created by processing a Scheimpflug image include an image created by applying any digital image processing such as correction, editing, or enhancement to a Scheimpflug image, a three-dimensional image (volume image) constructed from multiple Scheimpflug images, or an image created by applying any rendering to a three-dimensional image.

The inflammatory state information generated by the embodiments may be any information related to the inflammatory state of the subject's eye. The inflammatory state information of the embodiments may include one or more of the following: information on inflammatory cells present in the anterior chamber (anterior chamber cells), information on proteins present in the anterior chamber (anterior chamber flare), information on lens opacity, information on the onset and progression of disease, and information on disease activity. Some embodiments may generate comprehensive information (e.g., comprehensive evaluation information) based on two or more pieces of information.

Information about inflammatory cells includes information on arbitrary parameters such as the density (concentration), number, position, and distribution of inflammatory cells, as well as evaluation information based on information on specified parameters. Evaluation information about inflammatory cells is called cell evaluation information. Information about anterior chamber flare includes information on arbitrary parameters such as flare concentration, number, position, and distribution, as well as evaluation information based on information on specified parameters. Information about lens opacity includes information on arbitrary parameters such as opacity concentration, number, position, and distribution, as well as evaluation information based on information on specified parameters. Information about disease onset and progression includes information on arbitrary parameters such as the presence or absence of onset, the state of onset, duration of illness, and progress, as well as evaluation information based on information on specified parameters. Information about disease activity includes information on arbitrary parameters such as the state of disease activity, as well as evaluation information based on information on specified parameters.

An exemplary embodiment of generating evaluation information regarding an inflammatory state may be configured to generate the evaluation information based not only on information generated by an ophthalmic device that performs the process of generating the evaluation information, but also on information input to the ophthalmic device from outside (e.g., information acquired by another ophthalmic device, information input by a doctor).

The information referenced to generate the above-described inflammatory status information may be any information, and may include, for example, the classification criteria for uveitis diseases proposed by the Standardization of Uveitis Nomenclature (SUN) Working Group ("Standardization of Uveitis Nomenclature for Reporting Clinical Data. Results of the First International Workshop," American Journal of Ophthalmology, Volume 140, Issue 3, September 2005, Page 14). 509-516).

Note that the inflammation state information of the embodiment is not limited to the above example. Furthermore, the reference information used to generate the inflammation state information of the embodiment is not limited to the above example.

The embodiments also describe exemplary aspects in which slit scanning can be applied to the anterior segment of the subject's eye. The portion of the subject's eye to which slit scanning is applied includes at least a portion or the entire portion to which slit scanning is applied to acquire data used to generate inflammation state information. For example, when generating inflammation state information including information on inflammatory cells and/or information on anterior chamber flare, slit scanning is applied to an area including at least a portion of the anterior chamber. Furthermore, when generating inflammation state information including information on lens opacity, slit scanning is applied to an area including at least a portion of the lens. The same applies when two or more Scheimpflug images are acquired using a method other than slit scanning, or when only one Scheimpflug image is acquired.

Generally, the region (site) of the subject's eye to which the slit scan is applied includes at least a portion of the anterior segment (e.g., tissues such as the cornea, iris, anterior chamber, angle, ciliary body, zonules, lens, nerves, and blood vessels; lesions; treatment scars; and artificial structures such as intraocular lenses and minimally invasive glaucoma surgery (MIGS) devices) and/or at least a portion of the posterior segment (e.g., tissues such as the vitreous body, retina, choroid, sclera, optic disc, lamina cribrosa, macula, nerves, and blood vessels; lesions; treatment scars; and artificial structures such as retinal prostheses). In some exemplary embodiments, the slit scan may be applied to at least a portion of tissues near the eye, such as the eyelids and meibomian glands. In some exemplary embodiments, the slit scan may be applied to a three-dimensional region including any two or all of at least a portion of the anterior segment, at least a portion of the posterior segment, and at least a portion of tissues near the eye.

An embodiment capable of generating inflammation status information including the above-described cell evaluation information may be configured to perform at least one of the following three processes: (1) segmentation (referred to as first segmentation or anterior chamber segmentation) to identify an image region corresponding to the anterior chamber of the subject's eye (referred to as an anterior chamber region); (2) segmentation (referred to as second segmentation or cell segmentation) to identify an image region corresponding to inflammatory cells (referred to as a cell region); (3) processing to generate cell evaluation information (referred to as cell evaluation information generation processing). Some exemplary aspects of such an embodiment are described below.

A first exemplary embodiment is configured to perform at least one of the following processes: a first segmentation for identifying an anterior chamber region from a Scheimpflug image; a second segmentation for identifying a cellular region from the anterior chamber region identified by the first segmentation; and a cell evaluation information generation process for generating cell evaluation information from the cellular region identified by the second segmentation. The first segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The second segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

A second exemplary embodiment is configured to perform at least one of a second segmentation for identifying a cellular region from a Scheimpflug image and a cell evaluation information generation process for generating cell evaluation information from the cellular region identified by the second segmentation, without performing a first segmentation for identifying the anterior chamber region. The second segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

A third exemplary embodiment is configured to perform a cell evaluation information generation process for generating cell evaluation information from a Scheimpflug image without performing a first segmentation for identifying the anterior chamber region and a second segmentation for identifying the cellular region. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

A fourth exemplary embodiment is configured to perform at least one of a first segmentation for identifying an anterior chamber region from a Scheimpflug image and a cell evaluation information generation process for generating cell evaluation information from the anterior chamber region identified by the first segmentation, without performing a second segmentation for identifying a cellular region. The first segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

A fifth exemplary embodiment is implemented without using a neural network trained by machine learning. This embodiment is configured to perform a first segmentation for analyzing a Scheimpflug image to identify an anterior chamber region, a second segmentation for analyzing the anterior chamber region identified by the first segmentation to identify a cellular region, and a cell evaluation information generation process for generating cell evaluation information based on the cellular region identified by the second segmentation. Details of this embodiment will be described later.

The sixth exemplary aspect is configured to execute at least one or more of at least a portion of the first segmentation, at least a portion of the second segmentation, and at least a portion of the cell evaluation information generation process using a machine learning-based configuration, while executing processes other than those executed using the machine learning-based configuration using a non-machine learning-based configuration. This aspect is realized, for example, by partially combining any of the first to fourth exemplary aspects described above with the fifth exemplary aspect, and therefore a detailed description thereof will be omitted.

<Ophthalmological equipment>
An exemplary embodiment of the ophthalmic apparatus according to the present invention will be described below.

Figure 1 shows the configuration of an ophthalmic device according to this embodiment. The ophthalmic device 1000 includes an image acquisition unit 1010 and a data processing unit 1020.

The image acquisition unit 1010 is configured to acquire a Scheimpflug image of the subject's eye. In some exemplary embodiments, the image acquisition unit 1010 is configured to photograph the subject's eye and acquire a Scheimpflug image. An example configuration of such an image acquisition unit 1010 is shown in FIG. 2A.

The image acquisition unit 1010A shown in FIG. 2A includes an illumination system 1011 and an imaging system 1012. The illumination system 1011 is configured to project a slit of light onto the subject's eye. The imaging system 1012 is configured to capture an image of the subject's eye, and includes an image sensor 1013 and an optical system (not shown) that guides light from the subject's eye to the image sensor 1013.

The illumination system 1011 and the imaging system 1012 are configured to satisfy the Scheimpflug condition and function as a Scheimpflug camera. More specifically, the illumination system 1011 and the imaging system 1012 are configured so that a plane passing through the optical axis of the illumination system 1011 (a plane including the object plane), the principal plane of the imaging system 1012, and the imaging plane of the image sensor 1013 intersect on the same straight line. This allows imaging to be performed with the imaging system 1012 focused on all positions within the object plane (all positions in the direction along the optical axis of the illumination system 1011).

In some exemplary embodiments, the image acquisition unit 1010A shown in FIG. 2A is configured to collect a series of Scheimpflug images by scanning a three-dimensional region of the subject's eye with a slit of light. In this example, the image acquisition unit 1010A is configured to collect a series of Scheimpflug images by repeatedly capturing images while moving the projection position of the slit of light relative to the three-dimensional region of the subject's eye.

In some exemplary embodiments, the image acquisition unit 1010A may be configured to scan a three-dimensional region of the subject's eye by translating the slit light in a direction perpendicular to the longitudinal direction of the slit light. This scanning mode differs from that of conventional anterior segment imaging devices, which scan the anterior segment by rotating the slit light.

Here, the longitudinal direction of the slit light is the longitudinal direction of the beam cross section of the slit light at the projection position on the subject's eye, in other words, the longitudinal direction of the image of the slit light formed on the subject's eye, and may approximately coincide with the direction along the subject's body axis (body axis direction). Furthermore, the dimension of the slit light in the longitudinal direction may be equal to or greater than the corneal diameter in the subject's body axis direction, and the distance of translation of the slit light may be equal to or greater than the corneal diameter in a direction perpendicular to the subject's body axis direction.

The series of Scheimpflug images collected by the image acquisition unit 1010A in this example is a group of images (group of frames) collected continuously in time, but because they are a group of images collected sequentially from multiple different positions in the three-dimensional region of the subject's eye, they are a group of images that are spatially distributed, unlike general moving images.

In the image acquisition unit 1010A of this example, the illumination system 1011 projects slit light onto a three-dimensional area of the subject's eye, and the imaging system 1012 images the three-dimensional area of the subject's eye onto which the slit light from the illumination system 1011 is projected. The image acquisition unit 1010A of this example further includes a mechanism for moving the illumination system 1011 and the imaging system 1012.

The data processing unit 1020 of the ophthalmic apparatus 1000 to which the image acquisition unit 1010A of this example is applied may be configured to generate inflammation state information from a Scheimpflug image included in the series of Scheimpflug images collected by the image acquisition unit 1010A. The data processing unit 1020 of this example generates inflammation state information by processing one or more Scheimpflug images included in the series of Scheimpflug images collected by the image acquisition unit 1010A. Here, the number of Scheimpflug images used to generate the inflammation state information may be any number.

Furthermore, the data processing unit 1020 of the ophthalmic device 1000 to which the image acquisition unit 1010A of this example is applied may be configured to execute a process of processing a series of Scheimpflug images collected by the image acquisition unit 1010A to generate processed image data, and a process of generating inflammation status information based on the generated processed image data.

For example, the data processing unit 1020 in this example may be configured to execute a process of constructing a three-dimensional image (an example of processed image data) from multiple Scheimpflug images included in a series of Scheimpflug images, and a process of generating inflammation status information based on this three-dimensional image. Alternatively, the data processing unit 1020 in this example may be configured to execute a process of constructing a three-dimensional image from multiple Scheimpflug images included in a series of Scheimpflug images, a process of generating a rendering image (an example of processed image data) from this three-dimensional image, and a process of generating inflammation status information based on this rendering image.

In some exemplary embodiments, the imaging system 1012 of the image acquisition unit 1010A may include two or more imaging systems. For example, the imaging system 1012A of the image acquisition unit 1010B shown in FIG. 2B includes a first imaging system 1014 and a second imaging system 1015 that capture images from different directions.

In this example, the image acquisition unit 1010B may be configured to capture images of a three-dimensional region of the subject's eye from different directions using the first imaging system 1014 and the second imaging system 1015, respectively, during a slit scan to collect a series of Scheimpflug images. For example, if the image acquisition unit 1010B is configured so that the longitudinal direction of the beam cross section of the slit light at the incident position on the subject's eye is the vertical direction (Y direction) and the direction of movement of the slit light is the horizontal direction (left-right direction, X direction), the first imaging system 1014 and the second imaging system 1015 may be arranged so that one captures an image of the subject's eye from an oblique left direction and the other captures an image of the subject's eye from an oblique right direction.

The series of Scheimpflug images collected by the first imaging system 1014 is called the first Scheimpflug image group, and the series of Scheimpflug images collected by the second imaging system 1015 is called the second Scheimpflug image group. Such a series of Scheimpflug images collected by the image acquisition unit 1010B includes a first Scheimpflug image group and a second Scheimpflug image group.

Note that even when one Scheimpflug image (first Scheimpflug image) is acquired by the first imaging system 1014 and one Scheimpflug image (second Scheimpflug image) is acquired by the second imaging system 1015 without slit scanning, for convenience of terminology, the one Scheimpflug image acquired by the first imaging system 1014 may be referred to as the first Scheimpflug image group, and the one Scheimpflug image acquired by the second imaging system 1015 may be referred to as the second Scheimpflug image group. Thus, in this disclosure, the term "group" may be used not only when multiple elements are included, but also when only one element is included.

When the image acquisition unit 1010B in this example performs a slit scan, the first imaging system 1014 and the second imaging system 1015 capture images of the subject's eye in parallel. That is, the image acquisition unit 1010B performs imaging in parallel with the first imaging system 1014 and the second imaging system 1015 while moving the projection position of the slit light relative to the three-dimensional region of the subject's eye.

Furthermore, the image acquisition unit 1010B in this example may be configured to synchronize the image capture by the first imaging system 1014 (collection of Scheimpflug images) with the image capture by the second imaging system 1015 (collection of Scheimpflug images). By referencing this synchronization relationship, the first Scheimpflug image group and the second Scheimpflug image group can be easily associated without using image processing or the like. This association is performed, for example, to associate Scheimpflug images that have a small difference in acquisition time with each other.

In this manner, the ophthalmic device 1000 can reconstruct a series of Scheimpflug images corresponding to the slit scan from the first Scheimpflug image group and the second Scheimpflug image group by referring to the mutual synchronization relationship between the image capture by the first image capture system 1014 and the image capture by the second image capture system 1015.

Figure 2C shows an example configuration of an ophthalmic device 1000 to which the image acquisition unit 1010B shown in Figure 2B is applied. In the image acquisition unit 1010B of this example, the optical axis of the first imaging system 1014 and the optical axis of the second imaging system 1015 are arranged so as to be tilted in opposite directions relative to the optical axis of the illumination system 1011. Furthermore, the data processing unit 1020A of this example includes an image selection unit 1021 and an inflammation state information generation unit 1022.

The image selection unit 1021 is configured to select an image from the first Scheimpflug image group acquired by the first imaging system 1014 and the second Scheimpflug image group acquired by the second imaging system 1015. For example, the image selection unit 1021 may be configured to select one of the first Scheimpflug image acquired by the first imaging system 1014 and the second Scheimpflug image acquired by the second imaging system 1015.

The image selection unit 1021 is configured to select a new series of Scheimpflug images from the first Scheimpflug image group and the second Scheimpflug image group, corresponding to the slit scans that collected the first Scheimpflug image group and the second Scheimpflug image group, based on the correspondence between the first Scheimpflug image group and the second Scheimpflug image group, respectively, collected based on synchronization between image capture by the first imaging system 1014 and image capture by the second imaging system 1015. In short, the image selection unit 1021 is configured to reconstruct a series of Scheimpflug images from the first Scheimpflug image group and the second Scheimpflug image group collected by the first imaging system 1014 and the second imaging system 1015, respectively.

The image selection processing method performed by the image selection unit 1021 may be any method. For example, the image selection processing method may be determined and/or selected based on specified conditions or parameters, such as the configuration and/or arrangement of the first imaging system 1014 and the second imaging system 1015, or the purpose and/or use of image selection.

The image acquisition unit 1010B performs imaging using the first imaging system 1014 and imaging using the second imaging system 1015 in synchronization with each other. Furthermore, as described above, the optical axis of the first imaging system 1014 and the optical axis of the second imaging system 1015 are arranged so as to be tilted in opposite directions relative to the optical axis of the illumination system 1011. For example, the optical axis of the first imaging system 1014 is arranged so as to be tilted to the left relative to the optical axis of the illumination system 1011, and the optical axis of the second imaging system 1015 is arranged so as to be tilted to the right relative to the optical axis of the illumination system 1011. The first imaging system 1014 and second imaging system 1015 arranged in this manner may be referred to as the left imaging system and the right imaging system, respectively.

The tilt angle of the optical axis of the first imaging system 1014 relative to the optical axis of the illumination system 1011 and the tilt angle of the optical axis of the second imaging system 1015 relative to the optical axis of the illumination system 1011 may be equal to or different from each other. Furthermore, these tilt angles may be fixed or variable.

The illumination system 1011 in this example is constructed and arranged to project slit light, the longitudinal direction of the cross section of which is oriented in the Y direction, from the front onto the subject's eye. The image acquisition unit 1010B in this example also applies slit scanning to a three-dimensional region of the anterior segment of the subject's eye by integrally moving the illumination system 1011, the first imaging system 1014, and the second imaging system 1015 in the X direction.

The image selection unit 1021 in this example selects a new series of Scheimpflug images corresponding to the slit scan from which the first Scheimpflug image group and the second Scheimpflug image group were collected by selecting multiple Scheimpflug image groups that do not contain artifacts based on the correspondence between the first Scheimpflug image group collected by the first imaging system 1014 and the second Scheimpflug image group collected by the second imaging system 1015. This artifact may be any type of artifact. When performing an anterior segment scan as in this example, this artifact may be an artifact caused by corneal reflection (referred to as a corneal reflection artifact). Below, several examples of the processing performed by the image selection unit 1021 are described.

The projection position (Scheimpflug image, frame) of the slit light at which the corneal reflection artifact occurs differs between the left and right imaging systems. For example, as in this example, when slit scanning is performed by projecting slit light whose cross-section longitudinal direction is oriented in the Y direction onto the subject's eye from the front while moving the illumination system 1011, first imaging system 1014, and second imaging system 1015 together in the X direction, the corneal reflection light of the slit light is likely to enter the left imaging system when the slit light is projected to a position to the left of the corneal apex, and is likely to enter the right imaging system when the slit light is projected to a position to the right of the corneal apex.

Taking these circumstances into consideration, the image selection unit 1021 in some exemplary embodiments is configured to first identify a Scheimpflug image (first corneal apex image) corresponding to the corneal apex from the first Scheimpflug image group collected by the first imaging system 1014 as the left imaging system, and to identify a Scheimpflug image (second corneal apex image) corresponding to the corneal apex from the second Scheimpflug image group collected by the second imaging system 1015 as the right imaging system.

In some exemplary embodiments, the process of identifying the corneal apex image may include a process of detecting an image corresponding to the corneal surface from each Scheimpflug image included in the first Scheimpflug image group, a process of identifying the pixel closest to the ophthalmic device 1000 of this example based on the Z coordinates of the pixels of these detected images, and a process of setting the Scheimpflug image including the identified pixel as the first corneal apex image. Setting the second corneal apex image may be performed in the same manner.

Next, the image selection unit 1021 selects from the first Scheimpflug image group those Scheimpflug images located to the right of the first corneal apex image, and from the second Scheimpflug image group those Scheimpflug images located to the left of the second corneal apex image, thereby forming a series of Scheimpflug images consisting of the two selected Scheimpflug image groups (as well as the first corneal apex image and/or the second corneal apex image). This results in a series of Scheimpflug images that cover the three-dimensional region of the anterior segment to which the slit scan has been applied and that are (highly likely to) not contain corneal reflection artifacts.

Another example of the process for identifying a corneal apex image will now be described. In some exemplary embodiments, the image selection unit 1021 is configured to determine whether a corneal reflection artifact is included in either of two images acquired substantially simultaneously by the first imaging system 1014 (e.g., the left imaging system) and the second imaging system 1015 (e.g., the right imaging system). This corneal reflection artifact determination process includes a predetermined image analysis, such as threshold processing on brightness information assigned to pixels. Note that the process for determining whether two images were acquired substantially simultaneously can be performed based on the synchronization between the imaging by the first imaging system 1014 and the imaging by the second imaging system 1015.

The threshold processing used in the artifact determination process is performed, for example, to identify pixels assigned a brightness value exceeding a preset threshold. Typically, the threshold may be set higher than the brightness value of the slit light image (the slit light projection area) in the image. This allows the image selection unit 1021 to be configured to determine an image brighter than the slit light image as an artifact, without determining the slit light image as an artifact. Considering that an image brighter than the slit light image in a Scheimpflug image is likely to be an image caused by specular reflection from the cornea, it can be considered that artifacts detected by the image selection unit 1021 configured in this manner are likely to be corneal reflection artifacts.

The image selection unit 1021 may perform any image analysis other than threshold processing, such as pattern recognition, segmentation, or edge detection, to determine artifacts. In general, any information processing technology, such as image analysis, image processing, machine learning, artificial intelligence, or cognitive computing, can be applied to artifact determination.

When the artifact determination results in one of two images acquired substantially simultaneously by the first imaging system 1014 and the second imaging system 1015 being determined to contain an artifact, the image selection unit 1021 selects the other image. In other words, of the two images acquired substantially simultaneously by the first imaging system 1014 and the second imaging system 1015, the image selection unit 1021 selects the other image that is not the image determined to contain an artifact.

Assuming that both images are determined to contain artifacts, the image selection unit 1021 may be configured to, for example, perform a process of evaluating the magnitude of the adverse effect that the artifacts have on observation and diagnosis, and a process of selecting the image with the smaller adverse effect. This evaluation process may be performed based on, for example, one or more conditions of the artifact's size, intensity, shape, and position. Typically, artifacts with large size, high intensity, and artifacts located in or near a region of interest such as a slit light image are evaluated as having a large adverse effect.

Note that in cases where both images contain artifacts, the artifact removal technique disclosed in Patent Document 3 (JP 2019-213733 A) may be applied.

By providing the image selection unit 1021 as described above, it becomes possible to provide an image of a three-dimensional region of the subject's eye that does not contain artifacts that interfere with observation, analysis, or diagnosis. Furthermore, it becomes possible to provide an image of a three-dimensional region of the subject's eye that does not contain artifacts to subsequent processing. For example, it becomes possible to construct a three-dimensional image or a rendering image of the subject's eye based on a group of images that do not contain artifacts.

Even when images are obtained by photographing substantially the same position, the dimensions of the depicted specific area may differ between the image obtained by the left and right imaging systems. For example, the depicted corneal thickness, dimensions of inflammatory cells, and dimensions of anterior chamber flare may differ between left and right images obtained by photographing substantially the same position with the left and right imaging systems, respectively. Even in such cases, it is possible to align the dimensions of the specified area by using the image selection unit 1021.

The inflammation state information generation unit 1022 is configured to generate inflammation state information based on the Scheimpflug images selected by the selection unit 1021. The number of Scheimpflug images used to generate the inflammation state information may be set arbitrarily. The inflammation state information generation unit 1022 may also be configured to execute a process of processing one or more Scheimpflug images included in the series of Scheimpflug images selected by the selection unit 1021 to generate processed image data, and a process of generating inflammation state information based on the generated processed image data.

As described above, the inflammation state information generated by the data processing unit 1020 may include cell evaluation information, which is evaluation information regarding inflammatory cells in the anterior chamber of the subject's eye. In this case, the data processing unit 1020 may be configured to execute at least one of first segmentation, second segmentation, and cell evaluation information generation processing. Here, the first segmentation is processing for identifying an anterior chamber region corresponding to the anterior chamber, the second segmentation is processing for identifying a cellular region corresponding to inflammatory cells, and the cell evaluation information generation processing is processing for generating cell evaluation information.

The first segmentation may be a machine learning-based process or a non-machine learning-based process, or may be a combination of machine learning-based and non-machine learning-based processes. The second segmentation may be a machine learning-based process or a non-machine learning-based process, or may be a combination of machine learning-based and non-machine learning-based processes. The cell evaluation information generation process may be a machine learning-based process or a non-machine learning-based process, or may be a combination of machine learning-based and non-machine learning-based processes.

The types of data input into the first segmentation, the second segmentation, and the cell evaluation information generation process may all be arbitrary. Below, we will explain some examples of possible combinations of multiple processes, including the first segmentation, the second segmentation, and the cell evaluation information generation process.

The data processing unit 1030 shown in Figure 3 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1030 in this example includes a first segmentation unit 1031, a second segmentation unit 1032, and a cell evaluation information generation processing unit 1033.

The first segmentation unit 1031 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

Figure 4 shows an example configuration of the first segmentation unit 1031 when performing the first segmentation of this example using machine learning. The first segmentation unit 1031A of this example is configured to perform the first segmentation using a pre-constructed inference model 1034 (referred to as the first inference model). The first inference model 1034 includes a neural network 1035 (referred to as the first neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

The data input to the first neural network 1035 is a Scheimpflug image, and the data output from the first neural network 1035 is the anterior chamber region. That is, the first segmentation unit 1031A is configured to receive a Scheimpflug image (e.g., one or more Scheimpflug images, one or more processed image data, or one or more Scheimpflug images and one or more processed image data) acquired by the image acquisition unit 1010, input the Scheimpflug image to the first neural network 1035 of the first inference model 1034, and acquire output data from the first neural network 1035 (the anterior chamber region in the input Scheimpflug image).

The device (inference model construction device) that constructs the first inference model 1034 may be provided in the ophthalmic device 1000, in a peripheral device (such as a computer) of the ophthalmic device 1000, or may be another computer.

The model construction unit 2000 shown in Figure 5 is an example of an inference model construction device and includes a learning processing unit 2010 and a neural network 2020.

The neural network 2020 typically includes a convolutional neural network (CNN). Reference numeral 2030 in Figure 5 indicates an example of the structure of a convolutional neural network.

An image is input to the input layer. Behind the input layer, multiple pairs of convolutional layers and pooling layers are arranged. In the example shown in Figure 5, there are three pairs of convolutional layers and pooling layers, but the number of pairs can be any number.

The convolutional layer performs convolution operations to extract features (such as contours) from an image. A convolution operation is a multiplication and accumulation operation of a filter function (weighting coefficients, filter kernel) of the same dimension as the input image on the input image. The convolutional layer applies the convolution operation to multiple portions of the input image. More specifically, the convolutional layer multiplies the value of each pixel in the partial image to which the filter function has been applied by the value (weight) of the filter function corresponding to that pixel to calculate the product, and then calculates the sum of the products across the multiple pixels in this partial image. This resulting sum-of-products value is assigned to the corresponding pixel in the output image. By performing the multiplication and accumulation operation while shifting the location (partial image) to which the filter function is applied, the convolution operation result for the entire input image is obtained. This convolution operation uses multiple weighting coefficients to generate multiple images in which various features have been extracted. In other words, it generates multiple filtered images, such as smoothed images and edge images. The multiple images generated by the convolutional layer are called feature maps.

The pooling layer compresses (e.g., thins out) the feature map generated by the previous convolutional layer. More specifically, the pooling layer calculates statistical values for a specified number of neighboring pixels of a pixel of interest in the feature map at specified pixel intervals, and outputs an image with smaller dimensions than the input feature map. The statistical value applied to the pooling operation is, for example, the maximum value (max pooling) or the average value (average pooling). The pixel interval applied to the pooling operation is called the stride.

Convolutional neural networks can extract many features from input images by processing them using multiple pairs of convolutional and pooling layers.

A fully connected layer is provided after the final pair of convolutional and pooling layers. In the example shown in Figure 5, two fully connected layers are provided, but any number of fully connected layers may be used. In the fully connected layer, features compressed by a combination of convolution and pooling are used to perform processes such as image classification, image segmentation, and regression. An output layer is provided after the final fully connected layer to provide output results.

In some exemplary embodiments, the convolutional neural network may not include a fully connected layer (e.g., a fully convolutional network (FCN)), or may include a support vector machine, a recurrent neural network (RNN), or the like. Furthermore, the machine learning performed on the neural network 2020 may include transfer learning. That is, the neural network 2020 may include a neural network that has already been trained using other training data (training images) and whose parameters have been adjusted. Furthermore, the model construction unit 2000 (learning processing unit 2010) may be configured to be able to apply fine tuning to the trained neural network (neural network 2020). The neural network 2020 may be constructed using a known open-source neural network architecture.

The learning processing unit 2010 applies machine learning using training data to the neural network 2020. If the neural network 2020 includes a convolutional neural network, the parameters adjusted by the learning processing unit 2010 include, for example, the filter coefficients of the convolutional layer and the connection weights and offsets of the fully connected layer.

As described above, the training data may include one or more Scheimpflug images obtained for one or more eyes. Because the Scheimpflug images of the eyes are the same type of images as the images input to the first neural network 1035, the quality (accuracy, precision, etc.) of the output of the first neural network 1035 can be improved compared to when machine learning is performed using training data that includes only other types of images.

The types of images included in the training data are not limited to Scheimpflug images. For example, the training data may include images acquired by other ophthalmic modalities (fundus cameras, OCT devices, SLOs, surgical microscopes, etc.), images acquired by any medical department's imaging diagnostic modalities (ultrasound diagnostic devices, X-ray diagnostic devices, X-ray CT devices, magnetic resonance imaging (MRI) devices, etc.), images generated by processing actual eye images (processed image data), and pseudo-images. Furthermore, the number of images included in the training data may be increased using techniques such as data expansion and data augmentation.

The training method (machine learning method) for constructing the first neural network 1035 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

In some exemplary embodiments, supervised learning is performed using training data generated by annotation that assigns labels to input images. For example, this annotation involves identifying and labeling the anterior chamber region in each image included in the training data. The anterior chamber region is identified by, for example, at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the first neural network 1035 by applying supervised learning using such training data to the neural network 2020.

The first inference model 1034, which includes the first neural network 1035 constructed in this manner, is a trained model that receives as input a Scheimpflug image (e.g., a Scheimpflug image acquired by the image acquisition unit 1010, or its processed image data) and outputs the anterior chamber region in the input Scheimpflug image (e.g., information indicating the range or position of the anterior chamber region).

To avoid concentrating processing on specific units of the first neural network 1035, the learning processing unit 2010 may randomly select and disable some units of the neural network 2020 and perform learning using the remaining units (dropout).

The techniques used to build inference models are not limited to the examples shown here. For example, any technique, such as support vector machines, Bayesian classifiers, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing maps, random forests, or generative adversarial networks (GANs), can be used to build inference models.

In this example, the first segmentation unit 1031A uses this first inference model 1034 (first neural network 1035) to perform processing to identify the anterior chamber region from the Scheimpflug image of the subject's eye.

The second segmentation unit 1032 includes a processor that performs second segmentation to identify cellular regions, and is configured to identify cellular regions from the anterior chamber region identified by the first segmentation unit 1031.

Figure 6 shows an example configuration of the second segmentation unit 1032 when the second segmentation of this example is performed using machine learning. The second segmentation unit 1032A of this example is configured to perform the second segmentation using a pre-constructed inference model 1036 (referred to as the second inference model). The second inference model 1036 includes a neural network 1037 (referred to as the second neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with another modality, or Scheimpflug images of the eye and eye images acquired with another modality).

The eye images included in the training data in this example include images corresponding to at least a portion of the anterior chamber of the eye (referred to as an anterior chamber image). The eye images included in the training data in this example may include the results of manual or automatic segmentation of an anterior segment image (e.g., a Scheimpflug image, or an image acquired using another modality), and may be, for example, an anterior chamber image extracted from an anterior segment image, or information indicating the range or position of the anterior chamber image within the anterior segment image.

The data input to the second neural network 1037 is the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1031 (or the identified and extracted anterior chamber region (the same applies below)), and the data output from the second neural network 1037 is a cellular region. In other words, the second segmentation unit 1032A is configured to receive the anterior chamber region identified by the first segmentation unit 1031, input this anterior chamber region to the second neural network 1037 of the second inference model 1036, and obtain output data (cellular regions in the input anterior chamber region) from the second neural network 1037.

The construction of the second inference model 1036 (second neural network 1037) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the second inference model 1036 (second neural network 1037) is performed by the model construction unit 2000 shown in FIG. 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

The training data used to construct the second neural network 1037 may include one or more Scheimpflug images (e.g., anterior segment images with the anterior chamber region identified, anterior chamber images) acquired for one or more eyes. The types of images included in the training data are not limited to Scheimpflug images; for example, the training data may include images acquired using other ophthalmic modalities, images acquired using imaging diagnostic modalities of any medical department, images generated by processing images of actual eyes, pseudo-images, etc.

The training method (machine learning method) for constructing the second neural network 1037 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. This annotation involves, for example, identifying and labeling cellular regions in each image included in the training data. The identification of cellular regions is performed, for example, by at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the second neural network 1037 by applying supervised learning using such training data to the neural network 2020.

The second inference model 1036, which includes the second neural network 1037 constructed in this manner, is a trained model that receives as input the anterior chamber region identified by the first segmentation unit 1031 and outputs the cellular region (e.g., information indicating the range or position of the cellular region) within the input anterior chamber region.

In this example, the second segmentation unit 1032A uses this second inference model 1036 (second neural network 1037) to perform processing to identify cellular regions from the anterior chamber region in the Scheimpflug image of the subject's eye.

The cell evaluation information generation processing unit 1033 includes a processor that executes cell evaluation information generation processing to generate cell evaluation information, and generates cell evaluation information from the cell region identified by the second segmentation unit 1032.

Figure 7 shows an example configuration of the cell evaluation information generation processing unit 1033 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1033A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1038 (referred to as the third inference model). The third inference model 1038 includes a neural network 1039 (referred to as the third neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

The eye images included in the training data of this example include at least an anterior chamber image in which an image of inflammatory cells is depicted, and may further include an anterior chamber image in which an image of inflammatory cells is not depicted. The eye images included in the training data of this example may include the results of manual or automatic segmentation of anterior segment images (e.g., Scheimpflug images, images acquired with other modalities), and may be, for example, cell images extracted from the anterior chamber image in the anterior segment image, or information indicating the range or location of a cellular region in the anterior chamber image.

The data input to the third neural network 1039 is the output from the second segmentation unit 1032 or data based thereon (e.g., data indicating the range, position, distribution, etc. of the cellular region, or the anterior chamber region accompanied by the cellular region identification results), and the data output from the third neural network 1039 is cell evaluation information. That is, the cell evaluation information generation processing unit 1033A is configured to receive the cellular region identification results from the second segmentation unit 1032 or data based thereon, input the cellular region identification results or data based thereon to the third neural network 1039 of the third inference model 1038, and acquire output data (cell evaluation information) from the third neural network 1037. As described above, the cell evaluation information is evaluation information regarding specific parameters related to inflammatory cells (e.g., the density, number, position, distribution, etc. of inflammatory cells).

The construction of the third inference model 1038 (third neural network 1039) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the third inference model 1038 (third neural network 1039) is performed by the model construction unit 2000 shown in FIG. 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

The training data used to construct the third neural network 1039 may include one or more Scheimpflug images acquired for one or more eyes (e.g., an anterior segment image including an anterior chamber region in which cellular regions are identified, an anterior chamber image in which cellular regions are identified). The types of images included in the training data are not limited to Scheimpflug images; for example, the training data may include images acquired using other ophthalmic modalities, images acquired using an imaging diagnostic modality of any medical department, images generated by processing images of actual eyes, pseudo-images, etc.

The training method (machine learning method) for constructing the third neural network 1039 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. For example, in this annotation, each image (in which a cellular region is identified) included in the training data is labeled with cell evaluation information generated from that image. The generation of cell evaluation information from images is performed, for example, by at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the third neural network 1039 by applying supervised learning using such training data to the neural network 2020.

The third inference model 1038, which includes the third neural network 1039 constructed in this manner, is a trained model that receives as input the cell region identification results from the second segmentation unit 1032 or data based thereon, and outputs cell evaluation information based on the input cell region identification results or data based thereon.

In this example, the cell evaluation information generation processing unit 1033A uses this third inference model 1038 (third neural network 1039) to perform processing to generate cell evaluation information from a cell region in the anterior chamber region in a Scheimpflug image of the subject's eye.

The data processing unit 1040 shown in Figure 8 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1040 in this example includes a first segmentation unit 1041, a conversion processing unit 1042, a second segmentation unit 1043, and a cell evaluation information generation processing unit 1044.

The first segmentation unit 1041 has the same configuration and function as the first segmentation unit 1031 in FIG. 3 (e.g., the first segmentation unit 1031A in FIG. 4), and is configured to perform first segmentation to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

The conversion processing unit 1042 converts the anterior chamber region identified by the first segmentation unit 1041 into data with a structure corresponding to the second segmentation performed by the second segmentation unit 1043. In this example, the second segmentation unit 1043 is configured to perform the second segmentation using a neural network (second neural network) constructed by machine learning, like the second segmentation unit 1032A in FIG. 6. The conversion processing unit 1042 is configured to perform conversion processing to convert the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1041 into image data with a structure corresponding to the input layer of the second neural network of the second segmentation unit 1043.

For example, the input layer of the second neural network (convolutional neural network) of the second segmentation unit 1043 may be configured to accept data of a predetermined structure (shape, format). This predetermined data structure may be, for example, a predetermined image size (e.g., the number of vertical and horizontal pixels) or a predetermined image shape (e.g., square or rectangular). On the other hand, the image size and shape of the anterior chamber region identified by the first segmentation unit 1041 vary depending on the specifications of the ophthalmic device, the conditions and settings at the time of imaging, and individual differences in the dimensions and shape of the subject's eye. The conversion processing unit 1042 converts the structure of the anterior chamber region identified by the first segmentation unit 1041 (e.g., image size and/or image shape) into a structure that can be accepted by the input layer of the second neural network of the second segmentation unit 1043.

The image size conversion may be performed using any known image size conversion technique, and may include, for example, a process of dividing the anterior chamber region identified by the first segmentation unit 1041 into multiple partial images with image sizes corresponding to the input layer, or a process of resizing the anterior chamber region identified by the first segmentation unit 1041 into a single image with an image size corresponding to the input layer. The image shape conversion may be performed using any known image deformation technique. The same may be true for conversion processes of other data structures.

Note that while this disclosure provides detailed explanations of several examples of applying conversion processing corresponding to the structure of a neural network to the anterior chamber region of a Scheimpflug image, the manner of conversion processing and the configuration therefor are not limited to these.

For example, if the image input to the neural network is a Scheimpflug image, it is possible to adopt a configuration in which a similar conversion process is applied to the input Scheimpflug image. Also, if the image input to the neural network is arbitrarily processed image data of a Scheimpflug image, it is possible to adopt a configuration in which a similar conversion process is applied to the input processed image data.

The element that performs the conversion process (the processor that performs the conversion process, referred to as the conversion processing unit) may also be positioned arbitrarily. For example, the conversion processing unit may be positioned at a stage before the target neural network in the series of processes performed based on the acquired Scheimpflug image (for example, a stage before the inference model that includes this neural network, or a stage within this inference model but before this neural network), or it may be positioned inside the target neural network. When positioned inside the target neural network, the conversion processing unit is positioned at a stage before the input layer that accepts input that directly corresponds to the output of this neural network.

The second segmentation unit 1043 has the same configuration and function as the second segmentation unit 1032A in FIG. 6, and is configured to perform second segmentation to identify a cellular region from the anterior chamber region whose data structure has been converted by the conversion processing unit 1042. The second neural network of the second segmentation unit 1043 is configured to receive input of image data (the anterior chamber region whose data structure has been converted) generated by the conversion processing unit 1042 and output a cellular region. The machine learning for constructing the second neural network of this example may be performed in the same manner as the machine learning for constructing the second neural network 1037 in FIG. 6.

The cell evaluation information generation processing unit 1044 has the same configuration and function as the cell evaluation information generation processing unit 1033 in Figure 3 (e.g., the cell evaluation information generation processing unit 1033A in Figure 7), and is configured to execute cell evaluation information generation processing to generate cell evaluation information from the cell region identified by the second segmentation unit 1043.

In this way, the data processing unit 1040 in this example may be configured in such a way that the conversion processing unit 1042 is disposed between the first segmentation unit 1031 and the second segmentation unit 1032 of the data processing unit 1030 in Figure 3. However, the configuration of the data processing unit 1040 in this example is not limited to this.

The data processing unit 1050 shown in Figure 9 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1050 in this example includes a second segmentation unit 1051 and a cell evaluation information generation processing unit 1052.

The second segmentation unit 1051 includes a processor that performs second segmentation to identify cellular regions, and is configured to identify cellular regions from the Scheimpflug image acquired by the image acquisition unit 1010.

Figure 10 shows an example configuration of the second segmentation unit 1051 when the second segmentation of this example is performed using machine learning. The second segmentation unit 1051A of this example is configured to perform the second segmentation using a pre-constructed inference model 1053 (referred to as the fourth inference model). The fourth inference model 1053 includes a neural network 1054 (referred to as the fourth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with another modality, or Scheimpflug images of the eye and eye images acquired with another modality).

In some exemplary embodiments, the fourth neural network 1054 may include at least a portion of the first neural network 1035 of FIG. 4 and at least a portion of the second neural network 1037 of FIG. 6. For example, the fourth neural network 1054 may be a neural network in which the first neural network 1035 and the second neural network 1037 are arranged in series. The fourth neural network 1054 configured in this manner has the function of identifying the anterior chamber region from the Scheimpflug image and the function of identifying a cellular region from the anterior chamber region.

In some exemplary embodiments, the fourth neural network 1054 may be configured with machine learning to directly identify cellular regions from a Scheimpflug image without identifying the anterior chamber region. The fourth neural network 1054 is not limited to these embodiments and may include any machine learning-based neural network for identifying cellular regions from a Scheimpflug image.

The data input to the fourth neural network 1054 is a Scheimpflug image, and the data output from the fourth neural network 1054 is a cell region. That is, the second segmentation unit 1051A is configured to receive a Scheimpflug image, input this Scheimpflug image to the fourth neural network 1054 of the fourth inference model 1053, and obtain output data from the fourth neural network 1054 (cell regions in the input Scheimpflug image).

The construction of the fourth inference model 1053 (fourth neural network 1054) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the fourth inference model 1053 (fourth neural network 1054) is performed by the model construction unit 2000 shown in FIG. 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

The training data used to construct the fourth neural network 1054 may include one or more Scheimpflug images acquired for one or more eyes. The types of images included in the training data are not limited to Scheimpflug images; for example, the training data may include images acquired using other ophthalmic modalities, images acquired using imaging diagnostic modalities of any medical department, images generated by processing images of actual eyes, pseudo-images, etc.

Any images included in the training data may be annotated with information to assist in the processing performed by the fourth neural network 1054. For example, the anterior chamber region in the image may be labeled through prior annotation.

The training method (machine learning method) for constructing the fourth neural network 1054 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. This annotation involves, for example, identifying and labeling cellular regions in each image included in the training data. The identification of cellular regions is performed, for example, by at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the fourth neural network 1054 by applying supervised learning using such training data to the neural network 2020.

The fourth inference model 1053, which includes the fourth neural network 1054 constructed in this manner, is a trained model that receives as input the Scheimpflug image (or its processed image data, etc.) acquired by the image acquisition unit 1010, and outputs the cellular region (e.g., information indicating the range or position of the cellular region) in the input Scheimpflug image.

In this example, the second segmentation unit 1051A uses this fourth inference model 1053 (fourth neural network 1054) to perform a process of identifying cellular regions from the Scheimpflug image of the subject's eye.

The cell evaluation information generation processing unit 1052 includes a processor that executes cell evaluation information generation processing to generate cell evaluation information, and generates cell evaluation information from the cell region identified by the second segmentation unit 1051.

FIG. 11 shows an example configuration of the cell evaluation information generation processing unit 1052 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1052A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1055 (referred to as the fifth inference model). The fifth inference model 1055 includes a neural network 1056 (referred to as the fifth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

The data input to the fifth neural network 1056 is the output from the second segmentation unit 1051 or data based thereon (for example, data indicating the range, position, distribution, etc. of the cellular region, or the anterior chamber region accompanied by the cellular region identification results), and the data output from the fifth neural network 1056 is cell evaluation information. In other words, the cell evaluation information generation processing unit 1052A is configured to receive the cellular region identification results from the second segmentation unit 1051 or data based thereon, input the cellular region identification results or data based thereon to the fifth neural network 1056 of the fifth inference model 1055, and acquire output data (cell evaluation information) from the fifth neural network 1056.

The machine learning method for constructing the fifth inference model 1055 (fifth neural network 1056) may be the same as the machine learning method for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in FIG. 7. Furthermore, the training data used in the machine learning for constructing the fifth inference model 1055 (fifth neural network 1056) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

The fifth inference model 1055, which includes the fifth neural network 1056, is a trained model that receives as input the cell region identification results from the second segmentation unit 1051 or data based thereon, and outputs cell evaluation information based on the input cell region identification results or data based thereon.

In this example, the cell evaluation information generation processing unit 1052A uses this fifth inference model 1055 (fifth neural network 1056) to perform processing to generate cell evaluation information from a cellular region in a Scheimpflug image of the subject's eye.

The data processing unit 1060 shown in Figure 12 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1060 in this example includes a cell evaluation information generation processing unit 1061.

The cell evaluation information generation processing unit 1061 includes a processor that executes cell evaluation information generation processing to generate cell evaluation information, and generates cell evaluation information from the Scheimpflug image acquired by the image acquisition unit 1010.

FIG. 13 shows an example configuration of the cell evaluation information generation processing unit 1061 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1061A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1062 (referred to as the sixth inference model). The sixth inference model 1062 includes a neural network 1063 (referred to as the sixth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

In some exemplary embodiments, the sixth neural network 1063 may include at least a portion of the first neural network 1035 of FIG. 4, at least a portion of the second neural network 1037 of FIG. 6, and at least a portion of the third neural network 1039 of FIG. 7. For example, the sixth neural network 1063 may be a neural network in which the first neural network 1035, the second neural network 1037, and the third neural network 1039 are arranged in series. The sixth neural network 1063 configured in this manner has the functions of identifying an anterior chamber region from a Scheimpflug image, identifying a cellular region from the anterior chamber region, and generating cellular evaluation information from the cellular region.

In some exemplary embodiments, the sixth neural network 1063 may be subjected to machine learning to generate cell evaluation information directly from a Scheimpflug image without identifying the anterior chamber region and/or the cellular region. The sixth neural network 1063 is not limited to these embodiments and may include any machine learning-based neural network for identifying cell evaluation information from a Scheimpflug image.

The data input to the sixth neural network 1063 is the output from the image acquisition unit 1010 or data based thereon, and the data output from the sixth neural network 1063 is cell evaluation information. In other words, the cell evaluation information generation processing unit 1061A is configured to receive the Scheimpflug image (and/or data based on this Scheimpflug image) acquired by the image acquisition unit 1010, input this Scheimpflug image or data based thereon to the sixth neural network 1063 of the sixth inference model 1062, and acquire output data (cell evaluation information) from the sixth neural network 1063.

The machine learning method for constructing the sixth inference model 1062 (sixth neural network 1063) may be the same as the machine learning method for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in FIG. 7. Furthermore, the training data used in the machine learning for constructing the sixth inference model 1062 (sixth neural network 1063) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

The sixth inference model 1062, which includes the sixth neural network 1063, is a trained model that receives as input a Scheimpflug image (and/or data based on this Scheimpflug image) acquired by the image acquisition unit 1010, and outputs cell evaluation information based on the input Scheimpflug image (and/or data based on this Scheimpflug image).

In this example, the cell evaluation information generation processing unit 1061A uses this sixth inference model 1062 (sixth neural network 1063) to perform processing to generate cell evaluation information from a Scheimpflug image of the subject's eye (and/or data based on this Scheimpflug image).

The data processing unit 1070 shown in Figure 14 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1070 in this example includes a first segmentation unit 1071 and a cell evaluation information generation processing unit 1072.

The first segmentation unit 1071 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

FIG. 15 shows an example configuration of the first segmentation unit 1071 when performing the first segmentation of this example using machine learning. The first segmentation unit 1071A of this example is configured to perform the first segmentation using a pre-constructed inference model 1073 (referred to as the seventh inference model). The seventh inference model 1073 includes a neural network 1074 (referred to as the seventh neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

The machine learning technique for constructing the seventh inference model 1073 (seventh neural network 1074) may be similar to the machine learning technique for constructing the first neural network 1035 of the first segmentation unit 1031A in FIG. 4. Furthermore, the training data used in the machine learning for constructing the seventh inference model 1073 (seventh neural network 1074) may be similar to the training data used in the machine learning for constructing the first neural network 1035. In some exemplary embodiments, the seventh neural network 1074 may be the same as or similar to the first neural network 1035, and the seventh inference model 1073 may be the same as or similar to the first inference model 1034.

The seventh inference model 1073, which includes the seventh neural network 1074, is a trained model that receives as input the Scheimpflug image (or its processed image data, etc.) acquired by the image acquisition unit 1010, and outputs the anterior chamber region (e.g., information indicating the range or position of the anterior chamber region) in the input Scheimpflug image.

In this example, the first segmentation unit 1071A uses this seventh inference model 1073 (seventh neural network 1074) to perform processing to identify the anterior chamber region from the Scheimpflug image of the subject's eye.

The cell evaluation information generation processing unit 1072 includes a processor that executes cell evaluation information generation processing to generate cell evaluation information, and generates cell evaluation information from the anterior chamber region identified by the first segmentation unit 1071.

FIG. 16 shows an example configuration of the cell evaluation information generation processing unit 1072 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1072A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1075 (referred to as the eighth inference model). The eighth inference model 1075 includes a neural network 1076 (referred to as the eighth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired with other modalities, or Scheimpflug images of the eye and eye images acquired with other modalities).

In some exemplary embodiments, the eighth neural network 1076 may include at least a portion of the second neural network 1037 of FIG. 6 and at least a portion of the third neural network 1039 of FIG. 7. For example, the eighth neural network 1076 may be a neural network in which the second neural network 1037 and the third neural network 1039 are arranged in series. The eighth neural network 1076 configured in this manner has the function of identifying a cellular region from the anterior chamber region and the function of generating cellular evaluation information from the cellular region.

In some exemplary embodiments, the eighth neural network 1076 may be subjected to machine learning to generate cell evaluation information directly from the anterior chamber region without identifying the cellular region. The eighth neural network 1076 is not limited to these embodiments and may include any machine learning-based neural network for identifying cell evaluation information from the anterior chamber region.

The data input to the eighth neural network 1076 is the output from the first segmentation unit 1071 or data based thereon, and the data output from the eighth neural network 1076 is cell evaluation information. In other words, the cell evaluation information generation processing unit 1072A is configured to receive the anterior chamber region (and/or data based on this anterior chamber region) identified from the Scheimpflug image by the first segmentation unit 1071, input this anterior chamber region or data based thereon to the eighth neural network 1076 of the eighth inference model 1075, and obtain output data (cell evaluation information) from the eighth neural network 1076.

The machine learning method for constructing the eighth inference model 1075 (eighth neural network 1076) may be the same as the machine learning method for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in FIG. 7. Furthermore, the training data used in the machine learning for constructing the eighth inference model 1075 (eighth neural network 1076) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

The eighth inference model 1075, which includes the eighth neural network 1076, is a trained model that receives as input the anterior chamber region (and/or data based on this anterior chamber region) identified by the first segmentation unit 1071 and outputs cell evaluation information based on the input anterior chamber region (and/or data based on this anterior chamber region).

In this example, the cell evaluation information generation processing unit 1072A uses this eighth inference model 1075 (eighth neural network 1076) to perform processing to generate cell evaluation information from the anterior chamber region (and/or data based on this anterior chamber region) in the Scheimpflug image of the subject's eye.

The data processing unit 1080 shown in Figure 17 is an example of the configuration of the data processing unit 1020 in Figure 1. The data processing unit 1080 in this example includes a first segmentation unit 1081, a conversion processing unit 1082, and a cell evaluation information generation processing unit 1083.

The first segmentation unit 1081 has the same configuration and function as the first segmentation unit 1031 in FIG. 3 (e.g., the first segmentation unit 1031A in FIG. 4), and is configured to perform first segmentation to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

The conversion processing unit 1082 converts the anterior chamber region identified by the first segmentation unit 1081 into data with a structure corresponding to the cell evaluation information generation processing executed by the cell evaluation information generation processing unit 1083. In this example, the cell evaluation information generation processing unit 1083 is configured to execute the cell evaluation information generation processing using a neural network (eighth neural network) constructed by machine learning, as in the cell evaluation information generation processing unit 1072A in FIG. 16. The conversion processing unit 1082 is configured to execute conversion processing to convert the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1081 into image data with a structure corresponding to the input layer of the eighth neural network of the cell evaluation information generation processing unit 1083.

For example, the input layer of the eighth neural network (convolutional neural network) of the cell evaluation information generation processing unit 1083 may be configured to accept data of a predetermined structure (shape, format). This predetermined data structure may be, for example, a predetermined image size (e.g., the number of vertical and horizontal pixels) or a predetermined image shape (e.g., a square or rectangle). However, the image size and shape of the anterior chamber region identified by the first segmentation unit 1081 vary depending on factors such as the specifications of the ophthalmic device, the conditions and settings at the time of imaging, and individual differences in the dimensions and shape of the subject's eye. The conversion processing unit 1082 converts the structure (e.g., image size and/or image shape) of the anterior chamber region identified by the first segmentation unit 1081 into a structure that can be accepted by the input layer of the eighth neural network of the cell evaluation information generation processing unit 1083.

The image size conversion may be performed using any known image size conversion technique, and may include, for example, a process of dividing the anterior chamber region identified by the first segmentation unit 1081 into multiple partial images with image sizes corresponding to the input layer, or a process of resizing the anterior chamber region identified by the first segmentation unit 1081 into a single image with an image size corresponding to the input layer. The image shape conversion may be performed using any known image deformation technique. The same may be true for conversion processes of other data structures.

The cell evaluation information generation processing unit 1083 has the same configuration and function as the cell evaluation information generation processing unit 1072 in FIG. 14 (e.g., the cell evaluation information generation processing unit 1072A in FIG. 16), and is configured to execute cell evaluation information generation processing for generating cell evaluation information from data obtained by processing the anterior chamber region identified by the first segmentation unit 1081 using the conversion processing unit 1082.

In this way, the data processing unit 1080 in this example may be configured in such a way that the conversion processing unit 1082 is disposed between the first segmentation unit 1071 and the cell evaluation information generation processing unit 1072 of the data processing unit 1070 in Figure 14. However, the configuration of the data processing unit 1080 in this example is not limited to this.

So far, we have mainly described several examples of the data processing unit 1020 that include an inference model (neural network) constructed using machine learning. However, the data processing unit 1020 is not limited to such a machine learning-based configuration. The data processing unit 1020 according to the present disclosure may be implemented using only a machine learning-based configuration, may be implemented using a combination of a machine learning-based configuration and a non-machine learning-based configuration, or may be implemented using only a non-machine learning-based configuration.

Below, several examples of data processing unit 1020 with only a non-machine learning-based configuration are described. A person skilled in the art would be able to understand aspects of data processing unit 1020 with a combination of machine learning-based and non-machine learning-based configurations based on the several examples of machine learning-based configurations described above and the several examples of data processing unit 1020 with only a non-machine learning-based configuration described below.

The data processing unit 1090 shown in Figure 18 is an example of the configuration of the data processing unit 1020 in Figure 1, and has a non-machine learning-based configuration. The data processing unit 1090 in this example includes a first analysis processing unit 1091, a second analysis processing unit 1092, and a third analysis processing unit 1093.

The first analysis processing unit 1091 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to apply a predetermined analysis process (referred to as the first analysis process) to the Scheimpflug image (and/or its processed image data) acquired by the image acquisition unit 1010 to identify the anterior chamber region in this Scheimpflug image.

The first analysis process may include any known segmentation for identifying the anterior chamber region in the Scheimpflug image. For example, segmentation for identifying the anterior chamber region includes segmentation for identifying an image region corresponding to the cornea (particularly, the posterior surface of the cornea) and segmentation for identifying an image region corresponding to the lens (particularly, the anterior surface of the lens). The image region corresponding to the cornea is referred to as the corneal region, the image region corresponding to the posterior surface of the cornea is referred to as the posterior corneal region, the image region corresponding to the lens is referred to as the lens region, and the image region corresponding to the anterior surface of the lens is referred to as the anterior lens region.

Segmentation of the posterior corneal region may include any known segmentation method. Segmentation of the posterior corneal region can be problematic due to artifacts in the Scheimpflug image and saturation of pixel values. To resolve these issues, the configuration shown in FIG. 2C can be employed, for example. That is, by combining an imaging method using the first imaging system 1014 and the second imaging system 1015 with a Scheimpflug image selection method using the image selection unit 1021, a Scheimpflug image free of artifacts and saturation can be selected, and the posterior corneal region can be identified from this Scheimpflug image.

Segmentation of the anterior lens region may include any known segmentation method. Segmentation of the anterior lens region can be problematic because the appearance of the Scheimpflug image (the appearance of the Scheimpflug image) changes depending on the state of the pupil of the subject's eye (e.g., dilated, non-dilated, small-pupil, etc.). For example, when the subject's eye is non-dilated or has a small pupil, the area of the lens imaged is smaller than when the subject's eye is dilated. To resolve this issue, a process for uniforming the appearance of the Scheimpflug image can be applied, such as a process for estimating the position and shape of the unimaged portion of the anterior lens (the portion covered by the pupil) based on the anterior lens region depicted in the Scheimpflug image. The process for uniforming the appearance of the Scheimpflug image may be performed based on machine learning or non-machine learning. Furthermore, the process for estimating the position and shape of the anterior lens surface may include, for example, any known extrapolation process.

When the image acquisition unit 1010 collects a series of Scheimpflug images by slit scanning, some images may contain problems and some may not, or the degree of problems may vary between images. For example, some images may contain artifacts or saturation and some may not, or some images may contain artifacts of various states (positions, dimensions, shapes, etc.). These phenomena may adversely affect the quality (e.g., stability, robustness, reproducibility, accuracy, precision, etc.) of the processing performed by the data processing unit 1090. In some exemplary embodiments, measures can be taken to prevent these phenomena from occurring or to reduce the adverse effects caused by these phenomena. An example of the former measure is to combine an imaging method using the first imaging system 1014 and the second imaging system 1015 with a Scheimpflug image selection method using the image selection unit 1021. Examples of the latter measure include image correction, noise removal, noise reduction, and image parameter adjustment.

The second analysis processing unit 1092 includes a processor that performs second segmentation to identify cellular regions, and is configured to identify cellular regions by applying the second analysis processing to the anterior chamber region identified from the Scheimpflug image by the first analysis processing unit 1091.

In some exemplary embodiments, the second analysis processing unit 1092 may be configured to identify cellular regions based on the values of each pixel in the anterior chamber region (e.g., at least one of brightness value, R value, G value, and B value). In some exemplary embodiments, the second analysis processing unit 1092 may be configured to apply segmentation to the anterior chamber region to identify cellular regions. This segmentation is performed, for example, according to a program created based on the standard morphology (e.g., size, shape, etc.) of inflammatory cells (cellular regions). In some exemplary embodiments, the second analysis processing unit 1092 may be configured to identify cellular regions by at least a partial combination of these two techniques.

Measures that can be taken when the image acquisition unit 1010 collects a series of Scheimpflug images by slit scanning may be the same as those taken by the first analysis processing unit 1091. Furthermore, taking into consideration that cell regions are generally tiny image regions, measures may be taken to distinguish between cell regions and tiny artifacts. For example, by performing processing to remove artifacts (such as ghosts), it is possible to prevent artifacts from being erroneously detected when detecting cell regions.

The third analysis processing unit 1093 includes a processor that executes a cell evaluation information generation process to generate cell evaluation information, and is configured to generate cell evaluation information by applying the third analysis process to the cell region identified from the anterior chamber region of the Scheimpflug image by the second analysis processing unit 1092.

As mentioned above, the cell evaluation information may be any evaluation information related to inflammatory cells, and may include, for example, information representing the state of inflammatory cells (e.g., any parameter such as density, number, position, distribution, etc.), or may include evaluation information generated based on information on specified parameters related to the state of inflammatory cells.

In some exemplary embodiments, the third analysis processing unit 1093 can determine the density, number, position, distribution, etc. of one or more cellular regions identified by the second analysis processing unit 1092.

The process of determining the density of inflammatory cells includes, for example, setting an image area of a predetermined size (e.g., an image area of 1 mm square) and counting the number of cellular regions detected by the second analysis processing unit 1092 in the set image area. Here, the size of the image area (e.g., a dimension in real space, such as "1 mm") is defined, for example, based on the specifications of the optical system of the ophthalmic device 1000 (e.g., optical system design data and/or actual measurement data of the optical system), and is typically defined as the correspondence between pixels and dimensions in real space (e.g., dot pitch). The cell evaluation information may include information on the density of inflammatory cells determined in this manner, or may include evaluation information obtained from this density information. This evaluation information may include, for example, evaluation results using the classification criteria for uveitis diseases proposed by the SUN Working Group. This classification standard defines grades according to the number of inflammatory cells (i.e., the density (concentration) of inflammatory cells) present in one field of view (a field of view measuring 1 mm square), with grade "0" being defined as less than 1 cell, grade "0.5+" as 1 to 5 cells, grade "1+" as 6 to 15 cells, grade "2+" as 16 to 25 cells, grade "3+" as 26 to 50 cells, and grade "4+" as 50 or more cells. Note that the grade divisions in this classification standard may be made more detailed or coarse. Cell evaluation information may also be generated based on other classification standards.

In some exemplary embodiments, the data processing unit 1090 (third analysis processing unit 1093) may be configured to perform the following: identifying a partial region of the anterior chamber region (e.g., a 1-millimeter square image region) identified from the Scheimpflug image by the first analysis processing unit 1091; determining the number of cellular regions belonging to this partial region; and calculating the density of inflammatory cells based on this number and the dimensions of the partial region. Here, the data processing unit 1090 may be configured to select cellular regions located within this partial region from the cellular regions detected from the entire anterior chamber region by the second analysis processing unit 1092, and to have the third analysis processing unit 1093 determine the density based on the selected cellular regions. Alternatively, the data processing unit 1090 may be configured to analyze the partial region and identify the cellular regions using the second analysis processing unit 1092, and to have the third analysis processing unit 1093 determine the density based on the cellular regions identified from the partial region.

The process of determining the number of inflammatory cells includes, for example, counting the number of cellular regions detected by the second analysis processing unit 1092. The cell evaluation information may include information on the number of inflammatory cells determined in this manner, or may include evaluation information obtained from this number information. For example, the cell evaluation information can determine the average density of inflammatory cells in the entire anterior chamber region by dividing the number of cellular regions detected in the entire anterior chamber region by the dimensions of the anterior chamber region (e.g., area, volume, etc.). Furthermore, the cell evaluation information may include an evaluation result (e.g., grade) based on the number of cellular regions detected in the entire anterior chamber region, or may include the number of cellular regions in a partial region of the anterior chamber region and/or an evaluation result based thereon.

The process of determining the position of inflammatory cells may include, for example, a process of identifying the position of the cellular region detected by the second analysis processing unit 1092. The position of the cellular region may be expressed, for example, as coordinates in a defined coordinate system of the Scheimpflug image, or as a relative position (e.g., distance, direction, etc.) to a predetermined image region (reference region) depicted in the Scheimpflug image. This reference region may be, for example, the corneal region, the posterior corneal region, the lens region, the anterior lens region, or an image region corresponding to the axis of the eye (e.g., a straight line connecting the apex position of the cornea and the apex position of the anterior lens). The cell evaluation information may include information on the position of the inflammatory cells determined in this manner, or may include evaluation information obtained from this position information. For example, the cell evaluation information may include information representing the distribution of inflammatory cells (distribution of multiple cell regions), or may include evaluation results (e.g., grades) based on the positions of one or more inflammatory cells, or may include evaluation results (e.g., grades) based on the positions (distributions) of multiple inflammatory cells.

The operation of the ophthalmic device 1000 will be described. Note that the operation described below is merely an example. For example, any matter related to this disclosure, any matter related to the literature cited in this disclosure, any matter related to the technical field to which the embodiments of this disclosure belong, any matter related to the technical field related to the embodiments of this disclosure, etc. can be combined with the example operation below.

A first example of operation of the ophthalmic apparatus 1000 is shown in Figure 19. It is assumed that various operations (preparatory operations) performed before imaging by the ophthalmic apparatus 1000 have been completed. Preparatory operations include adjusting the table on which the ophthalmic apparatus 1000 is placed, adjusting the chair used by the subject, adjusting the face rest (chin rest, forehead rest, etc.) of the ophthalmic apparatus 1000, aligning the ophthalmic apparatus 1000 with the subject's eye, and adjusting the slit light (for example, adjusting the light intensity, width, length, and direction).

Upon receiving an instruction to start imaging, the ophthalmic device 1000 acquires a Scheimpflug image of the subject's eye using the image acquisition unit 1010 (S1).

Furthermore, the ophthalmic device 1000 generates inflammation state information indicating the inflammatory state of the subject's eye based on the Scheimpflug image acquired in step S1 using the data processing unit 1020 (S2).

In this operational example, the number of Scheimpflug images acquired in step S1 may be preset, and may be one, or two or more (for example, a series of Scheimpflug images collected by slit scanning). In step S2, one of the various data processing methods described above is performed depending on the number of Scheimpflug images acquired in step S1. This data processing may be machine learning-based processing, non-machine learning-based processing, or a combination of machine learning-based processing and non-machine learning-based processing.

If the ophthalmic device 1000 includes two or more imaging systems, it can process multiple Scheimpflug images acquired by these imaging systems. For example, if the ophthalmic device 1000 includes a first imaging system 1014 and a second imaging system 1015 as shown in FIG. 2C, the two or more Scheimpflug images acquired by the first imaging system 1014 and the second imaging system 1015 can be processed by the data processing unit 1020A (image selection unit 1021, inflammation state information generation unit 1022). The data processing performed by the data processing unit 1020A may be machine learning-based processing, non-machine learning-based processing, or a combination of machine learning-based processing and non-machine learning-based processing.

The ophthalmic device 1000 can display the Scheimpflug image acquired in step S1 and/or the inflammation state information generated in step S2 on a display device. The display device may be an element of the ophthalmic device 1000 or an external device connected to the ophthalmic device 1000.

Below, several examples of information display that can be performed by the ophthalmic device 1000 are described. The manner of information display is not limited to these examples. It is possible to at least partially combine at least two of these examples.

In a first example of information display, the ophthalmic device 1000 displays the Scheimpflug image acquired in step S1 and/or the inflammation state information generated in step S2 directly on the display device. The ophthalmic device 1000 may also display other information (referred to as additional information) along with the Scheimpflug image and/or the inflammation state information. The additional information may be any information that is useful for treating the subject's eye along with the Scheimpflug image and/or the inflammation state information.

In a second example of information display, the ophthalmic device 1000 generates an image that simulates an image of the eye acquired with a conventional slit lamp microscope (referred to as a slit lamp image) from the Scheimpflug image and displays the generated simulated image on the display device. This makes it possible to provide an image that simulates a slit lamp image, which has traditionally been used to observe the inflammatory state of the subject's eye and is familiar to many physicians.

The process of generating a pseudo-image from a slit lamp image is formed by machine learning-based processing and/or non-machine learning-based processing.

The machine learning-based processing is performed, for example, using a neural network constructed by machine learning using training data including multiple pairs of Scheimpflug images and slit lamp images. The neural network includes a convolutional neural network. The convolutional neural network is configured to receive a Scheimpflug image as input and output a pseudo-image.

Non-machine learning-based processing may include, for example, processing that transforms the appearance of an image, such as generating artificial blur, color conversion, or image quality conversion. Exemplary aspects of non-machine learning-based processing include constructing a three-dimensional image (e.g., an 8-bit grayscale volume) from a series of Scheimpflug images collected by slit scanning, reducing the maximum value of the pixel value range (pixel value gradation) (e.g., reducing 256 gradations to 10 gradations), setting a region of interest (e.g., a rectangular parallelepiped region of specified dimensions) in the gradation-converted three-dimensional image, and constructing a front image (enface image) of the set region of interest (e.g., maximum intensity projection (MIP)).

The pseudo-image may be an image representing the same area as the Scheimpflug image, which is broadly focused, or it may be a partial area of the area represented by the Scheimpflug image (for example, a field of view, i.e., a field of view measuring 1 mm square, which is the evaluation range in the classification criteria for uveitis diseases proposed by the SUN Working Group).

The ophthalmic device 1000 can, for example, highlight a portion of interest in the pseudo image. Examples of such portions of interest include an image region corresponding to inflammatory cells, an image region corresponding to anterior chamber flare, an image region corresponding to cataracts in the lens, etc.

In a third example of information display, the ophthalmologic device 1000 creates a map showing the inflammatory state of the subject's eye (referred to as an inflammatory state map) and displays it on the display device.

Examples of inflammation state maps include an inflammatory cell map showing the location (distribution) of inflammatory cells in the anterior chamber, and an inflammatory cell density map (inflammatory cell number map) showing the distribution of the density (or number) of inflammatory cells in the anterior chamber. The process of creating these inflammatory cell maps includes, for example, a process of identifying image regions (cell regions) corresponding to inflammatory cells from each of a series of Scheimpflug images collected by slit scanning (second segmentation), a process of determining the position of each identified cell region (e.g., two-dimensional coordinates in the definition coordinate system of the Scheimpflug images, or three-dimensional coordinates in the definition coordinate system of a series of three-dimensional images based on the Scheimpflug images), and a process of creating a map based on the determined position of each cell region.

The ophthalmic device 1000 can display an inflammation state map together with a Scheimpflug image and/or inflammation state information. For example, the ophthalmic device 1000 can display a front image based on a series of Scheimpflug images collected by slit scanning, and an inflammation state map generated based on the same series of Scheimpflug images. Specific examples include displaying the inflammation state map superimposed on the front image, or displaying the front image and the inflammation state map side by side.

A second operation example of the ophthalmic device 1000 is shown in Figure 20. This operation example is performed to obtain cell evaluation information. Unless otherwise specified, any of the items described with respect to the operation example in Figure 19 can be combined with this operation example.

First, the ophthalmic device 1000 acquires a Scheimpflug image of the subject's eye using the image acquisition unit 1010 (S11).

Next, the ophthalmic device 1000 uses the data processing unit 1020 to apply at least one of the first segmentation, second segmentation, and cell evaluation information generation processes described above to the Scheimpflug image acquired in step S11 (S12).

When the ophthalmic device 1000 applies the first segmentation, the second segmentation, and the cell evaluation information generation process to a Scheimpflug image, for example, the data processing unit 1030 in FIG. 3 or the data processing unit 1040 in FIG. 8 is used as the data processing unit 1020 of the ophthalmic device 1000. Furthermore, the data processing unit 1020 in this example may include at least one of the first segmentation unit 1031A in FIG. 4, the second segmentation unit 1032A in FIG. 6, and the cell evaluation information generation processing unit 1033A in FIG. 7.

In another example where the ophthalmic device 1000 applies the first segmentation, the second segmentation, and the cell evaluation information generation process to a Scheimpflug image, the data processing unit 1090 in Figure 18 is used as the data processing unit 1020 of the ophthalmic device 1000.

When the ophthalmic device 1000 applies the second segmentation and cell evaluation information generation processing to the Scheimpflug image (in other words, when the first segmentation is not performed), for example, the data processing unit 1050 in FIG. 9 is used as the data processing unit 1020 of the ophthalmic device 1000. Furthermore, the data processing unit 1020 in this example may include the second segmentation unit 1051A in FIG. 10 and/or the cell evaluation information generation processing unit 1052A in FIG. 11.

When the ophthalmic device 1000 applies the cell evaluation information generation process to the Scheimpflug image (in other words, when the first segmentation and second segmentation are not performed), for example, the data processing unit 1060 in FIG. 12 is used as the data processing unit 1020 of the ophthalmic device 1000. Furthermore, the data processing unit 1020 in this example may include the cell evaluation information generation processing unit 1061A in FIG. 13.

When the ophthalmic device 1000 applies the first segmentation and cell evaluation information generation processing to the Scheimpflug image (in other words, when the second segmentation is not performed), for example, the data processing unit 1070 in FIG. 14 or the data processing unit 1080 in FIG. 17 is used as the data processing unit 1020 of the ophthalmic device 1000. Furthermore, the data processing unit 1020 in this example may include the first segmentation unit 1071A in FIG. 15 and/or the cell evaluation information generation processing unit 1072A in FIG. 16.

The configurations that can be employed to perform step S12 are not limited to these. For example, the data processing unit 1020 of the ophthalmic device 1000 according to some exemplary embodiments may be configured to be able to perform only the first segmentation, only the second segmentation, or only the first and second segmentation out of the first segmentation, second segmentation, and cell evaluation information generation process.

Next, the ophthalmic device 1000 generates cell evaluation information using the data processing unit 1020 (S13). Note that if the cell evaluation information generation process is performed in step S12 and all of the cell evaluation information that should be obtained in this examination has been obtained in step S12, there is no need to perform step S13 (in other words, step S13 is included in step S12).

Step S12 and/or step S13 may partially include user operations. Examples of user operations include an operation for specifying an anterior chamber region in a Scheimpflug image, an operation for specifying a cellular region in a Scheimpflug image, an operation for specifying a cellular region in an anterior chamber region, an operation for creating cellular evaluation information from an anterior chamber region, an operation for creating cellular evaluation information from a cellular region, an operation for editing (correcting) an anterior chamber region identified in the first segmentation, an operation for editing (correcting) a cellular region identified in the second segmentation, an operation for editing (correcting) cell evaluation information generated in the cell evaluation information generation process, and an operation for creating other cellular evaluation information from the cellular evaluation information generated in the cell evaluation information generation process.

These operations are performed using a user interface. The user interface includes a display device and an operation device. The ophthalmic device 1000 may include at least a part of the user interface.

Next, the ophthalmic device 1000 displays on the display device the Scheimpflug image acquired in step S11, the information acquired in step S12 (e.g., information based on the Scheimpflug image, anterior chamber region, information based on the anterior chamber region, cell region, information related to the cell region, cell evaluation information), and the cell evaluation information generated in step S13 (S14).

A third operation example of the ophthalmic device 1000 is shown in Figure 21. This operation example is performed to obtain cell evaluation information. Unless otherwise specified, any of the matters described with respect to the operation example in Figure 19 and/or any of the matters described with respect to the operation example in Figure 20 can be combined with this operation example.

The data processing unit 1020 of the ophthalmologic apparatus 1000 in this example includes a second segmentation unit (e.g., second segmentation unit 1032A or second segmentation unit 1051A) that includes a convolutional neural network, and an element (e.g., conversion processing unit 1042 or conversion processing unit 1082) that performs data structure conversion to match this convolutional neural network.

First, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye using the image acquisition unit 1010 (S21).

Next, the ophthalmic device 1000 uses the data processing unit 1020 to apply a first segmentation (anterior chamber segmentation) for identifying the anterior chamber region from the Scheimpflug image to the Scheimpflug image acquired in step S21 (S22). The anterior chamber segmentation method in this step may be any method, and may include one or both of data processing by the data processing unit 1020 and user operation. Here, the anterior chamber segmentation by the data processing unit 1020 may be any of the various methods described above.

Next, the data processing unit 1020 removes ghosts within the anterior chamber region identified in the anterior chamber segmentation in step S22 (S23). This prevents the ghosts from being mistakenly detected as cell regions in the cell segmentation in step S25, which will be described later.

Next, the data processing unit 1020 converts the anterior chamber region from which ghosts have been removed in step S23 into image data with a structure corresponding to the input layer of the convolutional neural network used in the next step S25 (S24).

Note that in some exemplary embodiments, the order of ghost removal from the anterior chamber region (step S23 in this example) and data structure conversion of the anterior chamber region (step S24 in this example) may be reversed.

Next, the data processing unit 1020 inputs the image data (transformed data of the anterior chamber region) acquired in step S24 into a convolutional neural network of a second segmentation unit configured to perform cell segmentation (S25). This identifies the cellular region within the anterior chamber region acquired in step S22.

Next, the data processing unit 1020 evaluates the density of inflammatory cells in the anterior chamber of the test eye based on the results of the cell region identification performed in step S25 (S26). The processing of this step is performed, for example, by one of the various cell evaluation information generation processing units described above.

Next, the data processing unit 1020 generates cell evaluation information based on the results of the evaluation performed in step S26 (S27). The processing of this step is performed, for example, by one of the various cell evaluation information generation processing units described above.

Next, the ophthalmologic device 1000 displays the information acquired in steps S21 to S27 on the display device (S28).

Examples of information displayed in this step include the Scheimpflug image acquired in step S21, information acquired in step S22 (e.g., information based on the Scheimpflug image, anterior chamber region, information based on the anterior chamber region), information acquired in step S23 (e.g., anterior chamber region with ghosts removed, information based on the anterior chamber region with ghosts removed), information acquired in step S24 (e.g., shaped anterior chamber region, information based on the shaped anterior chamber region), information acquired in step S25 (e.g., cell region, information based on the cell region), information acquired in step S26 (e.g., inflammatory cell density, density evaluation information, information based on density evaluation information), and information acquired in step S27 (e.g., cell evaluation information, information based on cell evaluation information).

In some exemplary embodiments, in step S27, the inflammatory cell density value (the number of inflammatory cells present in a 1 mm square area) used in the evaluation proposed by the SUN Working Group and/or the grade corresponding to this density value are determined, and further, in step S28, at least the density value and/or grade determined in step S27 are displayed.

A fourth operation example of the ophthalmic device 1000 is shown in FIG. 22. This operation example is performed to obtain cell evaluation information. Unless otherwise specified, at least one of any of the matters described with respect to the operation example in FIG. 19, any of the matters described with respect to the operation example in FIG. 20, and any of the matters described with respect to the operation example in FIG. 21 can be combined with this operation example.

The data processing unit 1020 of the ophthalmologic apparatus 1000 in this example includes a cell evaluation information generation processing unit (e.g., cell evaluation information generation processing unit 1033A, cell evaluation information generation processing unit 1052A, cell evaluation information generation processing unit 1061A, or cell evaluation information generation processing unit 1072A) that includes a convolutional neural network, and an element (e.g., conversion processing unit 1042 or conversion processing unit 1082) that performs data structure conversion to match this convolutional neural network.

First, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye using the image acquisition unit 1010 (S31).

Next, the ophthalmic apparatus 1000 uses the data processing unit 1020 to apply a first segmentation (anterior chamber segmentation) for identifying the anterior chamber region from the Scheimpflug image to the Scheimpflug image acquired in step S31 (S32). As with step S22 in the third operation example, the anterior chamber segmentation method in this step may be any method.

Next, the data processing unit 1020 removes ghosts within the anterior chamber region identified in the anterior chamber segmentation of step S32 (S33). This prevents the ghosts from being reflected in the evaluation results in the cell evaluation information generation process of step S35, which will be described later.

Next, the data processing unit 1020 converts the anterior chamber region from which ghosts have been removed in step S33 into image data with a structure corresponding to the input layer of the convolutional neural network used in the next step S35 (S34).

Note that in some exemplary embodiments, the order of ghost removal from the anterior chamber region (step S33 in this example) and data structure conversion of the anterior chamber region (step S34 in this example) may be reversed.

Next, the data processing unit 1020 inputs the image data (converted data of the anterior chamber region) acquired in step S34 into a convolutional neural network of the cell evaluation information generation processing unit configured to execute cell evaluation information generation processing. This generates cell evaluation information based on the anterior chamber region acquired in step S32 (S35).

Next, the ophthalmologic device 1000 displays the cell evaluation information generated in step S35 on the display device (S36).

In some exemplary embodiments, in step S35, the inflammatory cell density value (the number of inflammatory cells present in a 1 mm square area) used in the evaluation proposed by the SUN Working Group and/or the grade corresponding to this density value are determined, and further, in step S36, at least the density value and/or grade determined in step S35 are displayed.

In addition, in some exemplary embodiments, in step S35, the density value of inflammatory cells (the number of inflammatory cells present in a 1 mm square area) used in the evaluation proposed by the SUN Working Group is determined. The data processing unit 1020 in this embodiment determines the grade corresponding to the density value determined in step S35 by referring to predefined data representing the correspondence between density values and grades. In step S36 of this embodiment, the grade determined by the data processing unit 1020 (and the density value determined in step S35 of this embodiment) are displayed.

In addition to the cell evaluation information generated in step S35 and/or information based thereon, the ophthalmic device 1000 can display any information acquired in steps S31 to S34 on the display device. Examples of information that can be displayed in addition to the cell evaluation information include the Scheimpflug image acquired in step S31, information acquired in step S32 (e.g., information based on the Scheimpflug image, anterior chamber region, information based on the anterior chamber region), information acquired in step S33 (e.g., anterior chamber region with ghosts removed, information based on the anterior chamber region with ghosts removed), and information acquired in step S34 (e.g., shaped anterior chamber region, information based on the shaped anterior chamber region).

When generating inflammation state information other than cell evaluation information, the inflammation state information can be generated in the same manner as when generating cell evaluation information. As mentioned above, examples of inflammation state information other than cell evaluation information include information about anterior chamber flare, information about lens opacity, information about the onset and progression of disease, and information about disease activity.

All of this exemplary inflammation status information, as with the cell evaluation information, may be generated and evaluated with reference to the classification criteria for uveitis diseases proposed by the SUN Working Group.

The processes for generating and evaluating these exemplary inflammation status information may be machine learning-based processes or non-machine learning-based processes, or may be a combination of machine learning-based processes and non-machine learning-based processes. The construction of a neural network for performing machine learning-based processes may be performed in the same manner as the construction of a neural network for generating cell evaluation information. A processor for performing non-machine learning-based processes may be configured, for example, to perform at least a process for determining an evaluation target (e.g., anterior chamber flare, crystalline lens opacity, disease onset/progression, disease activity, etc.), and may further be configured to perform a process for making an evaluation based on the determined evaluation target.

Figure 23 shows an example of a specific configuration of an ophthalmic device that can function as the ophthalmic device 1000 described above. The ophthalmic device in this example is a system (slit lamp microscope system 1) that combines a slit lamp microscope and a computer (information processing device).

The slit lamp microscope system 1 includes an illumination system 2, an imaging system 3, a video imaging system 4, an optical path coupling element 5, a movement mechanism 6, a control unit 7, a data processing unit 8, a communication unit 9, and a user interface 10. The cornea of the subject's eye E is indicated by the symbol C, and the crystalline lens is indicated by the symbol CL. The anterior chamber corresponds to the area between the cornea C and the crystalline lens.

As a non-limiting example of the arrangement of elements in the slit lamp microscope system 1, some exemplary embodiments of the slit lamp microscope system 1 include a microscope body, a computer, and a communication device that handles communication between the microscope body and the computer. The microscope body includes an illumination system 2, an imaging system 3, a video imaging system 4, an optical path coupling element 5, and a movement mechanism 6. The computer includes a control unit 7, a data processing unit 8, a communication unit 9, and a user interface 10. The computer may be installed, for example, near the microscope body or on a network.

The combination of the illumination system 2, the imaging system 3, and the movement mechanism 6 is an example of the image acquisition unit 1010 of the ophthalmic device 1000. The illumination system 2 is an example of the illumination system 1011 of the ophthalmic device 1000. The imaging system 3 is an example of the imaging system 1012 of the ophthalmic device 1000.

The illumination system 2 projects slit light onto the anterior segment of the subject's eye E. Reference symbol 2a denotes the optical axis of the illumination system 2 (referred to as the illumination optical axis). The illumination system 2 may have a configuration similar to that of a conventional slit lamp microscope. For example, although not shown, the illumination system 2 includes, in order from the side furthest from the subject's eye E, an illumination light source, a positive lens, a slit forming unit, and an objective lens. The illumination light output from the illumination light source passes through the positive lens and is projected onto the slit forming unit. The slit forming unit generates slit light by transmitting a portion of the illumination light. The slit forming unit has a pair of slit blades. The width of the slit light can be changed by changing the distance between these slit blades (referred to as the slit width). Furthermore, the longitudinal direction of the slit light can be changed by rotating the pair of slit blades. Furthermore, the slit forming unit can change the longitudinal dimension of the slit light. The slit light generated by the slit forming unit is refracted by the objective lens and projected onto the anterior segment of the subject's eye E. Note that the configuration for generating the slit light is not limited to this example, and any configuration that can be used to generate slit light may be used. The illumination system 2 may include a focusing mechanism for changing the focus position of the slit light. This focusing mechanism, for example, moves the objective lens along the illumination optical axis 2a. Alternatively, the focusing mechanism moves a focusing lens arranged between the objective lens and the slit forming section.

Figure 23 is a top view, in which the direction along the axis of the subject's eye E is the Z direction, the direction perpendicular to this, i.e., the left-right direction relative to the subject, is the X direction, and the direction perpendicular to both the X and Z directions (up-down direction, body axis direction) is the Y direction. In this embodiment, the slit lamp microscope system 1 can be aligned with the subject's eye E so that the illumination optical axis 2a coincides with the axis of the subject's eye E, or more broadly, alignment can be performed so that the illumination optical axis 2a is positioned parallel to the axis of the subject's eye E.

The imaging system 3 images the anterior segment of the eye onto which the slit light from the illumination system 2 is projected. Reference symbol 3a denotes the optical axis of the imaging system 3 (referred to as the imaging optical axis). The imaging system 3 includes an optical system 3A and an image sensor 3B. The optical system 3A guides light from the anterior segment of the subject's eye E onto which the slit light is projected to the image sensor 3B. The optical system 3A may have a configuration similar to that of the imaging system of a conventional slit lamp microscope. For example, the optical system 3A includes, in order from the side closest to the subject's eye E, an objective lens, a variable magnification optical system, and an imaging lens. Light from the anterior segment of the subject's eye E onto which the slit light is projected passes through the objective lens and the variable magnification optical system, and is imaged on the imaging surface of the image sensor 3B by the imaging lens. The image sensor 3B receives the light guided by the optical system 3A on its imaging surface. The image sensor 3B includes an area sensor with a two-dimensional imaging area. This area sensor may be, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor. The imaging system 3 may include a focusing mechanism for changing its focus position. This focusing mechanism, for example, moves the objective lens along the imaging optical axis 3a. Alternatively, the focusing mechanism moves a focusing lens disposed between the objective lens and the imaging lens along the imaging optical axis 3a.

The illumination system 2 and the imaging system 3 function as a Scheimpflug camera. That is, the illumination system 2 and the imaging system 3 are configured so that the object plane along the illumination optical axis 2a, the optical system 3A, and the imaging plane of the image sensor 3B satisfy the so-called Scheimpflug condition. More specifically, the YZ plane (including the object plane) passing through the illumination optical axis 2a, the principal plane of the optical system 3A, and the imaging plane of the image sensor 3B intersect on the same straight line. This allows imaging to be performed while focusing on all positions within the object plane (all positions in the direction along the illumination optical axis 2a).

In this embodiment, the illumination system 2 and the imaging system 3 are configured so that the imaging system 3 is focused on at least the area from the posterior surface of the cornea C to the anterior surface of the crystalline lens CL (the anterior chamber). Taking practicality into consideration, the illumination system 2 and the imaging system 3 may be configured so that the imaging system 3 is focused on at least the area from the anterior surface of the cornea C to the posterior surface of the crystalline lens CL. This allows the slit lamp microscope system 1 to capture images of the anterior segment of the subject's eye E with the imaging system 3 focused on the entire area from the apex of the anterior surface of the cornea C (Z = Z1) to the apex of the posterior surface of the crystalline lens CL (Z = Z2). The intersection of the illumination optical axis 2a and the imaging optical axis 3a is located at the coordinate Z = Z0. This condition is achieved according to the configuration and arrangement of the elements included in the illumination system 2, the configuration and arrangement of the elements included in the imaging system 3, and the relative positions of the illumination system 2 and the imaging system 3. Parameters indicating the relative positions of the illumination system 2 and the imaging system 3 include, for example, the angle θ formed between the illumination optical axis 2a and the imaging optical axis 3a. The angle θ is set to, for example, 17.5 degrees, 30 degrees, or 45 degrees. The angle θ may also be variable.

The video capture system 4 captures video of the anterior segment of the subject's eye E in parallel with the imaging of the subject's eye by the illumination system 2 and imaging system 3. The video capture system 4 functions as a video camera.

The optical path combining element 5 combines the optical path of the illumination system 2 (illumination optical path) with the optical path of the video recording system 4 (video recording optical path). The optical path combining element 5 may be, for example, a beam splitter such as a half mirror or a dichroic mirror.

A specific example of an optical system including an illumination system 2, an imaging system 3, a video imaging system 4, and an optical path coupling element 5 is shown in Figure 24. In this example, the imaging system 3 includes two imaging systems (a first imaging system and a second imaging system). In some exemplary embodiments, the optical system of the slit lamp microscope system 1 may include other elements (e.g., any element in the description of the ophthalmic device 1000, any element of a known slit lamp microscope, or any element of a known ophthalmic device) in addition to or in place of the elements shown in Figure 24.

The optical system shown in Figure 24 includes an illumination system 20, a left imaging system 30L, a right imaging system 30R, and a video imaging system 40. The illumination system 20 is an example of illumination system 2. The combination of the left imaging system 30L and the right imaging system 30R is an example of imaging system 3, and is an example of the combination of the first imaging system 1014 and the second imaging system 1015 of the ophthalmic device 1000. The video imaging system 40 is an example of video imaging system 4. The beam splitter 47 is an example of an optical path coupling element 5.

In FIG. 24, reference numeral 20a denotes the optical axis of the illumination system 20 (referred to as the illumination optical axis), reference numeral 30La denotes the optical axis of the left imaging system 30L (referred to as the left imaging optical axis), and reference numeral 30Ra denotes the optical axis of the right imaging system 30R (referred to as the right imaging optical axis). The orientations of the left imaging optical axis 30La and the right imaging optical axis 30Ra are different from each other. θL denotes the angle between the illumination optical axis 20a and the left imaging optical axis 30La, and θR denotes the angle between the illumination optical axis 20a and the right imaging optical axis 30Ra. Angles θL and θR may be equal to or different from each other. Angles θL and θR may each be variable. The illumination optical axis 20a, the left imaging optical axis 30La, and the right imaging optical axis 30Ra intersect at a single point. As in FIG. 23, the Z coordinate of this intersection is denoted as Z0.

The movement mechanism 6 in this example is configured to move the illumination system 20, left imaging system 30L, and right imaging system 30R in the direction indicated by arrow 49 (X direction). In some exemplary embodiments, the illumination system 20, left imaging system 30L, and right imaging system 30R are mounted on a stage that is movable at least in the X direction, and the movement mechanism 6 moves this movable stage in the X direction in accordance with a control signal from the control unit 7.

The illumination system 20 projects slit light onto the anterior segment of the subject's eye E. Similar to the illumination system of a conventional slit lamp microscope, the illumination system 20 includes, in order from the side furthest from the subject's eye E, an illumination light source 21, a positive lens 22, a slit forming section 23, and objective lens groups 24 and 25.

Illumination light (e.g., visible light) output from the illumination light source 21 is refracted by the positive lens 22 and projected onto the slit forming unit 23. A portion of the projected illumination light passes through the slit formed by the slit forming unit 23 to become slit light. The generated slit light is refracted by the objective lens groups 24 and 25, reflected by the beam splitter 47, and projected onto the anterior segment of the subject's eye E.

The left imaging system 30L includes a reflector 31L, an imaging lens 32L, and an image sensor 33L. The reflector 31L and the imaging lens 32L guide light from the anterior segment onto which the slit light is projected by the illumination system 20 (light traveling in the direction of the left imaging system 30L) to the image sensor 33L.

Light traveling from the anterior segment toward the left imaging system 30L is light from the anterior segment onto which the slit light is projected, and is light traveling in a direction away from the illumination optical axis 20a. The reflector 31L reflects this light in a direction approaching the illumination optical axis 20a. The imaging lens 32L refracts the light reflected by the reflector 31L and forms an image on the imaging surface 34L of the imaging element 33L. The imaging element 33L receives this light on the imaging surface 34L.

The left imaging system 30L repeatedly captures images in parallel with the movement of the illumination system 20, left imaging system 30L, and right imaging system 30R by the movement mechanism 6. This allows multiple anterior segment images (a series of Scheimpflug images) to be obtained.

The object plane along the illumination optical axis 20a, the optical system including the reflector 31L and imaging lens 32L, and the imaging plane 34L satisfy the Scheimpflug condition. More specifically, when considering the deflection of the optical path of the imaging system 30L by the reflector 31L, the YZ plane (including the object plane) passing through the illumination optical axis 20a, the principal plane of the imaging lens 32L, and the imaging plane 34L intersect on the same straight line. This allows the left imaging system 30L to capture images by focusing on all positions within the object plane (for example, from the anterior surface of the cornea to the posterior surface of the crystalline lens).

The right imaging system 30R includes a reflector 31R, an imaging lens 32R, and an image sensor 33R. The reflector 31R and imaging lens 32R guide light from the anterior segment onto which the slit light is projected by the illumination system 20 (light traveling in the direction of the right imaging system 30R) to the image sensor 33R. The right imaging system 30R acquires multiple images of the anterior segment (a series of Scheimpflug images) by repeatedly capturing images in parallel with the movement of the illumination system 20, left imaging system 30L, and right imaging system 30R by the movement mechanism 6. The object plane along the illumination optical axis 20a, the optical system including the reflector 31R and imaging lens 32R, and the image sensor 34R satisfy the Scheimpflug condition.

The collection of Scheimpflug images by the left imaging system 30L and the collection of Scheimpflug images by the right imaging system 30R are performed in parallel. The combination of the series of Scheimpflug images collected by the left imaging system 30L and the series of Scheimpflug images collected by the right imaging system 30R corresponds to the combination of the first Scheimpflug image group and the second Scheimpflug image group.

The control unit 7 can synchronize the repeated imaging by the left imaging system 30L and the repeated imaging by the right imaging system 30R. This allows a correspondence relationship to be obtained between the series of Scheimpflug images obtained by the left imaging system 30L and the series of Scheimpflug images obtained by the right imaging system 30R. This correspondence relationship is a temporal correspondence relationship, and more specifically, it pairs images acquired substantially simultaneously.

Alternatively, the control unit 7 or data processing unit 8 can execute processing to determine the correspondence between multiple anterior segment images obtained by the left imaging system 30L and multiple anterior segment images obtained by the right imaging system 30R. For example, the control unit 7 or data processing unit 8 can pair anterior segment images sequentially input from the left imaging system 30L with anterior segment images sequentially input from the right imaging system 30R based on the timing of their input.

The video capture system 40 captures video of the anterior segment of the subject's eye E from a fixed position in parallel with the capture by the left capture system 30L and the capture by the right capture system 30R. Here, the video capture system 40 does not need to be moved by the movement mechanism 6. The video capture system 40 is arranged coaxially with the illumination system 20, but the arrangement is not limited to this. In some exemplary embodiments, the video capture system can be arranged non-coaxially with the illumination system 20.

Light that passes through the beam splitter 47 is reflected by the reflector 48 and enters the video recording system 40. The light that enters the video recording system 40 is refracted by the objective lens 41 and then imaged by the imaging lens 42 onto the imaging surface of the image sensor 43. The image sensor 43 is an area sensor.

The video capture system 40 can be used to monitor, align, and track the movement of the subject's eye E. Furthermore, the video capture system 40 can be used to process a series of Scheimpflug images.

Referring back to Figure 23, the movement mechanism 6 is configured to move the illumination system 2 and the imaging system 3 together in the X direction.

The control unit 7 is configured to control each part of the slit lamp microscope system 1. For example, the control unit 7 controls elements of the illumination system 2 (illumination light source, slit formation unit, focusing mechanism, etc.), elements of the imaging system 3 (focusing mechanism of the optical system 3A, image sensor 3B, etc.), elements of the video imaging system 4 (focusing mechanism, image sensor, etc.), movement mechanism 6, data processing unit 8, communication unit 9, user interface 10, etc.

The control unit 7 can control the illumination system 2, the imaging system 3, and the movement mechanism 6, and the video capture system 4, in parallel. This makes it possible to perform slit scanning (collection of a series of Scheimpflug images) and video capture (collection of a series of time-series images) by the image acquisition unit 1010 of the ophthalmic device 1000 in parallel. Furthermore, the control unit 7 can synchronize the control of the illumination system 2, the imaging system 3, and the movement mechanism 6 with the control of the video capture system 4. This makes it possible to synchronize the slit scanning and video capture by the image acquisition unit 1010 of the ophthalmic device 1000.

In a configuration in which the imaging system 3 includes a left imaging system 30L and a right imaging system 30R, the control unit 7 can synchronize repeated imaging by the left imaging system 30L (collection of a first Scheimpflug image group) with repeated imaging by the right imaging system 30R (collection of a second Scheimpflug image group).

The control unit 7 includes a processor, a main memory device, an auxiliary memory device, etc. The auxiliary memory device stores computer programs such as various control programs. These computer programs may be stored on a computer or storage device accessible to the slit lamp microscope system 1. The functions of the control unit 7 are realized through cooperation between software such as the control programs and hardware such as the processor.

The control unit 7 can apply the following control to the illumination system 2, imaging system 3, and movement mechanism 6 in order to scan the three-dimensional area of the anterior segment of the subject's eye E with slit light.

First, the control unit 7 controls the movement mechanism 6 to position the illumination system 2 and the imaging system 3 at a predetermined scan start position (alignment control). The scan start position is, for example, a position corresponding to the edge (first edge) of the cornea C in the X direction, or a position further away from the axis of the subject's eye E. The symbol X0 in Figure 25A indicates the scan start position corresponding to the first edge of the cornea C in the X direction. Furthermore, the symbol X0' in Figure 25B indicates a scan start position further away from the axis EA of the subject's eye E than the position corresponding to the first edge of the cornea C in the X direction.

The control unit 7 controls the illumination system 2 to start projecting slit light onto the anterior segment of the subject's eye E (slit light projection control). The control unit 7 also controls the imaging system 3 to start capturing video of the anterior segment of the subject's eye E (imaging control). After performing alignment control, slit light projection control, and imaging control, the control unit 7 controls the movement mechanism 6 to start moving the illumination system 2 and imaging system 3 (movement control). The movement control moves the illumination system 2 and imaging system 3 together. That is, the illumination system 2 and imaging system 3 are moved while maintaining the relative position (angle θ, etc.) between the illumination system 2 and imaging system 3 (while satisfying the Scheimpflug condition). The illumination system 2 and imaging system 3 are moved from the aforementioned scan start position to a predetermined scan end position. The scan end position is, for example, a position corresponding to the end (second end) of the cornea C opposite the first end in the X direction, similar to the scan start position, or a position further away from the axis of the subject's eye E.

In this example, the slit scan is applied to the range from the scan start position to the scan end position. This slit scan is achieved by projecting a slit of light onto the anterior segment, with the X direction as the width direction and the Y direction as the length direction, moving the illumination system 2 and the imaging system 3 together in the X direction, and capturing video using the imaging system 3 in parallel (linked and synchronized). The length of the slit light (i.e., the dimension of the beam cross section of the slit light in the Y direction) is set to, for example, equal to or greater than the diameter of the cornea C on the surface of the subject's eye E. Furthermore, the distance traveled by the movement mechanism 6 of the illumination system 2 and the imaging system 3 is set to equal to or greater than the cornea diameter in the X direction. This makes it possible to apply the slit scan to a three-dimensional area including the entire cornea C, enabling imaging of a wide range of the anterior chamber.

Slit scanning in this manner produces multiple anterior segment images (a series of Scheimpflug images) with different projection positions of the slit light. In other words, a moving image is produced that depicts the movement of the slit light projection position in the X direction. An example of multiple such anterior segment images (i.e., a group of frames that make up a moving image) is shown in Figure 26.

Figure 26 shows multiple anterior segment images (frame groups) F1, F2, F3, ..., FN. The subscript n in these anterior segment images Fn (n = 1, 2, ..., N) represents the chronological order. That is, the nth anterior segment image acquired is represented by the symbol Fn. The anterior segment image Fn includes a slit light image An. As shown in Figure 26, the slit light images A1, A2, A3, ..., AN move rightward in time sequence. In the example shown in Figure 26, the scan start position and scan end position correspond to both ends of the cornea C in the X direction. Note that the scan start position and/or scan end position are not limited to this example and may be, for example, a position farther from the axis of the subject's eye E than the corneal edge. Furthermore, the scan direction and number of scans can also be set arbitrarily.

The data processing unit 8 is configured to perform various types of data processing. The data to be processed may be either data acquired by the slit lamp microscope system 1 or data input from an external source.

The data processing unit 8 includes a processor, a main memory device, an auxiliary memory device, etc. The auxiliary memory device stores computer programs such as various data processing programs. These computer programs may be stored in a computer or storage device accessible to the slit lamp microscope system 1. The functions of the data processing unit 8 are realized through cooperation between software such as the data processing programs and hardware such as the processor.

The data processing unit 8 may have any of the configurations described for the data processing unit 1020 of the ophthalmic device 1000 (see Figures 2C and 3 to 18). The configuration of the data processing unit 8 is not limited to these.

The following describes a case where the data processing unit 8 is equipped with an image selection unit 1021 (see Figure 2C). The image selection unit 1021 of this embodiment selects a new series of Scheimpflug images corresponding to the slit scan from the two series of Scheimpflug images collected by the left imaging system 30L and the right imaging system 30R based on the correspondence between these two series of Scheimpflug images (the correspondence between the first Scheimpflug image group and the second Scheimpflug image group). The data processing unit 8 generates inflammation state information based on the new series of Scheimpflug images selected by the image selection unit 1021, similar to the inflammation state information generation unit 1022 in Figure 2C.

The configuration of the data processing unit 8 is not limited to these examples. Some exemplary embodiments of the data processing unit 8 may have any data processing function related to the technology disclosed by any of the applicants of the present application, such as any data processing function disclosed in Patent Document 3 (JP 2019-213733 A).

The communication unit 9 performs data communication between the slit lamp microscope system 1 and other devices. That is, the communication unit 9 transmits data to other devices and receives data transmitted from other devices.

The data communication method implemented by the communication unit 9 may be any method. For example, the communication unit 9 includes one or more of various communication interfaces, such as an Internet-compliant communication interface, a dedicated line-compliant communication interface, a LAN-compliant communication interface, or a short-range communication-compliant communication interface. Data communication may be wired or wireless.

Data transmitted and received by the communication unit 9 may be encrypted. In this case, for example, the control unit 7 and/or the data processing unit 8 include at least one of an encryption processing unit that encrypts data transmitted by the communication unit 9 and a decryption processing unit that decrypts data received by the communication unit 9.

The user interface 10 includes any user interface device, such as a display device or an operation device. Users such as doctors, subjects, and assistants can use the user interface 10 to operate the slit lamp microscope system 1 and input information into the slit lamp microscope system 1.

The display device displays various types of information under the control of the control unit 7. The display device may include a flat panel display such as a liquid crystal display (LCD). The operation device includes devices for operating the slit lamp microscope system 1 and devices for inputting information. The operation device includes, for example, buttons, switches, levers, dials, handles, knobs, mice, keyboards, trackballs, operation panels, etc. A device in which the display device and operation device are integrated, such as a touch screen, may also be used.

At least a portion of the user interface may be arranged as a peripheral device of the slit lamp microscope system 1.

The elements of the slit lamp microscope system 1 are not limited to those described above. The slit lamp microscope system 1 may include any element that can be combined with a slit lamp microscope, or more generally, any element that can be combined with an ophthalmic device. The slit lamp microscope system 1 may also include any element for processing data of the subject's eye acquired by the slit lamp microscope, or more generally, any element for processing any ophthalmic data.

For example, the slit lamp microscope system 1 may be equipped with a fixation system that outputs light (fixation light) for fixating the subject's eye E. The fixation system typically includes at least one visible light source (fixation light source) or a display device that displays an image such as a landscape chart or fixation target. The fixation system is arranged, for example, coaxially or non-coaxially with the illumination system 2 or the imaging system 3.

The ophthalmic apparatus 1000 and slit lamp microscope system 1 described above are equipped with a function for photographing the subject's eye (photography function, image acquisition unit 1010), but the ophthalmic apparatus according to the present disclosure is not limited to such a device (ophthalmic photography device). Some exemplary embodiments of the ophthalmic apparatus include a computer (information processing device) that has a function for externally receiving images of the subject's eye instead of (or in addition to) the photography function.

An example configuration of an ophthalmic apparatus as such an information processing device is shown in Figure 27. The ophthalmic apparatus 3000 of this example includes an image acquisition unit 3010 and a data processing unit 3020. The image acquisition unit 3010 includes an image reception unit 3011. The image acquisition unit 3010 may further include a configuration similar to that of the image acquisition unit 1010 of the ophthalmic apparatus 1000. The data processing unit 3020 may include any of the configurations described for the data processing unit 1020 of the ophthalmic apparatus 1000, but is not limited to this.

The image receiving unit 3011 is configured to receive a previously acquired Scheimpflug image of the subject's eye (in other words, a Scheimpflug image of the subject's eye acquired by previous photography). The image receiving unit 3011 includes, for example, a communications device and/or a media drive. The communications device is configured to receive data stored in an external storage device, similar to, for example, the communications unit 9 of the slit lamp microscope system 1. The media drive is configured to read data recorded on a recording medium.

The data processing unit 3020 is configured to perform processing to generate inflammation state information indicating the inflammation state of the subject's eye from the Scheimpflug image received by the image receiving unit 3011. For information about the processing that can be performed by the data processing unit 3020, please refer to the description of the ophthalmic apparatus 1000 and the description of the slit lamp microscope system 1.

This disclosure presents some exemplary aspects of embodiments. These aspects are merely examples of the present invention. Therefore, any modifications (omissions, substitutions, additions, etc.) within the scope of the present invention may be applied to this disclosure.

It is possible to create a program that causes a computer to execute any one or more of the processes described in this disclosure. It is also possible to create a recording medium on which such a program is recorded. The recording medium is a non-transitory recording medium that can be read by a computer. Such a recording medium may take any form, and examples include a magnetic disk, optical disk, magneto-optical disk, and semiconductor memory.

The present invention may include a method including any one or more of the steps described in this disclosure. A method according to some exemplary aspects is a method for controlling an ophthalmic device (e.g., the ophthalmic device 1000, the slit lamp microscope system 1, or the ophthalmic device 3000) including a processor, and includes a step of causing the ophthalmic device to acquire a Scheimpflug image of the subject's eye (referred to as a first acquisition step), and a step of causing the processor to execute processing for generating inflammation state information indicating the inflammatory state of the subject's eye from the Scheimpflug image (referred to as a first generation step).

The first acquisition process and/or the first generation process may be embodied by any of the following: any of the items in the description of the ophthalmic apparatus 1000; any of the items in the description of the slit lamp microscope system 1; and any of the items in the description of the ophthalmic apparatus 3000. Furthermore, any of the steps in the description of the ophthalmic apparatus 1000; any of the steps in the description of the slit lamp microscope system 1; and any of the steps in the description of the ophthalmic apparatus 3000 may be combined with the first acquisition process and the first generation process.

Furthermore, a method according to some exemplary embodiments is a method for processing an image of an eye, and includes a step of acquiring a Scheimpflug image of the subject's eye (referred to as a second acquisition step), and a step of performing processing to generate inflammation state information indicating the inflammatory state of the subject's eye from the Scheimpflug image (referred to as a second generation step).

The second acquisition process and/or the second generation process may be embodied by any of the following: any of the items in the description of the ophthalmic apparatus 1000; any of the items in the description of the slit lamp microscope system 1; and any of the items in the description of the ophthalmic apparatus 3000. Furthermore, any of the steps in the description of the ophthalmic apparatus 1000; any of the steps in the description of the slit lamp microscope system 1; and any of the steps in the description of the ophthalmic apparatus 3000 may be combined with the second acquisition process and the second generation process.

The present invention may include a program (referred to as a first program) that causes a computer to execute a method for controlling an ophthalmic device. The present invention may also include a program (referred to as a second program) that causes a computer to execute a method for processing eye images. Furthermore, the present invention may include a computer-readable non-transitory recording medium on which the first program is recorded. The present invention may also include a computer-readable non-transitory recording medium on which the second program is recorded. Such non-transitory recording media may be in any form, and examples include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

In the above embodiment, the automatic evaluation of the density of inflammatory cells present in the anterior chamber has been described in particular detail. There are various points to consider when automatically evaluating the density of inflammatory cells. In addition to the various points described in the above embodiment, the inventors have also considered the following points: (1) distinguishing between artifacts and cellular regions in a Scheimpflug image (particularly the anterior chamber region); (2) achieving consistency with conventional evaluation methods performed using slit lamp microscopes (e.g., the evaluation method proposed by the SUN Working Group); and (3) ensuring the quality of the evaluation (e.g., stability, robustness, reproducibility, accuracy, precision, etc.) regardless of adjustments or changes to the imaging conditions (e.g., camera gain).

Regarding (1), several solutions have been proposed in the above embodiments, but it is also possible to use any known artifact detection or artifact removal technology, such as the detection or removal of artifacts caused by eyelashes.

Regarding (2), several solutions have been proposed in the above embodiments, but other solutions can also be used, for example, by determining a correspondence between a group of data obtained by a conventional evaluation method and a group of data obtained by the evaluation method according to the present disclosure, and improving consistency with the conventional evaluation method based on this correspondence. Machine learning-based processing and/or non-machine learning-based processing is used to create the correspondence between the groups of data. In machine learning-based processing, for example, machine learning is performed using training data including multiple pairs of a group of data obtained by a conventional evaluation method and a group of data obtained by the evaluation method according to the present disclosure. The inference model constructed in this way includes a neural network that receives as input the data obtained by the evaluation method according to the present disclosure and outputs data that mimics the data obtained by the conventional evaluation method.

Similarly, for (3), correspondences between various data obtained in response to various shooting conditions (e.g., various camera gain values) can be determined, and evaluation quality can be stabilized based on these correspondences. Machine learning-based processing and/or non-machine learning-based processing is used to create correspondences between data. In machine learning-based processing, for example, machine learning is performed using training data including multiple pairs of data obtained under first conditions and data obtained under second conditions. The inference model constructed in this way includes a neural network that receives data obtained under the first conditions (or second conditions) as input and outputs data modeled after data obtained under the second conditions (or first conditions).

The ophthalmic device, method for controlling the ophthalmic device, method for processing eye images, program, and recording medium disclosed herein make it possible to at least partially automate the evaluation of inflammatory conditions based on eye images, which has traditionally been performed manually.

In this disclosure, the inflammatory state is evaluated based on Scheimpflug images, allowing evaluation to be performed based on high-quality images that are in focus over a wide range. This makes it possible to perform high-quality evaluation of a wide range of the subject's eye.

In addition, by combining slit scanning, it is possible to quickly obtain a high-quality Scheimpflug image group (a series of Scheimpflug images) that is in focus over a wide three-dimensional area of the subject's eye, and evaluation can be performed based on this Scheimpflug image group, making it possible to evaluate a very wide area of the subject's eye with high quality. For example, it becomes possible to evaluate a wide area of the anterior chamber, and it is also possible to add the lens and cornea to the evaluation targets.

In the invention described in Patent Document 5 (WO 2018/003906), scattered light cannot be detected if the exposure time required to capture one frame is short, such as at video rate, so the exposure time required to capture one frame is set to approximately 100 milliseconds to 1 second. However, considering the effects of eye movement and blinking of the subject's eye, slit scanning as disclosed herein cannot be performed.

Furthermore, in the invention described in Patent Document 5 (WO 2018/003906), the dimensions of the projected image of the slit light on the cornea are set to 0.2 mm x 2 mm, making it difficult to image a wide area of the anterior segment. In contrast, in this embodiment, the dimensions of the projected image of the slit light on the cornea can be set to, for example, approximately 0.05 mm x 8 to 12 mm, making it possible to image a wide area of the anterior segment.

Furthermore, in this embodiment, white LEDs can be used instead of blue LEDs as in the invention described in Patent Document 5 (WO 2018/003906), and evaluation can be performed using color information (R signal, G signal, B signal) using a color camera instead of a monochrome camera.

As such, this embodiment makes it possible to image a wide area of the anterior segment, and in addition to obtaining anterior segment images, images of inflammatory cells, and information on the inflammatory state, it also makes it possible to present and analyze the shape of the anterior segment, which has the advantage of being able to provide a variety of information to doctors.

1000 Ophthalmic apparatus 1010 Image acquisition unit 1011 Illumination system 1012 Imaging system 1020 Data processing unit 1021 Image selection unit 1022 Inflammation state information generation unit 1031 First segmentation unit 1032 Second segmentation unit 1033 Cell evaluation information generation processing unit 3000 Ophthalmic apparatus 3010 Image acquisition unit 3011 Image reception unit 3020 Data processing unit

Claims (26)

an image acquisition unit that acquires a Scheimpflug image of the subject's eye;
a data processing unit that executes processing for generating inflammation state information indicating an inflammation state of the subject's eye from the Scheimpflug image,
the image acquisition unit includes an imaging system configured to satisfy the Scheimpflug condition, which images the subject's eye, and an illumination system configured to project slit light onto the subject's eye, and scans a three-dimensional region of the subject's eye with the slit light to collect a series of Scheimpflug images;
the data processing unit generates the inflammation state information indicating an inflammation state in a three-dimensional region in the anterior chamber of the subject's eye based on the series of Scheimpflug images.
Ophthalmology equipment. the inflammation state information includes cell evaluation information that is evaluation information regarding inflammatory cells in the anterior chamber of the subject's eye,
the data processing unit executes at least one of a first segmentation for identifying an anterior chamber region corresponding to the anterior chamber, a second segmentation for identifying a cellular region corresponding to the inflammatory cells, and a cell evaluation information generation process for generating the cell evaluation information.
The ophthalmic device of claim 1. the data processing unit executes the first segmentation for identifying the anterior chamber region from the Scheimpflug image, the second segmentation for identifying the cellular region from the anterior chamber region identified by the first segmentation, and the cell evaluation information generation process for generating the cell evaluation information from the cellular region identified by the second segmentation.
The ophthalmic device of claim 2. the data processing unit performs the first segmentation using a pre-constructed first inference model;
the first inference model includes a first neural network constructed by machine learning using training data including at least an eye image;
the first neural network is configured to receive an input of a Scheimpflug image and output an anterior chamber region.
The ophthalmic apparatus of claim 3. the data processing unit performs the second segmentation using a second inference model constructed in advance;
the second inference model includes a second neural network constructed by machine learning using training data including at least an eye image;
the second neural network is configured to receive an input of an anterior chamber region in a Scheimpflug image and output a cellular region.
5. The ophthalmic apparatus according to claim 3 or 4. the data processing unit further performs a conversion process of converting the anterior chamber region of the Scheimpflug image into image data having a structure corresponding to the input layer of the second neural network;
The second neural network is configured to receive the image data generated by the conversion process as an input and output a cell region.
The ophthalmic device of claim 5. the data processing unit executes the cell evaluation information generation process using a third inference model constructed in advance;
the third inference model includes a third neural network constructed by machine learning using training data including at least an eye image;
the third neural network is configured to receive an input of a cell region in a Scheimpflug image and output cell evaluation information;
The ophthalmic device according to any one of claims 3 to 6. the data processing unit executes the second segmentation for identifying the cell region from the Scheimpflug image, and the cell evaluation information generation process for generating the cell evaluation information from the cell region identified by the second segmentation.
The ophthalmic device of claim 2. the data processing unit performs the second segmentation using a fourth inference model constructed in advance;
the fourth inference model includes a fourth neural network constructed by machine learning using training data including at least an eye image;
the fourth neural network is configured to receive a Scheimpflug image as an input and output a cell region.
The ophthalmic device of claim 8. the data processing unit executes the cell evaluation information generation process using a fifth inference model constructed in advance;
the fifth inference model includes a fifth neural network constructed by machine learning using training data including at least an eye image;
The fifth neural network is configured to receive an input of a cell region in a Scheimpflug image and output cell evaluation information.
10. The ophthalmic apparatus according to claim 8 or 9. the data processing unit executes the cell evaluation information generation process for generating the cell evaluation information from the Scheimpflug image.
The ophthalmic device of claim 2. the data processing unit executes the cell evaluation information generation process using a sixth inference model constructed in advance;
the sixth inference model includes a sixth neural network constructed by machine learning using training data including at least an eye image;
the sixth neural network is configured to receive an input of a Scheimpflug image and output cell evaluation information;
The ophthalmic device of claim 11. the data processing unit executes the first segmentation for identifying the anterior chamber region from the Scheimpflug image, and the cell evaluation information generation process for generating the cell evaluation information from the anterior chamber region identified by the first segmentation.
The ophthalmic device of claim 2. the data processing unit performs the first segmentation using a seventh inference model constructed in advance;
the seventh inference model includes a seventh neural network constructed by machine learning using training data including at least an eye image;
the seventh neural network is configured to receive an input of a Scheimpflug image and output an anterior chamber region.
The ophthalmic device of claim 13. the data processing unit executes the cell evaluation information generation process using a pre-constructed eighth inference model;
the eighth inference model includes an eighth neural network constructed by machine learning using training data including at least an eye image;
the eighth neural network is configured to receive an input of an anterior chamber region in a Scheimpflug image and output cell evaluation information;
15. An ophthalmic device according to claim 13 or 14. the data processing unit further performs a conversion process of converting the anterior chamber region of the Scheimpflug image into image data having a structure corresponding to the input layer of the eighth neural network;
the eighth neural network is configured to receive the image data generated by the conversion processing as input and output cell evaluation information.
The ophthalmic device of claim 15. the data processing unit executes the first segmentation for specifying the anterior chamber region by applying a first analysis process to the Scheimpflug image, the second segmentation for specifying the cellular region by applying a second analysis process to the anterior chamber region specified by the first segmentation, and the cell evaluation information generation process for generating the cell evaluation information by applying a third analysis process to the cellular region specified by the second segmentation.
The ophthalmic device of claim 2. the data processing unit, in the third analysis process, identifies a partial region of the anterior chamber region identified by the first segmentation, calculates the number of the cellular regions belonging to the partial region, and calculates the density of the inflammatory cells based on the number and a dimension of the partial region;
The cell evaluation information includes the density.
18. The ophthalmic device of claim 17. the data processing unit processes the series of Scheimpflug images to generate processed image data, and generates the inflammation state information from the processed image data.
An ophthalmic device according to any one of claims 1 to 18 . the imaging system includes a first imaging system and a second imaging system that image the subject's eye from different directions,
An ophthalmic device according to any one of claims 1 to 19 . an optical axis of the first imaging system and an optical axis of the second imaging system are arranged to be inclined in opposite directions with respect to an optical axis of the illumination system;
the data processing unit selects one of a first Scheimpflug image acquired by the first imaging system and a second Scheimpflug image acquired by the second imaging system, and generates the inflammation state information based on the selected Scheimpflug image.
The ophthalmic device of claim 20 . the data processing unit selects a Scheimpflug image that does not include a corneal reflection artifact from among the first Scheimpflug image and the second Scheimpflug image.
22. The ophthalmic device of claim 21 . the image acquisition unit includes an image reception unit that receives the Scheimpflug image acquired in advance.
An ophthalmic device according to any one of claims 1 to 22 . A method for controlling an ophthalmic apparatus including a processor and an image acquisition unit that acquires a Scheimpflug image of a subject's eye , comprising:
the image acquisition unit includes an imaging system configured to satisfy the Scheimpflug condition, the imaging system capturing an image of the subject's eye, and an illumination system projecting slit light onto the subject's eye;
causing the image acquisition unit to scan a three-dimensional region of the subject's eye with the slit light to collect a series of Scheimpflug images ;
causing the processor to execute processing for generating inflammation state information indicating an inflammation state in a three-dimensional region within the anterior chamber of the subject's eye based on the series of Scheimpflug images ;
method. A program that causes a computer to execute the method of claim 24 . A computer-readable non-transitory recording medium on which the program of claim 25 is recorded.