JP7722757B2 - Image processing to improve the resolution of images taken of specimens - Google Patents

Description

The present invention relates to an image processing device, an image processing system, an image processing method, a program, and an information recording medium that perform estimation to improve the resolution of an input image of a specimen and obtain an output image.

Super-resolution technology has been proposed, which estimates an output image by improving the resolution of an input image. Super-resolution involves making estimates based on the characteristics of the object captured in the input image, and can use neural networks trained using deep learning, etc. To train a neural network, a photograph of the subject expected to be processed is prepared, and this photograph is then processed by adding noise, mosaicking, blurring, reducing the resolution, etc. to serve as the training input image. This photograph is then used as the training output (ground truth) image. (See, for example, Patent Document 1 and Non-Patent Document 2.)

On the other hand, various proposals have been made for image registration (image alignment) techniques that match feature points of two images of the same subject and obtain a mapping that represents the transformation from one image to the other, or the inverse transformation (see, for example, non-patent document 1).

Now, in the field of pathological photography using optical microscopes, for example,
The obtained specimen is immersed in formalin water or similar to fix the tissue,
The water in the tissue is replaced with paraffin and embedded.
Thinly slice the tissue using a microtome or other instrument to obtain sections.
Each section was attached to a glass plate.
The paraffin is deparaffinized by melting it back into water.
Tissue specimens are prepared by staining procedures using various dyes.

By observing and photographing this tissue specimen under an optical microscope, a digital pathology image is obtained.

However, optical microscopes have a resolution limit due to the diffraction of light. This limit is called the diffraction limit, and for typical optical systems, it is smaller than biological cells (1 μm to 100 μm) but larger than viruses (100 nm), proteins (10 nm), and less complex molecules (1 nm).

In order to obtain a resolution exceeding the diffraction limit, it is conceivable to use an electron microscope. In the field of electron microscopy, for example,
The obtained specimen is immersed in an aldehyde-based fixative to pre-fix the tissue.
After washing with rinse solution,
Post-fixation was performed with osmium solution.
Dehydrate using ethanol,
Embedding in epoxy resin,
Ultra-thin slices were obtained using an ultramicrotome, diamond knife, etc.
A tissue specimen is prepared by a procedure for performing electronic staining.

By observing and photographing this tissue specimen under an electron microscope, a digital electron micrograph is obtained.

Because electron microscopes use electron beams rather than light for observation, their diffraction limit exceeds that of optical microscopes and is generally considered to be around a few nanometers.

In Patent Document 2, machine learning is performed based on images captured using an optical microscope (first microscope) and an electron microscope (second microscope). The image captured by the first microscope is a cell nucleus A1 within an iPS cell (first region), and the image captured by the electron microscope is a cell nucleus A2 within a brain tissue cell (second region), with different specimens being observed for different regions.

As such, when the waves used for imaging are different, such as visible light and electron beams, the wavelengths of the waves also differ, and generally, the shorter the wavelength, the higher the spatial resolution.

In addition, when examining skin tumors or cancers, dermatoscopic diagnosis is performed, in which the affected area is observed and photographed visually or through a magnifying glass using normal light, and then, if necessary, observed and photographed using polarized light. An observation tool called a dermatoscope is capable of observation and photography using both normal and polarized light. Furthermore, technology has been proposed for dermatoscopes that switches the wavelength of the light emitted to a specific wavelength band, as disclosed in Patent Document 3, for example.

Here, normal light (unpolarized waves) makes it possible to observe and photograph the surface layer of the skin. Polarized light (polarized waves) makes it possible to observe and photograph the structure of layers deeper than the surface of the skin. Furthermore, by using light of different wavelength bands, it is possible to observe and photograph tissues with specific structures that correspond to the wavelength bands, with the tissues being highlighted.

Therefore, by using different waves, it is possible to improve resolution for areas that are farther away in the depth direction from the surface of the object being observed, as well as resolution based on tissue structure.

Japanese Patent Publication No. 2022-056769 Patent Publication No. 2021-18582 Special Publication No. 2022-527642

Jeremy Joslove and Emna Kamoun, "Image Registration: From SIFT to Deep Learning", Sicara's blog, [online], https://medium.com/sicara/image-registration-sift-deep-learning-3c794d794b7a (overview and introduction), https://www.sicara.fr/blog-technique/2019-07-16-image-registration-deep-learning (main text), July 16, 2019 Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi, "Image Super-Resolution via Iterative Refinement", Google Research, Brain Team [online], https://doi.org/10.48550/arXiv.2104.07636, April 15, 2021

Here, from the perspective of the diffraction limit, i.e., spatial resolution, the performance of electron microscopes exceeds that of optical microscopes. Therefore, there is a demand for observing pathological images taken with an optical microscope, and then observing the areas captured in the pathological images in more detail using electron microscope images to perform various diagnoses and assessments.

However, preparing specimens for electron microscopes often requires more effort, time, and cost than preparing specimens for light microscopes. The use of electron microscopes themselves can also be problematic in terms of effort, time, and cost compared to light microscopes.

In addition, because the procedures for preparing specimens differ between optical and electron microscopes, it is difficult to obtain appropriate electron microscope photographs even when observing and photographing specimens designed for optical microscopes using an electron microscope.

In addition, while it may be easy to photograph the surface of the affected area using a digital camera in today's working environment, photographing using specialized equipment such as a dermatoscope is often difficult due to the working environment, cost, and effort involved.

Therefore, there is a strong demand for an input image (for example, a pathology image based on a photograph of a pathology specimen taken of a specimen for an optical microscope, or an epidermal image of the skin surface of an affected area taken with a digital camera) to be estimated to improve the resolution of the input image, preferably to an extent that exceeds the diffraction limit at the time the input image was taken, in order to obtain an output image (for example, an estimated image that represents the results of estimating elements of how the tissue corresponding to the specimen appears as if it were photographed with an electron microscope, or how the affected area appears as if it were photographed with polarized light or light in a specific wavelength band in dermatoscopy), and to apply this to various diagnoses and assessments.

The present invention aims to solve the above-mentioned problems by providing an image processing device, image processing system, image processing method, program, and information recording medium that perform estimation to improve the resolution of an input image of a specimen and obtain an output image.

The image processing device according to the present invention comprises:
When an input image of a target specimen is inputted by a first wave having a first resolution,
By providing the input image to a trained super-resolution network, an output image that should be obtained by photographing the target specimen with a second wave having a second resolution is estimated.
Here, the second resolution is higher than the first resolution.

In addition, in the image processing device of the present invention,
The input image may be configured to capture a cross section of the target specimen.

In addition, in the image processing device of the present invention,
The super-resolution network
a training input based on a first training image obtained by photographing a first training cross section that appears by cutting a training specimen with the first wave;
a training output based on a second training image obtained by photographing a second training cross section that appears opposite to the first training cross section with the second wave;
The method can be configured to be trained using training data including

The present invention provides an image processing device, image processing system, image processing method, program, and information recording medium that perform estimation to improve the resolution of an input image of a specimen and obtain an output image.

1 is an explanatory diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention; FIG. 1 is an explanatory diagram illustrating an example of a super-resolution network used by an image processing device according to an embodiment of the present invention. 10 is a flowchart showing the control flow of a training process for training a super-resolution network executed in an image processing device according to an embodiment of the present invention. 4 is a flowchart showing a control flow of image processing executed by the image processing apparatus according to the embodiment of the present invention. 1 is an explanatory diagram showing a schematic configuration of an image processing system according to an embodiment of the present invention; FIG. 2 is an explanatory diagram showing an example of a display on a screen of a terminal computer according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of a display on a screen of a terminal computer according to an embodiment of the present invention. This is a photograph, which is a substitute for a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in grayscale. This is a photograph in place of a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in monochrome binary. This is a photograph, which is a substitute for a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in grayscale. This is a photograph in place of a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in monochrome binary. This is a photograph, which is a substitute for a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in grayscale. This is a photograph in place of a drawing, showing a photographed pathological image, an electron microscope image estimated from the pathological image, and the photographed electron microscope image arranged vertically in monochrome binary. 1 is an explanatory diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention; FIG. 1 is an explanatory diagram showing the configuration of an Encoder in a VAE, which is an example of a generation network in an image processing device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing the configuration of a decoder in a VAE, which is an example of a generation network in an image processing device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram showing the configuration of an Encoder in a VAE, which is another example of a generation network in an image processing device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram showing the configuration of a decoder in a VAE, which is another example of a generation network in an image processing device according to an embodiment of the present invention. This is a photograph in place of a drawing, which shows a captured pathological image, an electron microscope image estimated from the pathological image, and a generated reference image side by side in grayscale. This is a photograph in place of a drawing, showing the captured pathological image, the electron microscope image estimated from the pathological image, and the generated reference image side by side in monochrome binary.

The following describes an embodiment of the present invention. Note that this embodiment is for illustrative purposes only and does not limit the scope of the present invention. Therefore, a person skilled in the art could adopt an embodiment in which each or all of the elements of this embodiment are replaced with equivalents. Furthermore, elements described in each example may be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included within the scope of the present invention.

(composition)
The image processing device according to this embodiment is typically realized by a computer executing a program. The computer is connected to various output devices and input devices, and transmits and receives information to and from these devices.

Programs executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or they can be recorded on non-transitory information recording media such as CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording media can be distributed, sold, etc.

The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, EEPROM, etc., possessed by the computer. The image processing device of this embodiment is then realized by the computer. Generally, the computer's CPU (Central Processing Unit) reads the program from the information recording medium into RAM (Random Access Memory) under the management of the computer's OS (Operating System), and then interprets and executes the code contained in the program. However, in architectures where the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

Furthermore, as mentioned above, it is desirable for computers to be equipped with a GPU (Graphics Processing Unit) to perform various image processing calculations at high speed. By using a GPU and libraries such as TensorFlow, it becomes possible to utilize the learning and classification functions in various artificial intelligence processes under the control of the CPU.

In addition, rather than realizing the image processing device of this embodiment using a general-purpose computer, it is also possible to configure the image processing device of this embodiment using dedicated electronic circuitry. In such an embodiment, an electronic circuit that meets the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, realizing the image processing device of this embodiment.

(Image processing device)
For ease of understanding, the following description will be given assuming that the image processing device is realized by a computer executing a program. Fig. 1 is an explanatory diagram showing the general configuration of an image processing device according to an embodiment of the present invention.

As shown in this figure, the image processing device 101 of this embodiment comprises an input unit 102 and a super-resolution unit 103.

Here, an input image of the target specimen captured with a first wave having a first resolution is input to the input unit 102.

On the other hand, the super-resolution unit 103 provides an input image to the trained super-resolution network 104, causing it to estimate the output image that should be obtained by photographing the target specimen with a second wave having a second resolution.

Any neural network can be used as the super-resolution network 104. For example, a neural network such as that shown in Figure 2 may be used, or a complex neural network such as that shown in Non-Patent Document 2 may also be used.

Here, the second wavelength band of the second wave can be configured to be shorter than the first wavelength band of the first wave.

The first wave can also be configured to be light and the second wave to be an electron beam.

In these configurations, the output image to be captured by the electron microscope is estimated from an input image based on a photograph taken by an optical microscope.

The first wave can also be configured to be unpolarized light and the second wave to be polarized light.

In these configurations, the output image to be captured by the dermatoscope is estimated from an input image based on a photograph taken under normal light.

The super-resolution network 104 is trained using multiple training data.

Here, each training data includes training input (input data) and training output (correct answer data). The training data will be prepared by preparing images of the same subject taken in the past using two types of waves.

For example, when making inferences about dermatoscopes, photographs taken under normal light are used as training inputs, and photographs taken under polarized light are used as training outputs.

As mentioned above, specimens and samples for observation with an optical microscope and those for observation with an electron microscope require different processing methods. As such, it can be difficult to photograph the same object using two different types of waves.

However, when observing a specimen, the cross-section is often examined. Therefore, this embodiment takes advantage of the fact that when a specimen is cut, two opposing cross-sections are obtained.

In this embodiment, input data based on a first training image obtained by photographing a first training cross section that appears by cutting a training specimen with a first wave can be used as training input, and correct answer data based on a second training image obtained by photographing a second training cross section that appears opposite the first training cross section with a second wave can be used as training output.

When the training specimen is cut into two samples, the cross section itself may be significantly deformed due to the cutting. Therefore, after fixing the specimen, several thin slices may be cut from the original cross section, and the inner layer (the layer farthest from the original cross section) may be photographed and used as the training input and training output.

In this case, the training input and training output are each images of sections spaced apart from the specimen's cut surface. Therefore, the physical locations within the specimen of the areas imaged in the training input and training output are not the opposing cross sections themselves, but rather sections located in the vicinity. However, if the separation distance is sufficiently small, these sections can be treated as essentially opposing cross sections.

(Basic operation of image processing device)
The following describes the basic operation of the image processing device 101 that uses the trained super-resolution network 104. Fig. 4 is a flowchart showing the flow of control of image processing executed by the image processing device according to the embodiment of the present invention.

First, in the image processing device 101, the input unit 102 accepts input of an input image of a target specimen captured with a first wave having a first resolution (step S401).

Then, the super-resolution unit 103 provides the accepted input image to the super-resolution network 104 (step S402).

Based on the input image, the super-resolution network 104 estimates the image that should be obtained by photographing the target specimen with a second wave having a second resolution (step S403).

Then, the image processing device 101 outputs the estimated image as the output image (step S404) and terminates this processing.

Note that if the basic operation of the image processing device 101 is provided as a service by a server computer, after step S404, control returns to step S401 and the above processing is repeated.

(Training a super-resolution network)
As described above, the training input and training output used to train the super-resolution network are data obtained by photographing opposing cross sections of a single specimen using different waves, and when the specimen is cut and sampled, deformation of the specimen may occur.

Therefore, when training the super-resolution network 104, correction and alignment of the training input and training output are required to accommodate deformation of the sample.

In this embodiment, the image processing device 101 trains the super-resolution network 104 through the following process. Figure 3 is a flowchart showing the control flow of the training process for training the super-resolution network executed by the image processing device according to an embodiment of the present invention. The following description will be given with reference to this figure.

For ease of understanding, the following describes how the image processing device 101 trains the super-resolution network 104. However, this training process may be performed by the image processing device 101, or may be performed by a device other than the image processing device 101, and the super-resolution network 104 learned by the training process may then be made available to the image processing device 101.

First, the image processing device 101 receives a first training image obtained by photographing a first training section using a first wave motion and a second training image obtained by photographing a second training section opposite the first training section using a second wave motion (step S301).

Then, the image processing device 101 calculates a mapping that aligns the first training image with the second training image (step S302).

The first method for determining the mapping is as follows. That is, because the resolutions of the first and second training images are different, a scale mapping that enlarges or reduces the resolution of both images to match is prepared and applied to the first training image. The scaling ratio of the scale mapping can be determined based on the shooting conditions when the first and second training images were taken, i.e., the "number of pixels in the width and height of the image" determined by the lens magnification and image resolution, and the "actual width and height of the subject corresponding to one pixel in the image."

Then, a transform map is calculated to align the first and second training images after scaling, i.e., the two images with the same resolution.

Finally, the desired mapping is obtained by combining the scale mapping and the transform mapping.

As described above, the scale mapping can be uniquely determined by calculating the enlargement/reduction ratio based on the conditions under which the first training image and the second training image were captured.

On the other hand, the transform map can be obtained by applying various techniques for aligning two images of the same resolution, in addition to the technique disclosed in Non-Patent Document 1.

The second method for determining the mapping is as follows. Specifically, the combination of feature extraction and convolutional neural networks disclosed in Non-Patent Document 1, and image registration techniques such as reinforcement learning and deep learning that determine the direct sum of multiple homography mappings or diffeomorphisms, can be applied even when the resolutions of the two images do not match. In such cases, the desired mapping can be determined by combining the scale mapping and transform mapping into a single entity. Therefore, in such cases, there is no need to determine the mapping in two separate steps: scale mapping and transform mapping.

Then, the image processing device 101 corrects the first training image using the obtained mapping (step S303).

Then, the image processing device 101 divides the corrected first training image into multiple regions (step S304) and repeats the following process for each of the multiple regions (step S305). Note that, of the multiple regions, the region currently being processed in a repetition will be referred to as the "first region" hereinafter.

First, the image processing device 101 cuts out the first region of the corrected first training image to create a first partial image (step S306).

Then, the image processing device 101 determines candidate positions of the area (hereinafter referred to as the "second area") into which the first area is projected in the second training image by the above mapping (step S307).

Next, the image processing device 101 scans a predetermined scanning range around the candidate position using a window of the same shape as the second region, and obtains an image within the window at each position (step S308). For example, if the predetermined scanning range is a range centered on the candidate position and shifted up to n pixels above, below, left, and right, the number of images within the window obtained by scanning will be (2×n+1) ² if the center is included, and (2×n) ² if the center is not included.

Furthermore, the image processing device 101 calculates the similarity between each of the images within the multiple windows obtained by scanning with the window and the first partial image, and selects the image with the greatest similarity as the second partial image (step S309).

Then, the position of the second region is identified as the position of the window cut out from the second partial image (step S310).

Next, the image processing device 101 generates training data to be provided to the super-resolution network 104, in which the first partial image is used as the training input and the second partial image is used as the training output (step S311).

After performing this iterative process for each of the divided regions (step S312), the image processing device 101 removes training data whose similarity is an outlier based on the distribution of similarities between training inputs and training outputs in the generated training data (step S313), and then provides the remaining training data to the super-resolution network 104 to proceed with learning (step S314), thereby terminating this process.

In addition, if multiple pairs of first training images and second training images are prepared, the above process may be repeated for each pair, or after generating training data for all pairs, the data may be provided to the super-resolution network 104 all at once to train the super-resolution network 104.

In addition, in step S307, candidate positions for the second region relative to the first region are determined. At the beginning of the repetition, the value of n may be set to a value that covers a wide range of the second training image or the entire image, and after the repetition has progressed to a certain extent, candidate positions for the current first region may be determined based on the correspondence between the positions of the first and second regions previously identified, and the value of n may be reduced.

(Example of the first method for finding a mapping)
Below, we will explain in more detail an example of a first technique for finding a mapping that aligns a first training image with a second training image through two steps: a step for finding a scale mapping and a step for finding a transform mapping.

In this method, as described above, the resolution of the first training image and the second training image are matched by scale mapping, and then the image processing device 101 performs feature point detection and feature point matching. Any method can be applied to feature point detection and feature point matching, from classical methods such as SIFT and AKAZE to various methods such as those disclosed in Non-Patent Document 1.

Then,
(a) a first feature point detected in a first training image;
(b) second feature points detected in the second training image, the second feature points corresponding to the first feature points;
A plurality of pairs consisting of the above are extracted.

The desired mapping minimizes the difference between the projections of the first feature points of the multiple pairs obtained here and the second feature points of the multiple pairs. Therefore, the desired mapping can be obtained by solving the minimization problem.

Therefore, the image processing device divides the first training image into multiple polygons with feature points detected by feature point detection as vertices. Most simply, triangles can be used as polygons and Delaunay division can be used as division.

Next, consider several candidate mappings, each with a domain of several polygons.

Each candidate mapping of the plurality of candidate mappings is
(a) projecting a first polygon that is the domain of each candidate mapping onto a second polygon in a second training image;
(b) A mapping that projects the vertices of a first polygon onto the vertices of a second polygon that correspond to the vertices of the first polygon through feature point matching, and can be defined most simply by inversion, translation, rotation, scaling, shearing, trapezoidal transformation, or a combination of these.

Here, inversion, translation, rotation, scaling, shearing, and combinations thereof can be expressed as affine transformations, and trapezoidal transformations and combinations of the above can be expressed as projective transformations (homography transformations). Any of these transformations can be specified by a transformation matrix.

In an ideal world where the samples for the first training image and the second training image are both undeformed, the transformation matrices representing each candidate mapping obtained here should be identical, but in reality, effects such as deformation occur when the samples are created.

Therefore, clustering is performed to eliminate inappropriate candidates from among the multiple candidate mappings. As mentioned above, each candidate mapping is represented by a transformation matrix, so the difference between the transformation matrices is used as the distance to perform clustering and detect outliers. For example, using k-nearest neighbor (k-NN) or Local Outlier Factor (LOF), clusters are distinguished into clusters to which outliers belong, i.e., minority clusters, and clusters to which major values belong, i.e., majority clusters.

Candidate images belonging to the minority cluster, i.e., minority mappings, are considered to have incorrect matching between the domain area and the range area, or severe deformation of the domain area or range area.

Therefore, we identify the majority mappings that belong to the majority cluster and adopt the direct sum of the majority mappings as the target mapping.

After obtaining the mapping in this way, if the first partial image is not included in the domain of the mapping, or if the second partial image is not included in the range of the mapping, then this set will not be used as training data.

This type of processing allows us to obtain appropriate training data that removes the effects of incorrect feature point matching or deformations in parts of the sample.

(An example of the second method for finding the mapping)
As described above, the second method for obtaining a mapping uses image registration techniques such as a combination of feature extraction and a convolutional neural network, and reinforcement learning and deep learning that obtain multiple homography mappings and diffeomorphisms. These techniques can be applied even when the resolutions of the two images do not match, but the desired mapping can also be obtained through a two-step process in which the resolutions are matched using the scale mapping described above, and then registration is performed to obtain a transform mapping.

In medical image alignment such as in this embodiment, the desired mapping cannot be simply described by a single homography matrix, so it is necessary to find the direct sum of multiple homography mappings or a diffeomorphism expressed by a displacement vector field.

Such technologies include a robust registration technique using agent-based action learning by Julian Krebs et al., the DIRNET technique by Bob D. de Vos et al., and the Quicksilver technique by Xiao Yang et al., and these technologies can be applied to this embodiment.

(Image Processing System)
The training of the super-resolution network 104 and the estimation of images by the super-resolution network 104 can be performed on a server computer in which various medical images based on electronic medical records, etc. are collected. In other words, the server computer can function as the image processing device 101 described above.

In such an embodiment, an image processing system can be used in which a computer terminal provided at a hospital or the like serves as an input/output interface for actual diagnosis. Figure 5 is an explanatory diagram showing the general configuration of an image processing system according to an embodiment of the present invention. Figure 6 is an explanatory diagram showing an example of a display on the screen of a terminal computer according to an embodiment of the present invention. Figure 7 is an explanatory diagram showing an example of a display on the screen of a terminal computer according to an embodiment of the present invention. The following explanation will be given with reference to these figures.

The image processing system 201 comprises a server computer 202 that realizes the image processing device 101, a terminal computer 203, and a computer communication network 204 that connects the two so that they can communicate with each other.

Here, the terminal computer 203 has a first display unit 211, a reception unit 212, a transmission/reception unit 213, and a second display unit 214.

The first display unit 211 displays a group of images of multiple cross sections that appear when the target specimen is cut into layers in the first area 411 on the screen 401.

In this figure, each image 412 in the image group is projected obliquely and drawn side by side. The original shape of the edges of each image 412 is rectangular or square, and when projected obliquely, it becomes a parallelogram, but the edges are not shown in this figure.

In other words, the first display unit 211 arranges the multiple images included in the image group on multiple parallel planes set in a virtual three-dimensional space in the order in which the multiple cross sections for the multiple images are arranged in the target specimen, and depicts the appearance of the virtual three-dimensional space in the first area 411 by parallel (oblique) projection or perspective projection, thereby creating a pseudo-three-dimensional feeling.

On the other hand, the reception unit 212 receives an image selection instruction to select one of the images from the displayed group of images.

Image selection is performed by selecting the desired image using a mouse, keyboard, etc. In this figure, the edge of image 412 (which, as mentioned above, is originally a square or rectangle, but has been deformed into a parallelogram due to perspective) is highlighted with dotted line 413 to indicate that the image has been selected.

In the first area 411, by scrolling using the scroll bar 415, it is possible to sequentially view obliquely a large number of images that do not fit in the first area 411. In this case, the image located in the center of the first area 411 may be selected simply by scrolling.

Furthermore, the selected image is displayed in the second area 422 in its original, normal, rather than oblique, form. That is, in Figure 6, the portion of the first area 411 surrounded by dotted line 413 corresponds to an oblique projection of the image displayed in the second area 422.

For example, when an observer such as a doctor selects images in sequence in the first area 411, a three-dimensional image of the target specimen is displayed as an animation in the second area 422.

In addition, the image displayed in the second area 422 can be enlarged or reduced, and a portion of the image selected in the first area 411 can be displayed in the second area 422. When only a portion of the image selected by zooming in or the like is displayed in the second area 422, the area corresponding to that portion can be shown to the viewer by surrounding it with a thick line 414 in the first area 411, as shown in Figure 7. Note that when the entire image is displayed in the second area 422, the thick line 414 is not drawn, as shown in Figure 6.

The viewer can zoom in or out of the image using the zoom-in button 425 and zoom-out button 426 located near the second area 422, and can move the position displayed within the second area 422 using the scroll bars 427 and 428 located on the edges of the second area 422.

Furthermore, within the second area 422, it is possible to specify an area that one wishes to observe in detail. That is, the reception unit 212 can receive an area selection instruction to select an area from the selected image based on operation of a mouse, keyboard, etc.

In Figures 6 and 7, the area of the second area 422 surrounded by the thick frame 423 is the selected area. Of the selected images, the image within this area becomes the input image to be processed. For example, the position of the thick frame 423 can be changed by specifying and dragging within the thick frame 423. Furthermore, the size of the thick frame 423 may be changed by dragging the vertices or edges of the thick frame 423.

It is also possible to cancel the area selection instruction. In this case, the entire selected image will become the input image.

The selected input image is transmitted from the terminal computer 203 to the server computer 202 via the computer communication network 204, triggered by an explicit instruction from the observer or after a certain period of time has passed since the image and area selection, and is input to the image processing device 101 as an input image.

The output image estimated by the image processing device 101 is then transmitted from the server computer 202 to the terminal computer 203 via the computer communication network 204.

Here, the transmission and reception of input images and output images is handled by the transmission and reception unit 213 in the terminal computer.

The second display unit 214 displays the output image estimated by the image processing device 101 in the third area 433 on the screen 401.

Let us consider a case where the image group to be estimated is color photographic images of stained specimens taken with an optical microscope, and the estimation results are the results of photographs taken with an electron microscope. Generally, the results of photographs taken with an electron microscope are grayscale images.

Therefore, in order to easily grasp the degree to which the magnification ratio of the output image displayed in the third area 433 differs from the magnification ratio of the input image displayed in the second area 422, i.e., the degree to which the output image displayed in the third area 433 is estimated from the input image actually captured, the output image displayed in the third area 433 can be colored.

For example, in the second region 422, the brightness, hue, and saturation of the pixel value displayed for each pixel are respectively set as follows:
the brightness of the pixel in the output image corresponding to each pixel;
the hue of the pixel in the input image corresponding to each pixel;
By setting the saturation to a value determined according to the magnification ratio of the output image for the cross section of the target specimen, it becomes possible to easily grasp the magnification ratio of the output image.

In this way, when the image processing system 201 is applied to medical institutions and research institutions, medical images (photographic images taken with optical microscopes or electron microscopes) provided by each institution are stored on the server computer 202, and medical images of the same target specimen (opposite cross sections) are used as training data for the super-resolution network 104, enabling better estimation.

In addition, each institution can select medical images of the target specimen taken with an optical microscope on the terminal computer 203, and if necessary, select desired areas within the images, thereby obtaining an estimate of how the target specimen will appear when observed with an electron microscope, which can be useful for diagnosis and research.

(Experimental example)
An experimental example in which the above embodiment is applied to converting a pathological image captured by an optical microscope into an electron microscope image captured by an electron microscope will be described below.

First, the pathology images and electron microscope images used as training data are scale mapped to match their resolution, and then AKAZE features are detected for the entire image. A transform map is calculated to match each position on the entire electron microscope image, and the entire pathology image is corrected.

Furthermore, the entire corrected pathology image is slid by 32 pixels to extract a 256x256 pixel pathology tile image.

On the other hand, for electron microscope images, a scanning range of 1024 x 1024 pixels ((1024-256)/2 = 384 pixels up, down, left, and right) is used, and the image with the highest similarity is selected.

For the super-resolution network, we use SR3, a technology disclosed in non-patent document 2.

Figure 8A is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in grayscale. Figure 8B is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in binary monochrome. Figure 9A is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in grayscale. Figure 9B is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in binary monochrome. Figure 10A is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in grayscale. Figure 10B is a photograph in lieu of a drawing showing a photographed pathology image, an electron microscope image estimated from the pathology image, and the photographed electron microscope image arranged vertically in binary monochrome. In these figures, the representative photographs of drawings with drawing numbers ending in B are monochrome binarized versions of the representative photographs of drawings with the same drawing numbers and ending in A. For example, the drawing substitute photograph of Figure 8B is the drawing substitute photograph of Figure 8A that has been monochrome binarized.

In these figures,
A pathological image of one cross section of the target specimen taken with an optical microscope at 400x magnification,
an electron microscope image estimated from the pathological image according to this embodiment; and
a power image of the other cross section of the target specimen taken with an electron microscope at a magnification of 1000 times, the power image being at a position corresponding to the pathological image;
are shown side by side, enlarged and reduced for comparison.

As can be seen from these figures, the electron microscope images estimated from pathology images have higher resolution than the pathology images, but there is little difference between them and the actual electron microscope images taken, indicating that the estimation is of a quality that can be useful for doctors' diagnoses and researchers' research.

In this way, in this embodiment, a pathology image is used as an input image, an electron microscope image is estimated from the input image, and the electron microscope image is output as an output image. The output image is merely reference information and is intended to be used for diagnostic support. Researchers are expected to be the primary users of the output image, but clinicians can also use it.

For example, kidney disease is difficult to treat, and once dialysis is required, the condition persists for life, making it necessary to begin treatment early. In this embodiment, electron microscope images are generated from pathological photographs of glomeruli taken by the Renal Biomedical Laboratory, providing reference information for early diagnosis and treatment. The Renal Biomedical Laboratory itself is often handled by an internist, but may also be handled by a pathologist.

Furthermore, intestinal abnormalities caused by cancer, genes mutate and the structure of the intestinal tract changes. This makes early diagnosis using electron microscope images necessary, and the electron microscope images generated by this embodiment can be used as reference material for this.

At the Institute of Cardiology, electron microscope images can be generated from pathological images, allowing the state of the myocardium and fibers to be viewed as reference information. In this case, pathologists are often in charge.

Amyloidosis, which affects people all over the world, can be found in any organ other than the intestine, and is classified as a primary disease of unknown cause that is endemic, and as a secondary disease caused by decreased renal function, which also affects dialysis patients. The electron microscope images obtained using this embodiment can be used as reference information when observing the condition of these affected areas.

Generally, electron microscopy is expensive. This is because the methods of fixing specimens differ between pathology and electron microscopy, and the number of photographers skilled in electron microscopy is decreasing.

The electron microscope images output by this embodiment are generated from pathology images, so their cost is low. Therefore, by additionally using these electron microscope images as diagnostic support and reference information, they can be useful for low-cost early diagnosis and early treatment.

(Quantitative evaluation of output images)
In the above embodiment, an electron microscope image is estimated from a pathological image and used as an output image. In this embodiment, the plausibility and reliability of this output image are quantitatively evaluated, and the evaluation value is provided as reference information to users of the output image.

Figure 11 is an explanatory diagram showing the general configuration of an image processing device according to an embodiment of the present invention. The image processing device 101 shown in this figure is the image processing device 101 of the above embodiment, to which a generation unit 502 and an evaluation unit 503 have been added. The following description will be given with reference to this figure.

First, the generation unit 502 provides the output image output from the super-resolution unit 103 to the trained generation network 505 to generate a reference image.

The generative network 505 is a network that removes information from a given input, such as by reducing the dimensionality of the given input or adding noise to the given input, and then obtains an output that matches the original input as closely as possible.

The simplest generative network 505 is an autoencoder. Various autoencoders can be used, such as a stacked autoencoder, a convolutional autoencoder (CAE), a variational autoencoder (VAE), and a conditional variational autoencoder (CVAE).

Fig. 12 is an explanatory diagram showing the configuration of an encoder in a VAE, which is an example of a generative network in an image processing device according to an embodiment of the present invention. Fig. 13 is an explanatory diagram showing the configuration of a decoder in a VAE, which is an example of a generative network in an image processing device according to an embodiment of the present invention. A relatively simple VAE network as shown in this figure can be used as generative network 505.
Fig. 14 is an explanatory diagram showing the configuration of an encoder in a VAE, which is another example of a generative network in an image processing device according to an embodiment of the present invention. Fig. 15 is an explanatory diagram showing the configuration of a decoder in a VAE, which is another example of a generative network in an image processing device according to an embodiment of the present invention. A network using a VAE with another configuration as shown in this figure can also be used as generative network 505.
Also,
Neural network based on the diffusion model,
Generative Adversarial Network (GAN),
A neural network based on a flow-based generative model (Flow-based Generative Network),
Various networks can also be used as the generative network 505, such as neural networks that implement dimensionality reduction and restoration based on the Transformer.

It is desirable that the generative network 505 proceeds with learning using other training samples that are different from the training samples used to train the super-resolution network 104. The generative network 505 proceeds with learning using other training images obtained by photographing other training cross-sections that appear by cutting other training samples with the second wave as training inputs and training outputs.

To explain this in accordance with the above application example, in a mode in which an electron microscope image is estimated from a pathological image taken with an optical microscope and used as the output image, other electron microscope images of the same affected area are prepared as training images, and the generative network 505 is trained based on these electron microscope images.

The generative network 505 is then expected to, when given an image as input, output an image that closely represents the features specific to electron microscope images.

Therefore, if the output image output by the super-resolution unit 103 is a plausible image that closely resembles an electron microscope image, the difference between it and the reference image generated by the generation unit 502 will be small, and if the output image output by the super-resolution unit 103 is different from an electron microscope image (for example, an image that contains abnormal information), the difference between the output image and the reference image will be large.

Therefore, the evaluation unit 503 quantitatively evaluates the output image numerically based on the difference between the output image and the reference image, and outputs the result.

The parameter values used for quantitative evaluation may include the following:
The number of pixels that differ between the output image and the reference image, and the ratio of the number of pixels that differ to the total number of pixels.
The sum of the different pixel values between the output image and the reference image, or the average of the different pixel values.
The Mahalanobis distance between the output image and the reference image.
Cosine similarity between the output image and the reference image.
The similarity between the output image and the reference image calculated using machine learning and deep learning technologies such as AugNet.
The deviation in the distribution of any of the above parameter values, a variance used in autoregressive models and One Class SVM outlier detection.

Figure 16 is a photograph in place of a drawing, showing a captured pathology image, an electron microscope image estimated from the pathology image, and a generated reference image, all arranged in grayscale. Figure 17 is a photograph in place of a drawing, showing a captured pathology image, an electron microscope image estimated from the pathology image, and a generated reference image, all arranged in monochrome binary. In these figures, for example (a) and example (b), the electron microscope image in the center is estimated from the pathology image on the left, and a reference image is generated from the estimated electron microscope image using the VAE disclosed in Figures 12 and 13. In example (a), a plausible electron microscope image is estimated, but in example (b), abnormal shapes (arrows, squares, and circles) are drawn, indicating that the generation of the electron microscope image failed.

When the above VAE is used to generate a reference image, in both example (a) and example (b), the generated image appears different from the estimated electron microscope image.

However, when the difference between the electron microscope image and the reference image is calculated by summing the pixel value differences, the difference is 2,397,066 in example (a) and 3,871,511 in example (b), meaning that the difference is smaller in example (a).

In addition, the similarity between the electron microscope image and the reference image calculated by AugNet is 28.441 for example (a) and 34.211 for example (b).The value is smaller for example (a), and according to AugNet's definition of similarity, the smaller the value, the more similar the images are, so example (a) has a smaller difference.

According to the inventor's experiment, for 199 electron microscope images,
The percentage of pixels with a sum of pixel value differences of 1.5 million or less is 100%.
90% are under 2 million yen;
56% have incomes of 2.5 million yen or less;
27% are below 3 million yen;
7% for those under 3.5 million yen;
0% for items under 4 million yen
This is what happened.

Similarly,
The percentage of similarities below 25 is 100%.
77% were below 27.5;
57% are under 30;
38% had a score of 32.5 or less;
36% were under 35;
18% had a score of 37.5 or less;
0% for those under 40;
This is what happened.

In addition, when a reference image was generated for a similar electron microscope image using the VAE disclosed in Figures 14 and 15, the difference and similarity in example (a) were 2763465 and 28.517, and the difference and similarity in example (b) were 3342139 and 29.369. In addition, for 199 electron microscope images, the sum of the pixel value differences was
The percentage of items below 1.9 million yen is 100%.
68% have incomes of 2.3 million yen or less;
43% have incomes of 2.7 million yen or less;
41% have incomes of 3.1 million yen or less;
18% are below 3.5 million yen;
0% for items under 4 million yen
and the similarity is
The percentage of those under 16 is 92%.
68% were under 20;
50% for those under 24;
43% were 28 or younger;
42% are under 32;
0% for those under 36
This is what happened.

The percentages in these experimental examples indicate the percentage of electron microscope images in which the super-resolution unit 103 estimated them inappropriately and were of low quality. Therefore, in general, if you want to consider only high-quality output images, you can reduce the sum of pixel value differences or the AugNet similarity threshold and select images with parameter values smaller than that threshold.

In addition, by calculating the percentage of appropriate images when the difference value parameter for the estimated electron microscope image is used as a threshold, a numerical value representing the likelihood of the estimated electron microscope image can be obtained.

In this way, based on the difference between the estimated electron microscope image and the reference image generated from it, quantitative evaluation results can be obtained that represent the likelihood and quality of the electron microscope image.

Therefore, by providing a pathological image as an input image to the image processing device 101, researchers, doctors, etc. can obtain an output image from the image processing device 101 and a quantitative evaluation indicating how likely the output image is as an electron microscope image, allowing them to decide whether or not to use the output image as reference information for diagnostic support.

The above explanation describes a method for estimating an output image in which the resolution of an electron microscope and the electron beam are used as the second resolution and second wave from an input image captured using the first resolution, the resolution of an optical microscope, and light as the first resolution and first wave. However, the resolution and type of wave are not limited to these and can be applied to various other methods, and these methods are also within the scope of the present invention.

Furthermore, in a method in which an estimated output image is given to a generative network trained using images captured at a second resolution and a second wave to obtain a reference image, and the output image is quantitatively evaluated based on the difference between the output image and the reference image, the specific generative network and evaluation method are not limited to the above-mentioned aspects, and various outlier detection and anomaly detection technologies can be applied, and these aspects are also included within the scope of the present invention.

(summary)
As described above, the image processing apparatus according to this embodiment:
an input unit to which an input image of a target object captured with a first wave having a first resolution is input;
a super-resolution unit that estimates an output image that should be obtained by photographing the target specimen with a second wave having a second resolution by providing the input image to a trained super-resolution network;
The second resolution is configured to be higher than the first resolution.

In addition, in the image processing device according to this embodiment,
The second waveband of the second vibration may be configured to be shorter than the first waveband of the first vibration.

In addition, in the image processing device according to this embodiment,
the first wave is light,
The second wave may be configured to be an electron beam.

In addition, in the image processing device according to this embodiment,
the first wave is unpolarized light;
The second wave may be configured to be polarized light.

In addition, in the image processing device according to this embodiment,
The input image may be an image of a cross section of the target specimen.

In addition, in the image processing device according to this embodiment,
The super-resolution network
a training input based on a first training image obtained by photographing a first training cross section that appears by cutting a training specimen with the first wave;
a training output based on a second training image obtained by photographing a second training cross section that appears opposite to the first training cross section with the second wave;
The method can be configured to be trained using training data including

In addition, in the image processing device according to this embodiment,
The image processing device includes:
correcting the first training images with a mapping that aligns the first training images with the second training images;
a first region of the corrected first training image is cut out as a first partial image;
Scanning a predetermined scanning range from the second region using a window having the same shape as the second region projected onto the second training image by the mapping of the first region, and selecting an image within the scanned window that has the highest similarity to the first partial image as a second partial image;
The super-resolution network can be configured to be trained by using the first partial image as the training input and the second partial image as the training output.

In addition, in the image processing device according to this embodiment,
In the alignment,
By matching the resolution of the first training image and the second training image and then performing feature point detection and feature point matching, a plurality of pairs are obtained, each of the plurality of pairs being:
first feature points detected in the first training image;
second feature points detected in the second training image, the second feature points corresponding to the first feature points;
Extracting a plurality of pairs consisting of
The mapping is
a projection of each of the pairs of first feature points by the mapping;
the plurality of pairs of second feature points;
It can be constructed such that is a mapping that minimizes the difference between .

In addition, in the image processing device according to this embodiment,
In the alignment,
Dividing the first training image into a plurality of polygons having vertices that correspond to the feature points detected in the first training image by the feature point detection;
A plurality of candidate mappings each having a domain of the plurality of polygons, wherein each candidate mapping of the plurality of candidate mappings comprises:
projecting a first polygon that is the domain of each candidate mapping onto a second polygon in the second training image;
determining a plurality of candidate mappings for projecting the vertices of the first polygon onto the vertices of the second polygon that are respectively associated with the vertices of the first polygon by the feature point matching;
clustering the plurality of candidate mappings to identify minority mappings belonging to a minority cluster and majority mappings other than the minority cluster;
Let the direct sum of the majority maps be the map,
the first sub-image is included in the domain of the mapping;
The second partial image can be configured to be included in the range of the mapping.

In addition, in the image processing device according to this embodiment,
the polygon is a triangle,
the partitioning is a Delaunay partitioning,
The plurality of candidate mappings may be configured to be inversions, translations, rotations, scalings, shears, trapezoidal transformations, or combinations thereof.

In addition, in the image processing device according to this embodiment,
The mapping can be configured to be a direct sum or diffeomorphism of multiple homography mappings learned by reinforcement learning or deep learning.

In addition, in the image processing device according to this embodiment,
a generating unit that generates a reference image by providing the output image to a generating network that has been trained by using other training images, which are obtained by cutting other training cross sections that appear by cutting other training specimens and are photographed with the second wave, as training inputs and training outputs;
The image processing device may further include an evaluation unit that quantitatively evaluates the output image based on the difference between the output image and the reference image.

In addition, in the image processing device of this embodiment,
The generator network comprises:
Autoencoders, including stacked autoencoders, convolutional autoencoders (CAEs), variational autoencoders (VAEs), and conditional variational autoencoders (CVAEs).
Neural network based on the diffusion model,
Generative Adversarial Network (GAN),
A neural network based on a flow-based generative model (Flow-based Generative Network),
It can be configured to be either a Transformer-based neural network.

In addition, in the image processing device of this embodiment,
The evaluation unit
outlier detection based on the number of different pixels or the distribution of different pixel values between the output image and the reference image (including outlier detection based on an autoregressive model);
Outlier detection based on One Class SVM of the difference;
outlier detection based on a Mahalanobis distance between the output image and the reference image;
the cosine similarity between the output image and the reference image;
The output image may be quantitatively evaluated by either an AugNet-based similarity between the output image and the reference image.

The image processing system according to this embodiment includes a terminal and the image processing device described above.
The terminal
a first display unit that displays a plurality of images in a first area within the screen;
a reception unit that receives an image selection instruction for selecting one of the displayed images;
a transmitting/receiving unit that transmits the selected image to the image processing device as the input image and receives the output image estimated by the image processing device from the image processing device;
The device may be configured to include a second display unit that displays the received output image in a second area within the screen.

In addition, in the image processing system according to this embodiment,
The plurality of images may be configured as a group of images obtained by photographing a plurality of cross sections that appear when the target specimen is cut into layers.

In addition, in the image processing system according to this embodiment,
When the receiving unit receives an area selection instruction to select any area from the selected image, the second display unit can be configured to use the image within the selected area of the selected image as the input image, and display the output image obtained for the input image in the second area on the screen.

In addition, in the image processing system according to this embodiment,
the set of images are color images,
the output image is a grayscale image;
The lightness, hue, and saturation of the pixel value displayed in each pixel of the second region are respectively:
the brightness of the pixel of the output image corresponding to each pixel;
the hue of the pixel of the input image corresponding to each pixel;
The saturation may be set to a value determined according to the magnification ratio of the output image for the cross section of the target specimen.

In addition, in the image processing system according to this embodiment,
The first display unit is
arranging the images included in the image group on a plurality of planes parallel to each other set in a virtual three-dimensional space in the order in which a plurality of cross sections corresponding to the images are arranged in the target specimen;
The image group can be configured to be displayed in the first area by drawing the state of the virtual three-dimensional space in the first area.

The image processing method according to this embodiment includes:
an input step of inputting an input image of a target specimen captured with a first wave having a first resolution;
a super-resolution step of estimating an output image that should be obtained by photographing the target specimen with a second wave having a second resolution by providing the input image to a trained super-resolution network;
The second resolution is configured to be higher than the first resolution.

The program according to this embodiment executes the following steps:
an input unit to which an input image of a target object captured with a first wave having a first resolution is input;
providing the input image to the trained super-resolution network, thereby causing the network to function as a super-resolution unit that estimates an output image that should be obtained by photographing the target specimen with a second wave having a second resolution;
The second resolution is configured to be higher than the first resolution.

The program according to this embodiment can be recorded on a non-transitory computer-readable information recording medium and distributed or sold. It can also be distributed or sold via a temporary transmission medium such as a computer communication network.

The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of the present invention.
This application claims priority based on patent application No. 2022-192062, filed in Japan on Wednesday, November 30, 2022, and the contents of that basic application are incorporated into this application to the extent permitted by the laws and regulations of the designated countries.

The present invention provides an image processing device, image processing system, image processing method, program, and information recording medium that perform estimation to improve the resolution of an input image of a specimen and obtain an output image.

101 Image processing device
102 Input section
103 Super-resolution unit
104 Super-resolution Network
201 Image Processing System
202 Server Computer
203 Terminal Computer
204 Computer Communication Network
211 1st display section
212 Reception Department
213 Transmitter/Receiver
214 2nd display section
401 screen
411 First area
Each image in the 412 image set
413 Dotted line representing selected image
414 Thick lines representing the part displayed in the second area
415 Scrollbar
422 Second area
423 Bold border representing selected area
425 Zoom in button
426 Zoom out button
427 Scrollbar
428 Scrollbar
433 Third area
502 Generation part
503 Evaluation Department
505 Generative Network

Claims (18)

an input unit to which an input image of a cross section of a target specimen captured with a first wave having a first resolution is input;
a super-resolution unit that estimates an output image that should be obtained by photographing the cross section of the target specimen with a second wave having a second resolution by providing the input image to a trained super-resolution network;
the second resolution is higher than the first resolution;
The second waveband of the second vibration is shorter than the first waveband of the first vibration.
1. An image processing device comprising: the first wave is light,
2. The image processing device according to claim 1 , wherein the second wave is an electron beam. the first wave is unpolarized light;
2. The image processing device according to claim 1, wherein the second wave is polarized light. The super-resolution network
a training input based on a first training image obtained by photographing a first training cross section that appears by cutting a training specimen with the first wave;
a training output based on a second training image obtained by photographing a second training cross section that appears opposite to the first training cross section with the second wave;
2. The image processing device according to claim 1, wherein the image processing device is trained using training data including: The image processing device includes:
correcting the first training images with a mapping that aligns the first training images with the second training images;
a first region of the corrected first training image is cut out as a first partial image;
Scanning a predetermined scanning range from the second region using a window having the same shape as the second region projected onto the second training image by the mapping of the first region, and selecting an image within the scanned window that has the highest similarity to the first partial image as a second partial image;
5. The image processing device according to claim 4 , wherein the super-resolution network is trained by using the first partial image as the training input and the second partial image as the training output. 6. The image processing device according to claim 5 , wherein the mapping is a direct sum or a diffeomorphism of a plurality of homography mappings learned by reinforcement learning or deep learning. a generating unit that generates a reference image by providing the output image to a generating network that has been trained by using other training images, which are obtained by cutting other training cross sections that appear by cutting other training specimens and are photographed with the second wave, as training inputs and training outputs;
2. The image processing device according to claim 1, further comprising an evaluation unit that quantitatively evaluates the output image based on a difference between the output image and the reference image. The generator network comprises:
Autoencoders, including stacked autoencoders, convolutional autoencoders (CAEs), variational autoencoders (VAEs), and conditional variational autoencoders (CVAEs).
Neural network based on the diffusion model,
Generative Adversarial Network (GAN),
A neural network based on a flow-based generative model (Flow-based Generative Network),
8. The image processing device according to claim 7 , wherein the image processing device is a neural network based on a Transformer. The evaluation unit
outlier detection based on the number of different pixels or the distribution of different pixel values between the output image and the reference image (including outlier detection based on an autoregressive model);
Outlier detection based on One Class SVM of the difference;
outlier detection based on a Mahalanobis distance between the output image and the reference image;
the cosine similarity between the output image and the reference image;
8. The image processing device according to claim 7 , wherein the output image is quantitatively evaluated using any one of AugNet-based similarities between the output image and the reference image. An image processing system including a terminal and an image processing device,
The image processing device includes:
an input unit to which an input image of a target object captured with a first wave having a first resolution is input;
a super-resolution unit that estimates an output image that should be obtained by photographing the target specimen with a second wave having a second resolution by providing the input image to a trained super-resolution network;
the second resolution is higher than the first resolution;
a second waveband of the second vibration is shorter than a first waveband of the first vibration;
The terminal
a first display unit that displays a plurality of images in a first area within the screen;
a reception unit that receives an image selection instruction for selecting one of the displayed images;
a transmitting/receiving unit that transmits the selected image to the image processing device as the input image and receives the output image estimated by the image processing device from the image processing device;
an image processing system comprising a second display unit that displays the received output image in a second area within the screen. 11. The image processing system according to claim 10 , wherein an image of a cross section of the target specimen is used as the input image. 12. The image processing system according to claim 11 , wherein the plurality of images are a group of images obtained by photographing a plurality of cross sections that appear when the target specimen is cut into layers. 13. The image processing system according to claim 12, wherein when the receiving unit receives an area selection instruction to select any area from the selected image, the second display unit uses an image within the selected area of the selected image as the input image, and displays the output image obtained for the input image in the second area on the screen . the set of images are color images,
the output image is a grayscale image;
The lightness, hue, and saturation of the pixel value displayed in each pixel of the second region are respectively:
the brightness of the pixel of the output image corresponding to each pixel;
the hue of the pixel of the input image corresponding to each pixel;
14. The image processing system according to claim 13 , wherein the saturation is set to a value determined according to a magnification ratio of the output image for the cross section of the target specimen. The first display unit is
arranging the images included in the image group on a plurality of planes parallel to each other set in a virtual three-dimensional space in the order in which a plurality of cross sections corresponding to the images are arranged in the target specimen;
13. The image processing system according to claim 12 , wherein the group of images is displayed in the first area by drawing the state of the virtual three-dimensional space in the first area. an input step of inputting an input image of a cross section of a target specimen captured with a first wave having a first resolution;
a super-resolution step of estimating an output image that should be obtained by photographing the cross section of the target specimen with a second wave having a second resolution by providing the input image to a trained super-resolution network;
the second resolution is higher than the first resolution;
The second waveband of the second vibration is shorter than the first waveband of the first vibration.
An image processing method comprising: Computer,
an input unit to which an input image of a cross section of a target specimen captured with a first wave having a first resolution is input;
providing the input image to the trained super-resolution network, causing it to function as a super-resolution unit that estimates an output image that should be obtained by photographing the cross section of the target specimen with a second wave having a second resolution;
the second resolution is higher than the first resolution;
The second waveband of the second vibration is shorter than the first waveband of the first vibration.
A program characterized by: A non-transitory computer-readable information recording medium on which the program according to claim 17 is recorded.