JP6901510B2 - Observation method, image processing device, and electron microscope - Google Patents

Description

The present invention relates to an observation method, an image processing apparatus, and an electron microscope.

A method is known that enables observation of the same location between an optical microscope and an electron microscope (see, for example, Patent Document 1).

In the observation with an optical microscope, the localization of a target cell or a target protein can be observed by labeling a specific cell or a specific protein with a fluorescent protein or a fluorescent dye. Examples of such a fluorescent protein include GFP (green fluorescent protein). Moreover, as such a fluorescent dye, FITC (Fluorescein isothiocyanate), DAPI (4', 6-diamidino-2-phenylindole) and the like can be mentioned.

However, these fluorescent proteins and fluorescent dyes are incompatible with the preparation of samples for electron microscopy. For example, glutaraldehyde or osmium tetroxide is used for fixing the sample. Fluorescent proteins and fluorescent dyes lose their fluorescence due to cross-linking with glutaraldehyde and oxidation with osmium tetroxide. Therefore, it is difficult to perform optical microscope observation of a sample labeled with a fluorescent protein or a fluorescent dye after fixing the sample.

Further, for example, a method is known in which a sample labeled with a fluorescent protein that undergoes a DAB (3,3'-diaminobenzidine) reaction is observed with an optical microscope, and then the sample is fixed and observed with an electron microscope. Examples of such a fluorescent protein include ascorbic acid oxidase (APEX) and miniSOG (mini Singlet Oxygen Generator).

However, with this method, it is not possible to observe the sample after fixing the sample with an optical microscope. Further, in the preparation of an electron microscope sample, after the sample is fixed, the intracellular water is replaced with an organic solvent (dehydration), the organic solvent is replaced with a resin (resin embedding), and the resin is polymerized. Dehydration causes shrinkage of the sample, and polymerization of the resin causes shrinkage of the sample due to shrinkage of the resin. Therefore, it becomes difficult to align the optical microscope image and the electron microscope image.

Further, as a fluorescent dye that is not affected by cross-linking or oxidation, there are quantum dots made of a semiconductor material. However, when performing transmission electron microscopy, the sample must be sliced. When the sample is sliced, the amount of the fluorescent dye is reduced, so that a sufficient fluorescent signal cannot be obtained by observation with an optical microscope.

Japanese Unexamined Patent Publication No. 8-162059

As mentioned above, it was difficult to observe a sample in the same state with an optical microscope and an electron microscope.

(1) One aspect of the observation method according to the present invention is
A process of preparing a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light as markers, and a process of preparing a sample.
The process of acquiring an optical microscope image by photographing the sample with an optical microscope, and
The process of acquiring an electron microscope image by photographing the sample with an electron microscope, and
In the optical microscope image, a step of acquiring information on the positions of a plurality of the metal particles and the color of scattered light, and
A step of acquiring information on the positions and particle sizes of a plurality of the metal particles in the electron microscope image, and
Based on the position and scattered light color information of the plurality of metal particles acquired by the optical microscope image and the position and particle size information of the plurality of metal particles acquired by the electron microscope image. A step of obtaining information for associating the optical microscope image with the electron microscope image, and
Based on the information that associates the optical microscope image with the electron microscope image, the optical microscope image and the electron microscope image are aligned to generate one image in which the optical microscope image and the electron microscope image are superimposed. Process and
Only including,
In the step of obtaining information for associating the optical microscope image with the electron microscope image,
Based on the relationship between the particle size of the metal particles and the color of the scattered light of the metal particles, the particle size of the metal particles is estimated from the color of the scattered light of the metal particles in the optical microscope image.
The optical microscope image and the electron microscope are compared with the estimated particle size and position information of the metal particles in the optical microscope image and the particle size and position information of the metal particles in the electron microscope image. Find the corresponding metal particle with the image and
Based on the corresponding metal particles, the difference in magnification between the optical microscope image and the electron microscope image and the difference in the orientation of the optical microscope image and the electron microscope image were obtained.
In the step of generating one image,
Based on the obtained difference in magnification between the optical microscope image and the electron microscope image, the magnifications of the optical microscope image and the electron microscope image are made equal.
The orientations of the optical microscope image and the electron microscope image are aligned based on the obtained difference in orientation between the optical microscope image and the electron microscope image .

In such an observation method, a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light are used as markers. The plurality of metal particles function as markers in both the optical microscope image and the electron microscope image without being affected by the preparation of the electron microscope sample. Therefore, in such an observation method, a sample in the same state can be observed with an optical microscope and an electron microscope. Further, in such an observation method, the particle size of the metal particles can be estimated from the color of the bright spot in the optical microscope image. Therefore, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

(2) One aspect of the image processing apparatus according to the present invention is
An image acquisition unit that acquires an optical microscope image of a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light and an electron microscope image of the sample as markers.
In the optical microscope image, a first image information acquisition unit that acquires information on the positions of a plurality of the metal particles and the color of scattered light, and
In the electron microscope image, a second image information acquisition unit that acquires information on the positions and particle sizes of the plurality of metal particles, and
Based on the position and scattered light color information of the plurality of metal particles acquired by the optical microscope image and the position and particle size information of the plurality of metal particles acquired by the electron microscope image. An arithmetic unit that obtains information that associates the optical microscope image with the electron microscope image, and
Based on the information that associates the optical microscope image with the electron microscope image, the optical microscope image and the electron microscope image are aligned to generate one image in which the optical microscope image and the electron microscope image are superimposed. Image generator and
Including
The calculation unit
Using a table showing the relationship between the particle size of the metal particles and the color of the scattered light of the metal particles, the particle size of the metal particles was estimated from the color of the scattered light of the metal particles in the optical microscope image.
The optical microscope image and the electron microscope are compared with the estimated particle size and position information of the metal particles in the optical microscope image and the particle size and position information of the metal particles in the electron microscope image. Find the corresponding metal particle with the image and
Based on the corresponding metal particles, the difference in magnification between the optical microscope image and the electron microscope image and the difference in the orientation of the optical microscope image and the electron microscope image were obtained.
The image generation unit
Based on the obtained difference in magnification between the optical microscope image and the electron microscope image, the magnifications of the optical microscope image and the electron microscope image are made equal.
The orientations of the optical microscope image and the electron microscope image are aligned based on the obtained difference in orientation between the optical microscope image and the electron microscope image .

In such an image processing apparatus, the particle size of the metal particles can be estimated from the color of the bright spot in the optical microscope image. Therefore, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

(3) The electron microscope according to the present invention includes the above-mentioned image processing apparatus.

In such an electron microscope, the particle size of the metal particles can be estimated from the color of the bright spot in the optical microscope image. Therefore, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

The flowchart which shows an example of the observation method which concerns on 1st Embodiment. The figure for demonstrating the method of making a sample in the observation method which concerns on 1st Embodiment. The figure which shows typically the optical microscope image of a sample. The figure for demonstrating the method of finding the coordinates of a bright spot. The figure for demonstrating the method of finding the coordinates of a bright spot. The figure which shows typically the transmission electron microscope image of a sample. The figure for demonstrating the measurement of the particle size of a metal particle. The figure for demonstrating the method of obtaining the information which associates an optical microscope image with a transmission electron microscope image. The figure for demonstrating the method of obtaining the information which associates an optical microscope image with a transmission electron microscope image. The figure which shows typically the image generated by superimposing the optical microscope image and the transmission electron microscope image. The figure which shows the structure of an electron microscope. Optical microscope image of Paramecium and electron microscope image of Paramecium.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The embodiments described below do not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described below are essential constituent requirements of the present invention.

1. 1. First Embodiment First, the observation method according to the first embodiment will be described. FIG. 1 is a flowchart showing an example of the observation method according to the first embodiment.

(1) Sample preparation (S100)
FIG. 2 is a diagram for explaining a method for producing a sample in the observation method according to the first embodiment.

First, the observation object 2 is prepared. The observation object 2 is, for example, a biological sample. Next, after fixing the observation object 2, it is embedded in resin. Next, the observation object 2 embedded in the resin is sliced with a microtome or the like. As a result, a section of the observation object 2 is obtained. The method and order of pretreatment such as fixing, dyeing, and resin embedding for the observation object 2 are not particularly limited.

Next, a section of the observation object 2 is placed on the support film 4. The support film 4 is an extremely thin film such as a polymer or carbon. Next, an aqueous solution containing a plurality of metal particles 6 having different particle diameters is prepared. The produced aqueous solution containing the metal particles 6 is dropped onto the support film 4 and dried. As a result, a plurality of metal particles 6 can be arranged on the support film 4.

A sample can be prepared by the above processing.

In the first embodiment, the metal particles 6 that excite the localized surface plasmon resonance by irradiation with light are used as a mark for specifying the observation position. The particle size of the metal particles 6 is, for example, 20 nm or more and 1 μm or less. The shape of the metal particles 6 is, for example, spherical.

Hereinafter, the reason why the metal particles 6 are used as the marks will be described.

The vibration of electrons due to localized surface plasmon resonance depends on the particle size and surface structure of the particles. Due to the vibration of electrons due to this localized surface plasmon resonance, light of a specific wavelength is strongly absorbed or scattered. By absorbing or scattering light of a specific wavelength by localized surface plasmon resonance, nano-sized metal particles 6 can be observed in optical microscope observation. Furthermore, localized surface plasmon resonance changes the wavelength of absorbed light depending on the type of metal particles, particle size, surface structure, and the like. Therefore, in the observation with an optical microscope, the particle size of the metal particles 6 can be estimated from the color of the scattered light from the metal particles 6.

Further, by using the metal particles 6 as the mark, the absorption / scattering contrast can be obtained in the electron microscope observation, so that a sufficient contrast can be obtained as the mark in the electron microscope image.

As the metal particles 6, gold particles are preferable. Gold particles are not corroded by acids or alkalis other than aqua regia. In addition, it has low cytotoxicity and is unlikely to have an adverse effect even if it is put into cells. Gold nanoartin may be used as the gold particles. Further, as the metal particles 6, an alloy of gold and silver may be used.

(2) Observation with an optical microscope (S102)
Next, the prepared sample is observed with an optical microscope. In optical microscope observation, a dark field image is observed. That is, in the observation with an optical microscope, the scattered light from the sample is observed. In the optical microscope, the dark field image is taken by a camera capable of color photographing so that the color of the scattered light from the metal particles 6 can be confirmed. If the scattered light from the sample can be confirmed, a bright field image may be observed in the observation with an optical microscope.

FIG. 3 is a diagram schematically showing an optical microscope image I2 (dark field image) of a sample. In FIG. 3, for convenience, the color of the scattered light from the metal particles 6 is represented by hatching. That is, the metal particles 6 having the same hatching indicate that the scattered light has the same color.

As shown in FIG. 3, in the optical microscope image I2, scattered light from the metal particles 6 is observed. In the dark field image of the optical microscope, even scattered light from small metal particles 6 of about 20 nm can be observed due to the effect of localized surface plasmon resonance. In the optical microscope image I2, the scattered light from the metal particles 6 is observed as a bright spot.

(3) Extraction of bright spots (S104)
Next, a bright spot is extracted from the optical microscope image I2. That is, the metal particles 6 are extracted from the optical microscope image I2. Specifically, the coordinates (X, Y) of each bright spot and the color information of each bright spot are acquired from the optical microscope image I2. The information on the coordinates (X, Y) of the bright spot and the color of the bright spot can be said to be information on the coordinates of the metal particles and the color of the metal particles (the color of the scattered light from the metal particles) in the optical microscope image.

4 and 5 are diagrams for explaining a method for obtaining the coordinates of the bright spot. Note that FIG. 5 shows an enlarged view of the region V of FIG. 4 and an intensity profile of the bright spot. The horizontal axis of the graph shown in FIG. 5 is the position of the optical microscope image, and the vertical axis represents the brightness.

As shown in FIG. 5, it is assumed that the brightness of the bright spots is normally distributed, and the center of the bright spots in the optical microscope image I2 is defined as the coordinates (X, Y) of the bright spots. The information on the coordinates (X, Y) of the bright spot is recorded as the coordinates (X, Y) of the metal particles, and the information on the color of the bright spot is recorded as the information on the color of the metal particles. At this time, the coordinates (X, Y) of the metal particles and the color information of the metal particles are associated and recorded.

(4) Electron microscope observation (S106)
Next, the prepared sample is observed with a transmission electron microscope. Observation with a transmission electron microscope may be in a dark field or a bright field.

FIG. 6 is a diagram schematically showing a TEM image I4 (bright field image) of the sample. As shown in FIG. 6, since the electron beam is absorbed and scattered by the metal particles 6, the metal particles 6 are observed as black dots in the TEM image I4.

(5) Extraction of metal particles (S108)
Next, in the TEM image I4, the metal particles 6 are extracted. Since the metal particles 6 are observed as black dots in the TEM image I4, the coordinates of the center of the black dots are set to the coordinates (x, y) of the metal particles 6.

(6) Measurement of particle size of metal particles (S110)
Next, in the TEM image I4, the particle size (diameter) of the metal particles 6 is measured.

FIG. 7 is a diagram for explaining the measurement of the particle size of the metal particles 6.

As shown in FIG. 7, the particle size of each metal particle 6 is measured in the TEM image I4. The information on the particle size of the metal particles 6 is recorded in association with the information on the coordinates (x, y) of the metal particles 6.

(7) Calculation of information for associating an optical microscope image with an electron microscope image (S112)
Next, the information for associating the optical microscope image I2 with the TEM image I4 is obtained. The information for associating the optical microscope image I2 with the TEM image I4 includes the position and color information of the plurality of metal particles 6 acquired from the optical microscope image, the positions of the plurality of metal particles 6 acquired from the TEM image, and the information. It can be obtained based on the particle size information.

The color of the metal particles 6 in the optical microscope image I2, that is, the color of the scattered light from the metal particles 6, depends on the particle size of the metal particles 6. Specifically, as the particle size of the metal particles 6 increases, long-wavelength light is absorbed by the metal particles 6 due to localized surface plasmon resonance. Therefore, the particle size of the metal particles 6 can be estimated from the color (wavelength of light) information of the metal particles 6 obtained from the optical microscope image. Utilizing this, by comparing the particle size of the metal particles 6 and the positions of the metal particles 6 between the two images, the metal particles 6 seen in the optical microscope image I2 and the metal particles seen in the TEM image I4 are used. 6 and can be associated with each other.

By finding the corresponding metal particles 6 between the two images in this way, the coordinates (X, Y) of the metal particles 6 in the optical microscope image I2 and the coordinates (x, y) of the metal particles 6 in the TEM image I4. It can be seen that and indicates the same position on the sample. As a result, information for associating the optical microscope image I2 with the TEM image I4 can be obtained.

8 and 9 are diagrams for explaining a method for obtaining information for associating the optical microscope image I2 with the TEM image I4.

For example, as shown in FIG. 8, if two sets of corresponding metal particles 6 are found between the optical microscope image I2 and the TEM image I4, the difference in magnification between the optical microscope image I2 and the TEM image I4. And the difference in orientation can be obtained. For example, as shown in FIG. 8, the vector V2 connecting the two metal particles 6 is drawn in the optical microscope image I2, and similarly, the vector V4 connecting the two metal particles 6 is drawn in the TEM image I4. From the difference between the length of the vector V2 and the length of the vector V4, the difference in magnification between the two images can be obtained. Further, the difference in orientation between the two images can be obtained from the orientation of the vector V2 and the orientation of the vector V4. The difference in orientation between the two images can be expressed, for example, by the angle formed by the vector V2 and the vector V4.

Further, for example, as shown in FIG. 9, if three or more sets of corresponding metal particles 6 are found between the optical microscope image I2 and the TEM image I4, the magnification between the optical microscope image I2 and the TEM image I4 is increased. In addition to the difference and the difference in orientation, the distortion of the image can be obtained. In the example shown in FIG. 9, since there are four sets of corresponding metal particles 6 between the optical microscope image I2 and the TEM image I4, the difference in magnification, the difference in orientation, and the distortion of the image can be obtained.

The aberration of the optical system in an optical microscope and the aberration of the optical system in an electron microscope are different. Therefore, for example, when the optical microscope image I2 is used as a reference, the TEM image I4 is distorted with respect to the optical microscope image I2. As described above, if three or more sets of corresponding metal particles 6 are found between the optical microscope image I2 and the TEM image I4, it is determined how the TEM image I4 is distorted with respect to the optical microscope image I2. be able to. For example, it may be determined how the optical microscope image I2 is distorted with respect to the TEM image I4 with the TEM image I4 as a reference.

(8) Alignment of optical microscope image and TEM image (S114)
Next, the optical microscope image I2 and the TEM image I4 are aligned based on the information that associates the optical microscope image I2 with the TEM image I4, and the optical microscope image I2 and the TEM image I4 are overlapped to form one image.

For example, first, the magnifications of the optical microscope image I2 and the TEM image I4 are adjusted based on the information of the difference in magnification between the optical microscope image I2 and the TEM image I4. As a result, the magnifications of the optical microscope image I2 and the TEM image I4 are made equal. Further, the orientations of the optical microscope image I2 and the TEM image I4 are adjusted based on the information on the difference in orientation between the optical microscope image I2 and the TEM image I4. For example, the orientation can be adjusted by rotating the TEM image I4 with respect to the optical microscope image I2. As a result, the optical microscope image I2 and the TEM image I4 are oriented in the same direction. Further, the distortion between the optical microscope image I2 and the TEM image I4 may be corrected based on the information on the distortion between the two images.

Next, the optical microscope image I2 whose magnification and orientation are adjusted and the TEM image I4 whose magnification and orientation are adjusted are aligned, and the optical microscope image I2 and the TEM image I4 are overlapped to form one image. ..

FIG. 10 is a diagram schematically showing an image I6 generated by superimposing an optical microscope image I2 and a TEM image I4.

The image I6 shown in FIG. 10 is an image that makes the best use of the characteristics of both the optical microscope image I2 and the TEM image I4.

Through the above steps, a sample can be observed by combining an optical microscope and a transmission electron microscope.

In the above description, the case of acquiring the optical microscope image and the TEM image has been described, but the image to be combined with the optical microscope image may be an electron microscope image. Here, the electron microscope image includes a TEM image, a scanning electron microscope image (SEM image), and a scanning transmission electron microscope image (STEM image). In the SEM image and the STEM image, the particle size of the metal particles 6 can be measured in the same manner as in the TEM image. Therefore, even when acquiring the optical microscope image and the SEM image and when acquiring the optical microscope image and the STEM image, it is possible to observe the sample by the same processing as when acquiring the optical microscope image and the TEM image. is there.

Further, although the case where the object to be observed is a biological sample has been described above, it is possible to observe other samples by combining an optical microscope and an electron microscope as in the case of the above-mentioned biological sample.

The observation method according to the first embodiment has, for example, the following features.

The observation method according to the first embodiment includes a step of preparing a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light as markers, and a step of preparing a sample containing a plurality of metal particles as markers, and the positions and colors of the plurality of metal particles in an optical microscope image. The step of acquiring information, the step of acquiring the position and particle size information of a plurality of metal particles in an electron microscope image, the position and color information of a plurality of metal particles acquired in an optical microscope image, and the electron microscope image. This includes a step of obtaining information for associating an optical microscope image and an electron microscope image based on the position and particle size information of the plurality of metal particles obtained in.

In the observation method according to the first embodiment, a plurality of metal particles 6 that excite the localized surface plasmon resonance by irradiation with light are used as markers. The plurality of metal particles 6 function as markers in both the optical microscope image and the electron microscope image without being affected by the sample preparation for the electron microscope. Therefore, in the observation method according to the first embodiment, the sample in the same state can be observed with an optical microscope and an electron microscope.

Further, in the observation method according to the first embodiment, the particle size of the metal particles can be estimated from the color of the bright spot in the optical microscope image. Therefore, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

In the observation method according to the first embodiment, the optical microscope image is, for example, a dark field image. In the dark field image, the color of the scattered light from the metal particles 6 is easy to see. Therefore, the particle size of the metal particles 6 can be accurately estimated from the optical microscope image.

2. Second Embodiment Next, the image processing apparatus according to the second embodiment will be described with reference to the drawings. FIG. 11 is a diagram showing the configuration of the electron microscope 100. The electron microscope 100 includes the image processing device 10 according to the second embodiment.

The electron microscope 100 includes an image processing device 10 and an electron microscope main body 20.

The electron microscope main body 20 has a function as, for example, a transmission electron microscope. That is, the electron microscope main body 20 includes an electron gun, an irradiation system for irradiating a sample with an electron beam, a sample stage for holding the sample, an imaging system for forming an image with electrons transmitted through the sample, and an imaging system for photographing the image. Includes equipment. In the electron microscope main body 20, the electron beam emitted from the electron gun is focused by the irradiation system and irradiated to the sample. The electron beam irradiated to the sample passes through the sample. The imaging system forms a TEM image with electrons transmitted through the sample, and the TEM image is taken by an imaging device. The image data of the TEM image taken by the image pickup apparatus is sent to the image processing apparatus 10 and stored in the storage unit 124.

The image processing device 10 includes an operation unit 120, a display unit 122, a storage unit 124, and a processing unit 110.

The operation unit 120 acquires an operation signal according to the operation by the user and sends the operation signal to the processing unit 110. The function of the operation unit 120 can be realized by, for example, buttons, keys, a touch panel display, a microphone, and the like.

The display unit 122 displays the image generated by the processing unit 110. The function of the display unit 122 can be realized by, for example, a display such as an LCD (Liquid Crystal Display).

The storage unit 124 stores programs, data, and the like for the processing unit 110 to perform various calculation processes and control processes. The storage unit 124 is also used as a work area of the processing unit 110, and is also used to temporarily store the calculation results and the like executed by the processing unit 110 according to various programs. The function of the storage unit 124 can be realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk, or the like.

The processing unit 110 performs various control processing and calculation processing according to the program stored in the storage unit 124. The function of the processing unit 110 can be realized by executing a program on various processors (CPU (Central Processing Unit) or the like). The processing unit 110 includes an image acquisition unit 112, a first image information acquisition unit 114, a second image information acquisition unit 116, a calculation unit 118, and an image generation unit 119.

The image acquisition unit 112 acquires an optical microscope image of a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light as markers and an electron microscope image of the sample. For example, a user photographs a sample using an optical microscope, and stores the obtained optical microscope image I2 (see FIG. 3) in the storage unit 124. The image acquisition unit 112 reads out the optical microscope image I2 stored in the storage unit 124 and acquires the optical microscope image I2. Further, the image acquisition unit 112 acquires the TEM image I4 by reading out the TEM image I4 (see FIG. 6) taken by the electron microscope main body 20 and stored in the storage unit 124, for example.

The first image information acquisition unit 114 acquires information on the positions and colors of a plurality of metal particles in the optical microscope image I2. The first image information acquisition unit 114 extracts the bright spots in the optical microscope image I2, and acquires the coordinates of the bright spots and the color information of the bright spots. As a result, information on the position and color of each metal particle can be obtained. Information on the position and color of each metal particle is recorded in the storage unit 124.

The second image information acquisition unit 116 acquires information on the positions and particle sizes of the plurality of metal particles in the TEM image I4. The second image information acquisition unit 116 extracts the contrast portion corresponding to the metal particles 6 from the TEM image I4. For example, when the TEM image I4 is a bright field image, the metal particles 6 are observed as black dots. Therefore, the second image information acquisition unit 116 acquires information on the coordinates of the black dots and the particle size of the black dots. As a result, information on the position and particle size of each metal particle can be obtained. Information on the position and particle size of each metal particle is recorded in the storage unit 124.

The calculation unit 118 is based on the position and color information of the plurality of metal particles 6 acquired by the optical microscope image I2 and the position and particle size information of the plurality of metal particles acquired by the TEM image I4. Information for associating the optical microscope image I2 with the TEM image I4 is obtained. The information that associates the optical microscope image I2 with the TEM image I4 is information on the position of the corresponding metal particle 6 between the two images, information on the difference in magnification between the two images, and information on the difference in orientation between the two images. Contains information. Further, the information associating the optical microscope image I2 with the TEM image I4 may include information on the strain between the two images.

The calculation unit 118 obtains information on the positions of the corresponding metal particles 6 in the optical microscope image I2 and the TEM image I4. Specifically, the calculation unit 118 estimates the particle size of the metal particles 6 from the color information of the metal particles 6 in the optical microscope image I2. The calculation unit 118 estimates the particle size of the metal particles 6 from the color of the metal particles 6 by using a table showing the relationship between the particle size of the metal particles 6 and the color of the scattered light from the metal particles. The calculation unit 118 compares the estimated particle size and position information of the metal particle 6 with the information on the particle size and position of the metal particle 6 in the TEM image I4, and the corresponding metal particle between the two images. Find 6.

For example, as shown in FIG. 8, the calculation unit 118 has a vector V2 connecting two metal particles 6 in the optical microscope image I2 and a vector V4 connecting two metal particles 6 in the TEM image I4.
From the difference in length of, the difference in magnification between the two images is obtained. Further, the calculation unit 118 obtains the difference in orientation between the two images from the orientation of the vector V2 and the orientation of the vector V4.

Further, the arithmetic unit 118 obtains the distortion of the TEM image I4 with respect to the optical microscope image I2 if there are three or more sets of corresponding metal particles 6 between the two images. The distortion of the optical microscope image I2 with respect to the TEM image I4 may be obtained.

The image generation unit 119 aligns the optical microscope image I2 and the TEM image I4 based on the information for associating the optical microscope image I2 with the TEM image I4, and superimposes the optical microscope image I2 and the TEM image I4. To generate.

The image generation unit 119 equalizes the magnifications of the optical microscope image I2 and the TEM image I4 based on the information of the difference in magnification between the optical microscope image I2 and the TEM image I4. Further, the image generation unit 119 makes the directions of the optical microscope image I2 and the TEM image I4 the same based on the information of the difference in the orientations of the optical microscope image I2 and the TEM image I4. Further, when the information on the distortion of the TEM image I4 with respect to the optical microscope image I2 is obtained, the distortion between the optical microscope image I2 and the TEM image I4 is corrected. The image generation unit 119 aligns the optical microscope image I2 and the TEM image I4 having the same magnification and the same orientation in this way, and superimposes the optical microscope image I2 and the TEM image I4 to obtain one image I6 (FIG. 10). See). The image generation unit 119 causes the display unit 122 to display the generated image I6.

The image processing device 10 has, for example, the following features.

In the image processing apparatus 10, the calculation unit 118 is based on the position and color information of the plurality of metal particles acquired by the optical microscope image and the position and particle size information of the plurality of metal particles acquired by the electron microscope image. Then, the information for associating the optical microscope image with the electron microscope image is obtained. As described above, in the image processing apparatus 10, since a plurality of metal particles that excite the localized surface plasmon resonance by irradiation with light are used as markers, the particle size of the metal particles is determined by the color of the bright spot in the observation with an optical microscope. You can guess. Therefore, in the image processing apparatus 10, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

In the image processing apparatus 10, the image generation unit 119 aligns the optical microscope image and the electron microscope image based on the information for associating the optical microscope image with the electron microscope image, and superimposes the optical microscope image and the electron microscope image. Generate one image. Therefore, the image processing apparatus 10 can obtain an image that makes the best use of the characteristics of both the optical microscope image I2 and the TEM image I4.

3. 3. Modification Examples In the following, points different from the observation method according to the first embodiment described above will be described, and description of the same points will be omitted.

In the first embodiment described above, the metal particles 6 are placed on the support film 4 by dropping an aqueous solution containing the metal particles 6 onto a section of the observation object 2 on the support film 4 and drying the mixture. The method for preparing a sample containing the plurality of metal particles 6 is not limited to this.

For example, gold nanoparticles can be used as labels for cells to be observed, organelles, proteins, and the like. Gold nanoparticles are gold particles having a particle size of nano-order. The surface of the gold nanoparticles can be decorated in various ways, and proteins and substrates can be bound to the gold nanoparticles. When labeling cells with gold nanoparticles, gold nanoparticles are added to the medium and the gold nanoparticles are incorporated into the cells by endocytosis. It is also possible to inject gold nanoparticles into cells using microelectrodes.

When gold nanoparticles are labeled on a protein, an antibody against the protein to be observed is prepared and immunostaining is performed. At that time, the gold nanoparticles are bound to the secondary antibody. If the protein is an enzyme, it can also be labeled by binding gold nanoparticles to a substrate for the enzyme.

Further, for example, organisms such as Paramecium may be fed with metal particles. Hereinafter, a sample preparation method when the object to be observed is Paramecium will be described.

First, gold nanoparticles are added to the medium in which Paramecium is cultivated. This allows the Paramecium to feed on the gold nanoparticles. The edible gold nanoparticles are stored in the vesicles of Paramecium.

Next, a sample for a transmission electron microscope of Paramecium that has eaten gold nanoparticles is prepared. For example, Paramecium is fixed with glutaraldehyde and osmium tetroxide, dehydrated with an ethanol series, and then embedded in an epoxy resin. The resin-embedded Paramecium is cut into a thickness of 100 nm or less with a microtome, and a section thereof is placed on a support membrane. The sample section is then double stained with uranium acetate and lead citrate. Through the above steps, a sample for a transmission electron microscope can be prepared.

FIG. 12 is an optical microscope image (dark field image) and a TEM image of Paramecium. The image (a) shown in FIG. 12 is an optical microscope image (dark field image), and the image (b) is a TEM image having the same field of view as (a). Further, the image (c) is a magnified image of the image (a), and the image (d) is a magnified image of the image (b). The image (e) is an image in which the image (c) and the image (d) are superimposed.

As described above, the gold nanoparticles as markers are exposed to glutaraldehyde, osmium tetroxide, uranium oxide, and lead citrate at the time of sample preparation. However, in the optical microscope image shown in FIG. 12, orange scattered light from the gold nanoparticles can be seen mixed with the scattered light from the paramecium. Since the region glowing orange in the image (c) is the paramecium cyst, the paramecium cyst can be identified in the image (d).

Further, as shown in the image (e), by using the gold nanoparticles as markers, the optical microscope image and the TEM image can be accurately aligned.

Here, labeling using gold particles is used in immunoelectron microscopy. Immune electron microscopy is a method of visualizing the localization of components that make up cells and tissues at the electron microscope level using an immune response. The particle size of gold particles used in immunoelectron microscopy is 5 nm or less in order to penetrate cells and tissues. However, with gold particles smaller than 10 nm, the scattered light is weak and the scattered light cannot be observed by optical microscope observation.

Therefore, the particle size of the gold nanoparticles may be increased after the gold nanoparticles-labeled antibody is impregnated into the cells or tissues to be observed.

First, after fixing the cultured cells, an anti-mitochondrial antibody is reacted as a primary antibody. Next, the cultured cells are reacted with a gold colloid-labeled anti-rabbit IgG antibody as a secondary antibody. A gold colloid-labeled anti-rabbit IgG antibody is one in which gold nanoparticles are bound to an anti-rabbit IgG antibody. The particle size of the gold nanoparticles is 5 nm or less.

After reacting the cultured cells with the primary antibody and the secondary antibody as described above, the particle size of the gold nanoparticles is increased by the gold sensitization method. Next, the cultured cells are dehydrated and resin-embedded, and the resin-embedded cells are sliced with an ultramicrotome or the like. The sample section thus prepared is placed on the support membrane.

Through the above steps, a sample for a transmission electron microscope can be prepared.

As described above, when the particle size of the gold nanoparticles is increased by the gold sensitization method, the particle size of the gold nanoparticles varies. Therefore, a plurality of bright spots having different colors are observed in the optical microscope image, and a plurality of metal particles having different particle sizes are observed in the electron microscope image. Therefore, the metal particles of the optical microscope image and the metal particles of the electron microscope image can be easily associated with each other. Therefore, the optical microscope image and the electron microscope image can be accurately aligned.

As described above, after introducing metal particles having a small particle size into a sample, the particle size of the metal particles is increased to improve visibility in optical microscope observation while maintaining permeability to cells and tissues. Can be done.

The present invention includes substantially the same configurations as those described in the embodiments (eg, configurations with the same function, method and result, or configurations with the same purpose and effect). The present invention also includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced. The present invention also includes a configuration that exhibits the same effects as the configuration described in the embodiment or a configuration that can achieve the same object. The present invention also includes a configuration in which a known technique is added to the configuration described in the embodiment.

2 ... Observation object, 4 ... Support film, 6 ... Metal particles, 10 ... Image processing device, 20 ... Electron microscope main body, 100 ... Electron microscope, 110 ... Processing unit, 112 ... Image acquisition unit, 114 ... First image information Acquisition unit, 116 ... Second image information acquisition unit, 118 ... Calculation unit, 119 ... Image generation unit, 120 ... Operation unit, 122 ... Display unit, 124 ... Storage unit

Claims (7)

A process of preparing a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light as markers, and a process of preparing a sample.
The process of acquiring an optical microscope image by photographing the sample with an optical microscope, and
The process of acquiring an electron microscope image by photographing the sample with an electron microscope, and
In the optical microscope image, a step of acquiring information on the positions of a plurality of the metal particles and the color of scattered light, and
A step of acquiring information on the positions and particle sizes of a plurality of the metal particles in the electron microscope image, and
Based on the position and scattered light color information of the plurality of metal particles acquired by the optical microscope image and the position and particle size information of the plurality of metal particles acquired by the electron microscope image. A step of obtaining information for associating the optical microscope image with the electron microscope image, and
Based on the information that associates the optical microscope image with the electron microscope image, the optical microscope image and the electron microscope image are aligned to generate one image in which the optical microscope image and the electron microscope image are superimposed. Process and
Only including,
In the step of obtaining information for associating the optical microscope image with the electron microscope image,
Based on the relationship between the particle size of the metal particles and the color of the scattered light of the metal particles, the particle size of the metal particles is estimated from the color of the scattered light of the metal particles in the optical microscope image.
The optical microscope image and the electron microscope are compared with the estimated particle size and position information of the metal particles in the optical microscope image and the particle size and position information of the metal particles in the electron microscope image. Find the corresponding metal particle with the image and
Based on the corresponding metal particles, the difference in magnification between the optical microscope image and the electron microscope image and the difference in the orientation of the optical microscope image and the electron microscope image were obtained.
In the step of generating one image,
Based on the obtained difference in magnification between the optical microscope image and the electron microscope image, the magnifications of the optical microscope image and the electron microscope image are made equal.
An observation method in which the directions of the optical microscope image and the electron microscope image are aligned based on the obtained difference in orientation between the optical microscope image and the electron microscope image. In claim 1,
The observation method, wherein the metal particles are gold particles. In claim 1 or 2,
The observation method, wherein the optical microscope image is a dark field image. In any one of claims 1 to 3 ,
The observation method, wherein the sample includes an observation object labeled with the metal particles. In any one of claims 1 to 4 ,
The step of preparing the sample is
The step of labeling the observation object with the metal particles and
The step of enlarging the metal particles by the sensitization method and
Observation methods, including. An image acquisition unit that acquires an optical microscope image of a sample containing a plurality of metal particles that excite localized surface plasmon resonance by irradiation with light and an electron microscope image of the sample as markers.
In the optical microscope image, a first image information acquisition unit that acquires information on the positions of a plurality of the metal particles and the color of scattered light, and
In the electron microscope image, a second image information acquisition unit that acquires information on the positions and particle sizes of the plurality of metal particles, and
Based on the position and scattered light color information of the plurality of metal particles acquired by the optical microscope image and the position and particle size information of the plurality of metal particles acquired by the electron microscope image. An arithmetic unit that obtains information that associates the optical microscope image with the electron microscope image, and
Based on the information that associates the optical microscope image with the electron microscope image, the optical microscope image and the electron microscope image are aligned to generate one image in which the optical microscope image and the electron microscope image are superimposed. Image generator and
Including
The calculation unit
Using a table showing the relationship between the particle size of the metal particles and the color of the scattered light of the metal particles, the particle size of the metal particles was estimated from the color of the scattered light of the metal particles in the optical microscope image.
The optical microscope image and the electron microscope are compared with the estimated particle size and position information of the metal particles in the optical microscope image and the particle size and position information of the metal particles in the electron microscope image. Find the corresponding metal particle with the image and
Based on the corresponding metal particles, the difference in magnification between the optical microscope image and the electron microscope image and the difference in the orientation of the optical microscope image and the electron microscope image were obtained.
The image generation unit
Based on the obtained difference in magnification between the optical microscope image and the electron microscope image, the magnifications of the optical microscope image and the electron microscope image are made equal.
An image processing device that aligns the directions of the optical microscope image and the electron microscope image based on the obtained difference in orientation between the optical microscope image and the electron microscope image. An electron microscope including the image processing apparatus according to claim 6.

Images