JP6720165B2 - Image acquisition apparatus and image acquisition method - Google Patents

Description

The present invention relates to an image acquisition device and an image acquisition method.

The DMD generates a plurality of illumination lights having different irradiation patterns, irradiates a plurality of different positions on the sample, and the fluorescence from the plurality of positions detected by the detector is mixed to give the illumination light by the DMD. There is known a microscope that restores the fluorescence emitted from each irradiation position based on the irradiation pattern (see, for example, Patent Document 1).

WO2011/023593 pamphlet

The technique of Patent Document 1 is for irradiating the specific position of the sample with the illumination light generated by the DMD. If this technique is applied to image acquisition, the response speed of the DMD is higher than that of optical scanning. There is an inconvenience that a sufficient frame rate cannot be secured due to the problem of being slow.
Further, the above problem can be solved by using a device capable of high-speed modulation such as EOM instead of DMD, but there is a disadvantage that the system becomes very expensive because modulation devices for multiple numbers are required. is there.
The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image acquisition apparatus and an image acquisition method capable of improving a frame rate without requiring devices for the number of multiplexes. There is.

One aspect of the present invention is an image acquisition apparatus that separately acquires local signals corresponding to respective illumination areas while simultaneously illuminating a plurality of areas, and modulates the intensity of light emitted from a light source to mutually acquire the signals. An illumination light generation unit that generates a plurality of illumination lights including a modulated illumination light that is modulated and a non-modulated illumination light that is not modulated, and a plurality of illumination light generated by the illumination light generation unit. Illuminating optical system for irradiating different positions of a sample with the illuminating light, and detecting and synthesizing synthetic signal light obtained by synthesizing signal light generated at a plurality of irradiation positions irradiated with the illuminating light by the illuminating optical system. It corresponds to the modulated illumination light by using the time integration of the product of the demodulated signal corresponding to the modulated illumination light and the synthesized signal from the photodetector that outputs a signal and the synthesized signal output by the photodetector. And a demodulation unit that separates the modulated local signal, separates the sum of the modulated local signals from the time integration of the combined signal, and separates the unmodulated local signal corresponding to the unmodulated illumination light. ..

According to this aspect, the illumination light generation unit modulates the intensity of the light emitted from the light source to have a pattern that is linearly independent of each other, and modulates the modulated illumination light and the unmodulated illumination light that is not modulated. A plurality of illumination lights including the generated illumination lights are generated, and the generated illumination lights are applied to different positions of the sample by the illumination optical system. Since the signal lights are simultaneously generated at the irradiation positions of the respective illumination lights, the combined signal lights of the plurality of signal lights generated at the plurality of irradiation positions are detected by the photodetector and the combined signals are output. Then, the output combined signal is input to the demodulation unit so that the modulated local signal corresponding to the modulated illumination light is obtained from the combined signal by using the time integration of the product of the demodulated signal corresponding to the modulated illumination light and the combined signal. And the sum of the modulated local signals is subtracted from the time integration of the combined signal to separate the non-modulated local signal corresponding to the non-modulated illumination light, and the signal light generated at each irradiation position is demodulated.

In this case, since the signal light demodulation unit performs the demodulation processing by using the demodulation signal synchronized with the pattern of each illumination light, the accuracy of each signal light from the combined signal light is obtained by utilizing the primary independence of the illumination lights. Can be well extracted. Therefore, the signal lights emitted simultaneously from a plurality of irradiation positions can be accurately separated and detected, and the frame rate can be improved without requiring devices for the multiplex number.

In the above aspect, the time integration of the product of the demodulated signal not corresponding to the modulated illumination light and the modulated local signal not corresponding to the modulated illumination light and the non-modulated local signal may be zero.
In the above aspect, the time integration of the product of the demodulated signal corresponding to the non-modulated illumination light and the composite signal may be the time integration of the composite signal.

Further, in the above aspect, a modulator that modulates the intensity of light may be provided in at least one optical path of the plurality of illumination lights generated by the illumination light generation unit.

Further, in the above aspect, two or more of the modulators may be provided, and each of the modulators may generate the modulated illumination light having the demodulated signals that are orthogonal to each other.
By doing so, two or more illumination lights having demodulation signals that are orthogonal to each other can be generated by the plurality of modulators, and the amount of signals obtained by using the demodulation signals that have orthogonality increases. The frame rate can be further improved.

Further, in the above aspect, it may have a function of arranging a plurality of the irradiation positions at arbitrary positions in the sample.

Further, another aspect of the present invention is that a plurality of modulated illumination lights that are modulated independently of each other in intensity of light emitted from a light source and that have modulated and unmodulated illumination lights are provided. Illumination light generation step of generating the illumination light of, the irradiation step of irradiating different positions of the sample with the plurality of illumination lights generated by the illumination light generation step, and a plurality of the illumination light irradiated by the irradiation step. Of the signal light generated at the irradiation position of the light detecting step of detecting the combined signal light and outputting the combined signal, and the demodulation corresponding to the modulated illumination light from the combined signal output by the light detecting step. The modulated local signal corresponding to the modulated illumination light is separated by using the time integration of the product of the signal and the combined signal, and the non-modulated illumination light is obtained by subtracting the sum of the modulated local signals from the time integration of the combined signal. And a signal light demodulation step of separating an unmodulated local signal corresponding to the above.

According to the present invention, there is an effect that the frame rate can be improved without requiring devices for the number of multiplexing.

It is a schematic diagram which shows the image acquisition apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the modulation signal of the pattern which received the modulation|alteration by the electro-optic modulator in the image acquisition apparatus of FIG. It is a figure which shows an example of the modulation signal of the pattern which is not modulated by the electro-optic modulator in the image acquisition device of FIG. It is a figure which shows an example of the demodulation signal of the pattern which received the modulation|alteration by the electro-optic modulator in the image acquisition apparatus of FIG. It is a figure which shows an example of the demodulation signal of the pattern which is not modulated by the electro-optic modulator in the image acquisition device of FIG. 6 is a flowchart illustrating an image acquisition method according to an exemplary embodiment of the present invention. It is a modification of the image acquisition device of FIG. 1, and is a schematic diagram which shows the example applied to the confocal microscope. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the other method of combining several illumination light. It is a modification of the image acquisition apparatus of FIG. 1, and is a figure which shows the method of arrange|positioning a pair of mirror in the position of the broken line part S in FIG. 5A. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the other method of combining several illumination light. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the method of adjusting the space|interval of several illumination light. It is a modification of the image acquisition apparatus of FIG. 1, and is a figure which shows the case where three illumination lights are multiplexed. It is a figure which shows an example of the modulation signal of the 1st modulation pattern in the image acquisition apparatus of FIG. It is a figure which shows an example of the modulation signal of the 2nd modulation pattern in the image acquisition apparatus of FIG. It is a figure which shows an example of the modulated signal of the pattern which is not modulated in the image acquisition apparatus of FIG. It is a figure which shows an example of the demodulation signal of the 1st modulation pattern in the image acquisition apparatus of FIG. It is a figure which shows an example of the demodulation signal of the 2nd modulation pattern in the image acquisition apparatus of FIG. It is a figure which shows an example of the demodulation signal of the pattern which is not modulated in the image acquisition apparatus of FIG. It is a figure which is a modification of the image acquisition device of FIG. 1 and shows the case where it modulates using an acousto-optic deflection element. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the case where four acousto-optic deflection elements are used and four illumination lights are multiplexed. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the case where four illumination lights are multiplexed by making effective use of the light repelled at the time of the modulation by an electro-optic modulator. It is a figure which is a pattern of the light used in the image acquisition apparatus of FIG. 12, Comprising: The pattern of the modulation signal of the light from a light source. It is a figure which is a pattern of the light used in the image acquisition apparatus of FIG. 12, Comprising: The pattern of the modulation signal of the 1st illumination light immediately after being output from the electro-optic modulator. It is a figure which is a pattern of the light used in the image acquisition apparatus of FIG. 12, Comprising: The pattern of the modulation signal of the 2nd illumination light immediately after being output from the electro-optic modulator. It is a figure which is a pattern of the light used in the image acquisition device of FIG. 12, Comprising: The pattern of the modulation signal of the 3rd illumination light immediately after being branched by the beam splitter. It is a figure which is a pattern of the light used in the image acquisition device of FIG. 12, Comprising: The pattern of the modulation signal of the 4th illumination light immediately after being branched by the beam splitter. FIG. 13 is a light pattern used in the image acquisition device of FIG. It is a figure which is a pattern of the light used in the image acquisition apparatus of FIG. 12, Comprising: The pattern of the demodulation signal of the 1st illumination light immediately after being output from the electro-optic modulator. It is a figure which is a pattern of the light used in the image acquisition apparatus of FIG. 12, Comprising: The pattern of the demodulation signal of the 2nd illumination light immediately after being output from the electro-optic modulator. It is a figure which is a pattern of the light used in the image acquisition device of FIG. 12, Comprising: The pattern of the demodulation signal of the 3rd illumination light immediately after being branched by the beam splitter. It is a figure which is a pattern of the light used in the image acquisition device of FIG. 12, Comprising: The pattern of the demodulation signal of the 4th illumination light immediately after being branched by the beam splitter. It is a figure which shows the pattern of the modulation signal of the 1st illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the modulation signal of the 2nd illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the modulation signal of the 3rd illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the modulation signal of the 4th illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the demodulation signal of the 1st illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the demodulation signal of the 2nd illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the demodulation signal of the 3rd illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a figure which shows the pattern of the demodulation signal of the 4th illumination light immediately after being multiplexed by the beam splitter in the image acquisition apparatus of FIG. It is a modification of the image acquisition device of FIG. 1, and is a figure which shows the case where four illumination lights are multiplexed by making effective use of the light repelled at the time of multiplexing by a beam splitter. It is a modification of the image acquisition device of FIG. 1, and is a schematic diagram which shows the case where a sheet illumination is utilized.

An image acquisition device 1 and an image acquisition method according to an embodiment of the present invention will be described below with reference to the drawings.
The image acquisition device 1 according to the present embodiment is a multiphoton excitation type scanning fluorescence microscope, and as shown in FIG. 1, emits ultrashort pulsed laser light having a constant peak intensity at predetermined time intervals. Illumination that generates two illumination lights L1 and L2 that have a light source 2 and a pattern that is linearly independent of each other from the light from the light source 2 and that includes a modulated illumination light and a non-modulated illumination light that is not modulated. The light generation unit 3, the illumination optical system 4 for irradiating the plurality of generated illumination lights L1, L2 to different positions of the sample A, and the fluorescence (signal light) generated at the plurality of irradiation positions of the illumination lights L1, L2. A photodetection unit 5 that detects the synthetic fluorescence (synthetic signal light) and outputs the synthetic signal, and a demodulation unit (signal light demodulation unit) 6 that demodulates the fluorescence generated at each irradiation position from the output synthetic signal. I have it.

The illumination light generation unit 3 is provided in a first beam splitter (light splitting unit) 7 that splits the light from the light source 2 into two optical paths and one of the split optical paths, and changes the intensity of passing light with time. An electro-optic modulator (modulator) 8 that modulates a pattern, imaging lenses 9a and 9b, mirrors 10a and 10b, and a second beam splitter (combining unit) 11 that multiplexes the lights of the two optical paths. I have it. The mirror 10a is a position where the primary imaging position by the imaging lens 9a provided on one optical path is shifted in the direction orthogonal to the optical axis with respect to the primary imaging position by the imaging lens 9b provided on the other optical path. It is located in.

A signal controller 12 is connected to the electro-optical modulator 8. The signal control unit 12 sends a modulated signal as shown in FIG. 2A to the electro-optical modulator 8 to block every other pulsed light from the light source, so that the duty becomes 50%. It is designed to be modulated like. For example, when the light emitted from the light source 2 has a repetition frequency of 80 MHz, the electro-optical modulator 8 modulates the intensity of the light from the light source 2 to generate the illumination light L1 of 40 MHz. .. As a result, the illumination light L1 output from the electro-optic modulator 8 has a modulation pattern whose intensity changes at 40 MHz.

In addition, the electro-optical modulator 8 is not provided in the other optical path, and the light emitted from the light source 2 is illuminated as it is, that is, in a pattern that is not modulated by a pulse train having a constant peak intensity. The light L2 is generated.
As a result, the illumination light L1 of the modulated pattern that has passed through the two optical paths and the illumination light L2 of the pattern that has not been modulated have primary-independent patterns having temporally varying intensities.

The illumination optical system 4 simultaneously projects the pupil projection lens 13 that projects the two illumination lights L1 and L2 combined by the second beam splitter 11 onto the scanner 14 and the two illumination lights L1 and L2 in the same direction (XY directions). A two-dimensional scanning scanner 14, relay lenses 15a and 15b that relay the illumination lights L1 and L2 scanned by the scanner 14, and illumination lights L1 and L2 relayed by the relay lenses 15a and 15b to the sample. While irradiating a plurality of irradiation positions of A, the objective lens 16 that collects the fluorescence generated at each irradiation position of the sample A is provided. The scanner 14 is, for example, a galvanometer mirror.

The scanner 14 and the pupil position of the objective lens 16 are arranged at optically conjugate positions. The inter-lens distance between the imaging lenses 9a and 9b and the pupil projection lens 13 is set to be equal in the two optical paths.
A dichroic mirror 17 having a characteristic of transmitting the illumination lights L1 and L2 and reflecting fluorescence is disposed between the relay lenses 15a and 15b of the illumination optical system 4 and the objective lens 16. The photodetection section 5 includes, for example, a photodetector 5a such as a photomultiplier tube and an amplifier 5b that amplifies the fluorescence intensity signal detected by the photodetector 5a.

The demodulation unit 6 receives the demodulation signal synchronized with the modulation signal from the signal control unit 12, and demodulates the fluorescence generated at each irradiation position from the synthetic fluorescence detected by the photodetector 5a. Specifically, the modulated local signal corresponding to the modulated illumination light is separated by using the time integration of the product of the demodulated signal corresponding to the modulated illumination light and the combined signal, and the sum of the modulated local signals is calculated from the time integration of the combined signal. The differential is used to separate the non-modulated local signal corresponding to the non-modulated illumination light. In this case, the time integration of the product of the demodulated signal that does not correspond to the modulated illumination light and the modulated local signal and the unmodulated local signal that does not correspond to the modulated illumination light becomes zero, and the demodulation code and the combined signal corresponding to the unmodulated illumination light The time integration of the product becomes the time integration of the composite signal.

In this example, the demodulation signal is synchronized with the ON/OFF of the modulation, and the ON time is 1 and the OFF time is -1. In the above example, the demodulation signal shown in FIG. 2C, which is synchronized with the modulation signal shown in FIG. 2A for modulating the intensity of the light from the light source 2 to generate the illumination light L1 having the modulation pattern, is code1=(−1,1 ,-1,1). On the other hand, the demodulated signal when modulation is not performed as shown in FIG. 2B can be expressed as code2=(1,1,1,1) as shown in FIG. 2D. Each code length corresponds to the exposure time of one pixel. In this example, the code and the optical modulation correspond to each pulse, but the modulation may be performed for each pulse train of arbitrary length.

In the demodulation unit 6, the intensity of the fluorescence of the modulation pattern is multiplied by the intensity of the synthetic fluorescence detected by the photodetector 5a by the demodulation signal code1 synchronized with the modulation timing, and integrated over the exposure time. The modulated local signal corresponding to is separated and demodulated. On the other hand, the intensity of the fluorescence of the pattern which is not modulated is the intensity of the fluorescence of the demodulated modulation pattern (modulation) from the integrated value of the synthetic fluorescence obtained by integrating the intensity of the synthetic fluorescence detected by the photodetector 5a over the exposure time. By subtracting the sum of local signals), the non-modulated local signal corresponding to the non-modulated illumination light is separated and demodulated. Since multiplying the demodulated signal code2 is equivalent to not multiplying anything, it is only necessary to subtract the intensity of the fluorescence of the modulation pattern from the integrated value of the detected synthetic fluorescence.

As a specific example, the fluorescence signal intensity corresponding to one pulse of the illumination light L1 is α, and the fluorescence signal corresponding to one pulse of the illumination light L2 is β.
Further, when the exposure time is four pulses, the time series data D1 of the fluorescence signal during the exposure time corresponding to the illumination light L1 becomes (0, α, 0, α). On the other hand, the time series data D2 of the fluorescence signal during the exposure time corresponding to the illumination light L2 becomes (β, β, β, β). Therefore, the time series data D of the synthetic fluorescence signal is D=D1+D2=(β, α+β, β, α+β).

By taking the inner product (the same as multiplication and integration) of the time series data D of the synthetic fluorescence signal and code1, 2α corresponding to the integrated value of D1 can be calculated. Further, the inner product of the time-series data D of the combined fluorescence signal and code2 becomes the integrated value 2α+4β of the combined fluorescence signal, and the fluorescence signal intensity 2α corresponding to the previously obtained illumination light L1 is subtracted from this to obtain the illumination light. The fluorescence signal 4β corresponding to L2 can be obtained.

An image acquisition method using the image acquisition device 1 according to the present embodiment configured as described above will be described below.
As shown in FIG. 3, the image acquisition method according to the present embodiment is an illumination for generating a plurality of illumination lights L1 and L2 having mutually independent primary patterns from the ultrashort pulsed laser light emitted from the light source 2. The light generation step S1, the irradiation step S2 of irradiating the generated plurality of illumination lights L1 and L2 to different positions of the sample A, and the fluorescence generated at the plurality of irradiation positions of the illumination lights L1 and L2 are combined. The light detection step S3 for detecting the combined fluorescence, and the fluorescence demodulation step for demodulating the fluorescence generated at each irradiation position from the detected combined fluorescence using a demodulation signal synchronized with the pattern of the illumination lights L1 and L2. Signal light demodulation step) S4.

According to the image acquisition device 1 and the image acquisition method according to the present embodiment configured as described above, since the electro-optic modulator 8 only needs to be provided in one of the two optical paths branched, a single electro-optic modulation is performed. There is an advantage that the two illumination lights L1 and L2 can be multiplexed only by the device 8, and the device can be configured easily and inexpensively.
Then, since the sample A is simultaneously irradiated with the two multiplexed illumination lights L1 and L2, two different areas in the sample are simultaneously scanned by scanning the spots of the two illumination lights L1 and L2 on the sample A by the scanner 14. Since imaging can be performed, there is an advantage that the frame rate for acquiring an image in the observation range can be doubled without requiring multiple devices for multiplexing.

Further, since it is not necessary to modulate the illumination light L2 passing through one of the optical paths, the maximum acquisition signal amount is 1 as compared with the case where the two illumination lights L1 and L2 are generated by the two electro-optic modulators. There is an advantage that it can be improved by 5 times.

In the present embodiment, a multiphoton excitation type fluorescence microscope is illustrated as the image acquisition device 1, and therefore, the dichroic mirror 17 that branches the fluorescence from the illumination lights L1 and L2 is provided between the objective lens 16 and the relay lens 15b. However, instead of this, as shown in FIG. 4, it is applied to a confocal microscope, the dichroic mirror 17 is arranged closer to the light source 2 than the scanner 14, and each illumination light L1 from the objective lens 16 is arranged. , L2 may be arranged at a position optically conjugate with the focal position of L2.

The number and spacing of the through holes 18a in the pinhole member 18 are set according to the number of illumination lights L1 and L2 and the distance between the focal positions.
In this case, the light source 2 may be one that emits continuous light having a constant intensity.

Although the beam splitter is used as the light branching unit 7, a polarized beam splitter may be used. Although the beam splitter is used as the multiplexing unit 11, a polarization beam splitter or a combination of the triangular prism 19 and the mirrors 20a and 20b may be used as shown in FIG. 5A. In this case, if the triangular prism 19 and the mirrors 20a and 20b are moved relative to each other in the direction of the arrow B, the distance between the illumination lights L1 and L2 in the X direction can be changed.

Further, instead of combining the light between the imaging lenses 9a and 9b and the pupil projection lens 13, as shown in FIG. 9 may be shared. Further, branching and multiplexing may be performed by any other method.

Further, while the mirror 10a is fixed and the shift amount of the primary image formation position is fixed, the shift amount of the primary image formation position is changed by moving the mirror 10a in the direction of arrow C shown in FIG. It may be adjustable.
Although the electro-optic modulator 8 is illustrated as the modulator, a modulator such as an acousto-optic modulator may be used instead of the electro-optic modulator 8.

Further, as shown in FIG. 5B, in addition to the combination of the triangular prism 19 and the mirrors 20a and 20b, in order to change the ray height of the illumination light L1, the spacing between the illumination lights L1 and L2 in the Y direction is set in the optical path. It is also possible to introduce a pair of mirrors 37a and 37b that can change the distance. Thereby, by narrowing or widening the interval between the mirrors 37a and 37b, the beam can be moved in the Y direction (the direction perpendicular to the adjacent beams) on the plane perpendicular to the optical axis.
Further, the lenses 9a and 9b are provided so as to be movable in the direction along the optical axis, and the lenses 9a and 9b are moved along the optical axis to adjust the Z direction position of each condensing point in the sample A. Good.

By doing so, by combining the movement in the XY direction perpendicular to the optical axis and the movement in the Z direction along the optical axis, it is possible to arrange the respective focal points at arbitrary positions. As a result, it is possible to perform imaging of distant arbitrary volumes with temporal synchronism. As a result, in the elucidation of the function of the brain, it is possible to clarify the interaction of regions having different functions by using simultaneous imaging of multiple regions.

Further, in the present embodiment, the light from the light source 2 is branched into two, but instead of this, it may be branched into three or more. Since the arrival pulse number can be considered to be the same as the dimension of the vector space used for demodulation, the illumination light can be multiplexed by the same number as the arrival pulse number during the exposure time. This makes it possible to further increase the speed.

FIG. 8 illustrates a case of branching into three optical paths.
The branch to the three optical paths is a beam splitter 7a for branching the light from the light source 2 into the optical paths of the illumination lights L1 and L2 and the optical path of the illumination light L3, and the branched splitter arranged downstream of the beam splitter 7a. The beam splitter 7b further divides the optical paths of the lights L1 and L2 into the optical path of the illumination light L1 and the optical path of the illumination light L2. Imaging lenses 9a, 9b, and 9c are arranged in the three optical paths, respectively.
There is no electro-optical modulator 8 in one optical path, and electro-optical modulators (modulators) 8a that generate illumination light (modulated illumination light) L1 and L2 having mutually orthogonal modulation patterns in the other two optical paths. , 8b are arranged respectively.

Specifically, as shown in FIGS. 9A and 9D, the modulation signal corresponding to one electro-optical modulator 8a and the demodulation signal of the modulation pattern code1=(-1,1,-1,1,1), FIG. 9B and As shown in FIG. 9E, the modulation signal corresponding to the other electro-optical modulator 8b and the demodulation signal of the modulation pattern code2=(1,-1,-1,1), as shown in FIGS. 9C and 9F, It is assumed that the modulation signal that has not been modulated by the optical modulators 8a and 8b and the demodulation signal of the modulation pattern code3=(1,1,1,1).
The fluorescence signal corresponding to one pulse of the illumination light L1 is α, the fluorescence signal corresponding to one pulse of the illumination light L2 is β, and the fluorescence signal corresponding to one pulse of the illumination light L3 is γ.

Further, when the exposure time is four pulses, the time series data D1 of the fluorescence signal during the exposure time corresponding to the illumination light L1 becomes (0, α, 0, α). Then, the time series data D2 of the fluorescence signal during the exposure time corresponding to the illumination light L2 becomes (β, 0, 0, β), and the time series data D3 of the fluorescence signal during the exposure time corresponding to the illumination light L3 becomes ( γ, γ, γ, γ). Therefore, the time series data D of the synthetic fluorescence signal is D=D1+D2+D3=(β+γ, α+γ, γ, α+β+γ).

When the inner product of the time series data D of the synthetic fluorescence signal and code1 is taken, it is 2α which corresponds to the integral value of D1, and when the inner product of the time series data D of the synthetic fluorescence signal and code2 is taken, it is 2β which corresponds to the integral value of D2. It can be calculated. Further, the inner product of the time series data D of the combined fluorescence signal and code3 becomes the integrated value 2α+2β+4γ of the combined fluorescence signal, on the other hand, the fluorescence signal intensity 2α corresponding to the previously obtained illumination light L1 and the fluorescence corresponding to the illumination light L2. By subtracting the signal intensity 2β, the fluorescence signal 4γ corresponding to the illumination light L3 can be obtained.

Fluorescence corresponding to the illumination light L1 of the first modulation pattern can be demodulated by multiplying the intensity of the detected synthetic fluorescence by the demodulation signal code1 synchronized with the modulation and integrating. Further, by multiplying the intensity of the detected synthetic fluorescence by the demodulation signal code2 timing-synchronized with the modulation and integrating, the fluorescence corresponding to the illumination light L2 of the second modulation pattern can be demodulated. Then, the fluorescence corresponding to the illumination light L3 of the pattern that is not modulated is demodulated by subtracting the fluorescence corresponding to the first and second modulation patterns from the integrated value of the intensity of the detected combined fluorescence. You can

The combination of the illumination lights L1 and L2 having the two modulation patterns, and the combination of the illumination lights L1 and L2 having the combined modulation pattern and the illumination light L3 having the unmodulated pattern are reflected by the mirrors 21a and 21b, respectively. It is supposed to be performed in combination with the triangular prism 22. This makes it possible to efficiently combine the three or more illumination lights L1, L2, and L3. In this case, the relative movement of the mirrors 21a and 21b and the triangular prism 22 in the direction of the arrow D can change the distance between the illumination lights L1, L2, and L3.

Further, when the light is branched into three optical paths, the demodulation code corresponding to the illumination lights L1 and L2 has a modulation pattern in which the demodulation codes are orthogonal to each other. However, the present invention is not limited to this and corresponds to the illumination lights L1 and L2. A demodulation code having a modulation pattern that is not orthogonal to each other may be used.

Specifically, the demodulation signal code1=(1,0,0,−1) of the modulation pattern corresponding to one electro-optic modulator 8a, and the demodulation signal code2=of the modulation pattern corresponding to the other electro-optic modulator 8b. (0,1,0,-1), demodulation signal code3=(1,1,1,1) having a pattern that is not modulated by the electro-optic modulators 8a and 8b. In this case, the light modulation corresponding to 0 is OFF.
The fluorescence signal corresponding to one pulse of the illumination light L1 is α, the fluorescence signal corresponding to one pulse of the illumination light L2 is β, and the fluorescence signal corresponding to one pulse of the illumination light L3 is γ.

Further, when the exposure time is four pulses, the time series data D1 of the fluorescence signal during the exposure time corresponding to the illumination light L1 becomes (α,0,0,0). Then, the time series data D2 of the fluorescence signal during the exposure time corresponding to the illumination light L2 becomes (0, β, 0, 0), and the time series data D3 of the fluorescence signal during the exposure time corresponding to the illumination light L3 becomes ( γ, γ, γ, γ). Therefore, the time series data D of the synthetic fluorescence signal is D=D1+D2+D3=(α+γ, β+γ, γ, γ).

When the inner product of the time series data D of the synthetic fluorescence signal and code1 is taken, α corresponding to the integrated value of D1 is obtained, and when the inner product of the time series data D of the synthetic fluorescence signal and code2 is taken, β corresponding to the integrated value of D2 is obtained. It can be calculated. Further, the inner product of the time series data D of the combined fluorescence signal and code3 becomes the integrated value α+β+4γ of the combined fluorescence signal, on the other hand, the fluorescence signal intensity α corresponding to the previously obtained illumination light L1 and the fluorescence corresponding to the illumination light L2. By subtracting the signal intensity β, the fluorescence signal 4γ corresponding to the illumination light L3 can be obtained.

Further, in the present embodiment, a resonant electro-optic modulator (not shown) may be adopted instead of the electro-optic modulator 8. Since the resonant electro-optic modulator can be driven at a low voltage, there is an advantage that the power amplifier is unnecessary and the cost can be kept low. By making the resonance cycle of the resonance electro-optic modulator the same as the repetition cycle, an arbitrary pattern can be created during the exposure time.

Further, the intensity of the light on one optical path is modulated by the electro-optic modulator 8, but instead of this, as shown in FIG. 10, an acousto-optic deflecting element 23, a lens 24, and a prism 25 are provided. Alternatively, the mirrors 26a and 26b and the beam splitter 27 may be combined to modulate to a beat frequency due to interference of light having different frequencies.

For example, in the example shown in FIG. 10, the light that is not frequency-modulated (pattern that is not modulated at the light source frequency YMHz) L1 and the light that is frequency-modulated to the frequency XMHz (first modulation pattern) L2a by the acousto-optic deflection element 23. , Light (second modulation pattern) L2b frequency-modulated to the frequency X+40 MHz is generated. The generated lights L2a and L2b having the first and second modulation patterns are combined by the beam splitter 27 after being guided through the lens 24, the prism 25, and the mirrors 26a and 26b through different optical paths. Due to the interference, the illumination light L2 having a difference frequency (40 MHz) between the two is generated. The mirror 26a is provided so as to be movable in the direction of arrow E in order to adjust the optical path length.

This makes it possible to irradiate different positions on the sample A with the illumination light L1 of the pattern that is not modulated at the frequency YMHz and the illumination light L2 of the modulation pattern at the frequency 40MHz.
In this case, the signal control unit 12 can control the driver 28 of the acousto-optic deflection element 23 to arbitrarily adjust the frequencies of the lights L2a and L2b of the two modulation patterns, and the emission direction can be changed according to the frequency. By utilizing the difference, the irradiation position of the illumination light L2 on the sample A can be arbitrarily adjusted by the electrical control of the acousto-optic deflection element 23.

Further, as shown in FIG. 11, it is possible to irradiate the sample A with illumination light L1, L2, L3 having three types of modulation patterns and illumination light L4 having a pattern that has not been modulated in total at the same time. ..
For example, in the example shown in FIG. 11, two sets of acousto-optic deflection elements 23a and 23b, lenses 24a and 24b, prisms 25a and 25b, a mirror 26, and beam splitters 27a and 27b are used.

Of the light from the light source 2, one polarization direction, for example, P-polarized light, is incident on the one acousto-optic deflecting element 23a. The acousto-optic deflector 23a splits the incident light into four lights of XMHz, YMHz, X+20MHz and Y+20MHz. The branched light passes through the lens 24a, the prism 25a, and the mirror 26, respectively, and the XMHz light and the X+20MHz light, the YHMz light and the Y+20MHz light are multiplexed by the beam splitter 27a, respectively, and interfere with each other. To be made. Thereby, two illumination lights L1 and L2 having a beat frequency of 20 MHz are generated. The two illumination lights L1 and L2 are adjusted in phase by 90° on the beat frequency by adjusting the phase of the drive voltage waveform input to the driver 28 of the acousto-optic deflection element 23a. This makes it possible to create patterns that are orthogonal to each other.

The other acousto-optic deflecting element 23b receives light of another polarization direction, for example, S-polarized light, of the light from the light source 2. The acousto-optic deflecting element 23b splits the incident light into three lights of Ω=80 MHz, Z+40 MHz and ZMHz. The branched lights pass through the lens 24b, the prism 25b, and the mirror 26, respectively, and are caused to interfere with each other by the Z+40 MHz light and the Z MHz light being combined by the beam splitter 27b. As a result, the illumination light L3 having the beat frequency of 40 MHz is generated. The light of Ω MHz is guided to the beam splitter 27b as it is through the lens 24b, the prism 25b, and the mirror 26, and becomes the illumination light L4 of the pattern which is not modulated.

These four illumination lights L1 to L4 are combined by the polarization beam splitter 29 via the mirror 36, and irradiated on different positions of the sample A.
By doing so, there is an advantage that the four illumination lights L1 to L4 are multiplexed and applied to the sample A, and the frame rate can be further improved.

Further, as shown in FIG. 12, when the electro-optic modulator 8 modulates with 50% duty, the light blocked by the polarization element in the electro-optic element may be used for multiplexing.
That is, since the two illumination lights L1 and L2 whose phases are inverted are emitted from the electro-optic modulator 8, the optical path length of one illumination light L2 is adjusted to reduce the time delay of half the oscillation period of the laser light. By giving the illumination light, the illumination lights L1 and L2 having different detection timings can be generated. Further, the beam splitter 30 is arranged in the optical path not having the electro-optical modulator 8 and branched, and the optical path length of the branched light is adjusted to give a time delay of half of the oscillation cycle of the laser light, thereby detecting timing. It is possible to generate illumination lights L3 and L4 different from each other.

Specifically, as shown in FIGS. 13A and 13F, it is assumed that the light source 2 emits pulsed laser light having a constant peak intensity at a frequency of 80 MHz.
When the electro-optic modulator 8 modulates the duty to 50%, the first illumination light L1 that is turned on and off at 40 MHz shown in FIGS. 13B and 13G and the second phase-inverted second illumination light L1 shown in FIGS. 13C and 13H. The two illumination lights L1 and L2 of the illumination light L2 are generated.
On the other hand, the light passing through the optical path not having the electro-optic modulator 8 is branched by the beam splitter 30, so that the light is turned on and off at a frequency of 80 MHz as shown in FIGS. 13D, 13E, 13I and 13J. Third and fourth illumination lights L3 and L4 are generated.

Then, for the second illumination light L2 and the fourth illumination light L4, by adjusting the optical path length so as to give a time delay of 6.25 ns, which is a half of the oscillation period of the laser light of 12.5 ns, FIG. 14H, the sample A is irradiated with the pair of the first illumination light L1 and the third illumination light L3 and the pair of the second illumination light L2 and the fourth illumination light L4 at different times. Therefore, the fluorescence generated corresponding to the pair of illumination lights L1 and L3 or the pair of illumination lights L2 and L4 can be temporally separated. Since the two illumination lights L1 and L3 and the illumination lights L2 and L4 forming each pair have a first-order independent relationship with each other, they can be demodulated in the same manner as above.

By doing so, it is possible to combine the first-order independence of the illumination lights L1 to L4 and temporal modulation, perform more multiplexing, and further improve the frame rate. .. Since the light bounced at the time of modulation in the electro-optical modulator 8 is also used, there is an advantage that energy can be efficiently used.

Further, as shown in FIG. 15, in the image acquisition device 1 for performing the modulation using the acousto-optic deflecting element 23 shown in FIG. 10, the light bounced when being combined by the beam splitter 27 is guided to the delay optical path. Therefore, it may be delayed and multiplexed in time. In the figure, reference numeral 31 is a λ/2 plate, reference numeral 32 is a mirror, reference numeral 33 is a relay lens, and reference numeral 34 is a folding mirror. The mirror 32 is located off the optical path of the light that has passed through the λ/2 plate 31, and is arranged at a position that reflects only the light that is reflected by the folding mirror 34.

Similarly to FIG. 10, the X+40 MHz light reflected by the beam splitter 27 and the X MHz light transmitted through the beam splitter 27 are combined to become illumination light L1 of 40 MHz, which is transmitted through the beam splitter 27. It is reflected by the polarization beam splitter 35 together with the illumination light L2 of Y=80 MHz.

The X+40 MHz light that has passed through the beam splitter 27 without being reflected and the X MHz light that has been reflected without passing through the beam splitter 27 are combined to become 40 MHz illumination light L3 and enter the delay optical path. To be done. In the delay optical path, the polarization direction is changed by the λ/2 plate 31, is deflected by the mirror 32 on the way through the relay lens 33 and the folding mirror 34, and is passed through the polarization beam splitter 35.

On the other hand, the illumination light L4 of Y=80 MHz reflected without passing through the beam splitter 27 has its polarization direction changed by the λ/2 plate 31 in the delay optical path and returns via the relay lens 33 and the folding mirror 34. It is deflected by the mirror 32 on the way and is passed through the polarization beam splitter 35.

Thereby, the pair of the illumination light L1 of 40 MHz and the illumination light L2 of 80 MHz which are primary independent of each other, and the pair of the illumination light L3 of 40 MHz and the illumination light L4 of 80 MHz which are primary independent of each other at different times. Are irradiated at different positions. As a result, there is an advantage that the first-order independence of the illumination lights L1 to L4 and the temporal modulation can be combined to perform more multiplexing, and the frame rate can be further improved. Since the light reflected by the beam splitter 27 is also used, there is an advantage that energy can be efficiently used.

Further, in each of the above-described embodiments, the epi-illumination type image acquisition device 1 has been described, but the present invention is not limited to this. For example, as shown in FIG. 16, sheet-like illumination lights L1 and L2 having mutually independent primary patterns are orthogonal to the optical axis of the objective lens 16 at positions separated in the direction along the optical axis of the objective lens 16. It may be applied to the light sheet microscope 40 in which the light beams are incident from the same direction.

In the example shown in FIG. 16, by switching the intensity of one of the illumination lights L1 and L2 for each frame of the camera 41 and scanning in two planes separated in the direction along the optical axis of the objective lens 16, the illumination light is illuminated. L1 and L2 are generated in a pseudo sheet form. When the light from the sample A at each scanning position is condensed by the microscope lens 42 and the imaging lens 43, and a plurality of images are obtained by the camera 41, for example, when two pulses are obtained during four exposures during the exposure time, , Four images are acquired for each pulse arrival, and two fluorescence images corresponding to each irradiation position are demodulated by calculating the images acquired by using the primary independence of the illumination lights L1 and L2. Can be obtained. By using this, it is possible to simultaneously capture the phenomena occurring at different places at the same time. Instead of the pseudo sheet-like illumination light, sheet-like illumination light generated using a cylindrical lens is adopted. May be.

1 Image Acquisition Device 2 Light Source 3 Illumination Light Generation Section 4 Illumination Optical System 5 Photo Detection Section 6 Demodulation Section (Signal Light Demodulation Section)
7, 7a, 7b First beam splitter (optical branching unit)
8,8a,8b Electro-optical modulator (modulator)
S1 illumination light generation step S2 irradiation step S3 light detection step S4 fluorescence demodulation step (signal light demodulation step)
L1, L2, L2a, L2b, L3, L4 Principal ray of illumination light A sample

Claims (7)

While simultaneously illuminating a plurality of regions, an image acquisition device that separates and acquires a local signal corresponding to each illumination region,
An illumination light generation unit that modulates the intensity of light emitted from a light source and has a pattern that is linearly independent of each other, and that generates a plurality of illumination lights including modulated illumination light and unmodulated illumination light that is not modulated. When,
An illumination optical system that irradiates different positions of a sample with the plurality of illumination lights generated by the illumination light generation unit,
A photodetector that detects a combined signal light obtained by combining signal lights generated at a plurality of irradiation positions irradiated with the illumination light by the illumination optical system and outputs a combined signal;
A modulated local signal corresponding to the modulated illumination light is separated from the combined signal output by the photodetector using time integration of a product of a demodulation signal corresponding to the modulated illumination light and the combined signal, An image acquisition apparatus comprising: a demodulation unit that separates a non-modulated local signal corresponding to the non-modulated illumination light by subtracting a sum of the modulated local signals from a time integration of a combined signal. The image acquisition device according to claim 1, wherein a time integral of a product of the demodulated signal that does not correspond to the modulated illumination light and the modulated local signal and the non-modulated local signal that does not correspond to the modulated illumination light is zero. The image acquisition apparatus according to claim 2, wherein the time integration of a product of the demodulated signal corresponding to the unmodulated illumination light and the combined signal is time integration of the combined signal. The image acquisition device according to claim 1, further comprising a modulator that modulates the intensity of light in at least one optical path of the plurality of illumination lights generated by the illumination light generator. Two or more modulators are provided,
The image acquisition device according to claim 4, wherein each of the modulators respectively generates the modulated illumination light having the demodulated signals that are orthogonal to each other. The image acquisition device according to claim 1, which has a function of arranging a plurality of the irradiation positions at arbitrary positions in the sample. An illumination light generation step of modulating the intensity of light emitted from a light source to generate a plurality of illumination lights having a primary independent pattern with respect to each other and including a modulated illumination light that is modulated and an unmodulated illumination light that is not modulated. When,
An irradiation step of irradiating different positions of a sample with the plurality of illumination lights generated by the illumination light generation step;
A light detection step of detecting a combined signal light obtained by combining the signal lights generated at a plurality of irradiation positions irradiated with the illumination light by the irradiation step and outputting a combined signal;
A modulated local signal corresponding to the modulated illumination light is separated from the combined signal output by the light detection step using time integration of a product of a demodulated signal corresponding to the modulated illumination light and the combined signal, And a signal light demodulation step of separating a non-modulated local signal corresponding to the non-modulated illumination light by subtracting a sum of the modulated local signals from a time integration of a combined signal.

Images