JP7780256B2 - Magnification observation device, magnified image observation method, magnified image observation program, computer-readable recording medium, and storage device - Google Patents

Description

The present invention relates to a magnification observation device, a magnified image observation method, a magnified image observation program, a computer-readable recording medium, and a device storing the program.

Optical microscopes using optical lenses and digital microscopes are used as magnification observation devices for enlarging and displaying subjects such as microscopic specimens and workpieces. Digital microscopes receive light that passes through an optical system and is reflected or transmitted from an object to be observed placed on a stage using an image sensor such as a CCD or CMOS that electrically reads each pixel arranged two-dimensionally, and displays the electrically read image on a display (see, for example, Patent Document 1). To move the observation field displayed on the display, some digital microscopes are equipped with a stage movement mechanism, such as an XY stage that can move in the X and Y directions, on the stage.

In addition to those that manually move the stage, such stage movement mechanisms have also been developed that include motorized stages for electrically moving the stage. To operate such motorized stages, the user specifies the destination of the stage using an input console such as a mouse or joystick. For example, when moving an XY stage, the user specifies the direction of movement of the stage by tilting the joystick in the desired direction, as shown in Figure 7. By using such motorized stages, the stage can be positioned faster and more accurately than when the stage is operated manually. Furthermore, with such digital microscopes, even when multiple observation locations are far apart, it is possible to move between them at high speed and position them with high precision.

On the other hand, there are cases where users want to observe the presence of scratches or chips along a specific shape, such as the edge of a blade or tool. In such cases, rather than specifying and moving between multiple observation locations, the field of view is moved along a specific shape, such as a line, for observation. For example, as shown in Figure 6, the edge of a cylindrical workpiece WK2 may be observed by moving the field of view along an arc. However, when observing with a microscope, the higher the magnification, the greater the relative movement of the observation field compared to the movement of the motorized stage. For this reason, when observing at high magnification, moving the stage may move more than intended, requiring the user to adjust the observation field to the desired observation position. Consider the case where a user uses a joystick to observe the edge of a cylindrical workpiece WK2 along an arc, as shown in Figure 6. In this case, the user must make fine adjustments in the X and Y directions to ensure that the arc does not deviate from the observation field. As such, when moving the observation field along a specific shape using conventional operating methods, the user is forced to perform tedious tasks while observing.

JP 2015-127770 A

One object of the present invention is to provide a magnification observation device, a magnified image observation method, a magnified image observation program, a computer-readable recording medium, and a device storing the same, which allow the user to move the field of view along a desired trajectory without requiring detailed operations.

Means for solving the problem and effects of the invention

A magnification observation device according to one aspect of the present invention comprises a stage unit for placing an observation object thereon; an objective lens unit arranged to face the observation object on the stage unit; a camera unit that captures an image of the observation object formed through the objective lens unit and generates image data representing the image; a display control unit that causes a display unit to display an image of an observation field including the observation object based on the image data generated by the camera unit; and a display control unit that controls the objective lens unit to change the position of the optical axis of the objective lens unit on the stage unit so that the observation field output to the display unit by the display control unit moves. The system includes a field of view movement mechanism that moves the optical axis of the lens unit and the stage unit relative to one another, a movement direction instruction unit that instructs the movement direction of the field of view movement mechanism in accordance with user input indicating the movement direction of the observation field on the display unit, a trajectory instruction unit that instructs trajectory information related to the definition of the movement direction of the observation field, a trajectory calculation unit that calculates a field of view movement trajectory for moving the observation field based on the trajectory information instructed by the trajectory instruction unit, and a movement control unit that controls movement of the field of view movement mechanism along the field of view movement trajectory calculated by the trajectory calculation unit in accordance with the movement direction instruction by the movement direction instruction unit. This configuration allows the user to easily move the observation field of view along the field of view movement trajectory calculated by the trajectory calculation unit without having to instruct detailed movement of the observation field using the movement direction instruction unit, thereby simplifying operation.

Another aspect of the present invention is a magnified image observation method in which an object to be observed placed on a stage unit is imaged by a camera unit via an objective lens unit and displayed on a display unit, and the optical axis of the objective lens unit and the stage unit are moved relatively by a field of view movement mechanism so that the position of the optical axis of the objective lens unit on the stage unit changes and the observation field of view output to the display unit moves. The method includes the steps of: a step in which a trajectory instruction unit prompts the user to specify trajectory information relating to the direction of movement in which the user wants to move the observation field; a step in which a trajectory calculation unit calculates a field of view movement trajectory for moving the observation field of view based on the trajectory information specified by the trajectory instruction unit; and a step in which a movement control unit controls the movement of the field of view movement mechanism along the field of view movement trajectory calculated by the trajectory calculation unit in accordance with a user input indicating the direction of movement of the observation field of view on the display unit and in accordance with a movement direction instruction by a movement direction instruction unit that specifies the movement direction of the field of view movement mechanism. This allows the user to easily move the observation field of view along the field of view movement trajectory calculated by the trajectory calculation unit, without having to give detailed instructions for moving the observation field of view using the movement direction instruction unit, simplifying operation.

Furthermore, a magnified image observation program according to another aspect of the present invention includes a stage unit for placing an object to be observed, an objective lens unit arranged to face the object to be observed on the stage unit, a camera unit that captures an image of the object to be observed formed through the objective lens unit and generates image data representing the image, a display unit that displays an image of an observation field including the object to be observed based on the image data generated by the camera unit, a field of view movement mechanism that moves the optical axis of the objective lens unit relative to the stage unit so that the position of the optical axis of the objective lens unit on the stage unit changes and the observation field output to the display unit moves, and a front This magnified image observation program operates a magnified observation device equipped with a movement direction indicator that indicates the movement direction of the field of view movement mechanism in accordance with user input on the display unit indicating the movement direction of the observation field. The magnified image observation program causes a computer to implement a trajectory indicator function that indicates trajectory information related to the specification of the movement direction of the observation field, a trajectory calculation function that calculates a field of view movement trajectory for moving the observation field based on the trajectory information indicated by the trajectory indicator function, and a movement control function that controls movement of the field of view movement mechanism along the field of view movement trajectory calculated by the trajectory calculation function in accordance with the movement direction indicated by the movement direction indicator. With the above configuration, the user can easily move the observation field of view along the field of view movement trajectory calculated by the trajectory calculation unit without having to specify detailed movement instructions for the observation field of view using the movement direction indicator, thereby simplifying operation.

Furthermore, according to another aspect of the present invention, a computer-readable recording medium or recorded device stores the above-mentioned program. Recording media include magnetic disks such as CD-ROM, CD-R, CD-RW, flexible disks, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD±R, DVD±RW, HD DVD (AOD), Blu-ray (product name), UHD BD (product name), USB memory, SSD memory, optical disks, magneto-optical disks, semiconductor memory, and other media capable of storing programs. Programs distributed by downloading over a network such as the Internet, as well as those stored on the above-mentioned recording media, are also included. Recording media also include devices capable of recording programs, such as general-purpose or dedicated devices on which the above-mentioned program is implemented in an executable form, such as software or firmware. Furthermore, each process and function included in the program may be performed by computer-executable program software, or the processing of each part may be realized by hardware such as a specified gate array (FPGA, ASIC), or by a combination of program software and partial hardware modules that realize some of the hardware elements. In this specification, computer-readable recording media also includes non-transitory tangible media and transient propagated signals.

1 is a perspective view of the appearance of a magnification observation device according to a first embodiment. FIG. 2 is a block diagram of the magnification observation device of FIG. 1. FIG. 2 is a schematic diagram illustrating a configuration of an illumination unit. FIG. 1 is a schematic diagram of a ring illumination. FIG. 2 is a schematic diagram showing the tilt direction of the joystick in a side view. FIG. 2 is a schematic side view showing the tilt angle of the joystick in a plan view. 10 is a schematic diagram showing the direction of movement of a joystick when moving the observation field along the edge of a cylindrical workpiece. FIG. FIG. 10 is a schematic diagram showing how the direction of movement of the stage is instructed with a joystick. 10A and 10B are schematic diagrams showing the tilt direction of the joystick and the movement direction of the observation field in the route tracing mode. FIG. 10 is a schematic diagram showing an example in which a movement direction indicator is realized by button operation. FIG. 10 is a schematic diagram showing an example in which a movement direction indicator is implemented by mouse operation. 11A and 11B are schematic diagrams showing the relationship between a visual field movement trajectory and a movement direction instruction. 10 is a flowchart showing a method for observing an enlarged image that realizes a route guide function. 13A-13F are perspective views illustrating example route-defining geometries. FIG. 10 is a schematic diagram showing how a field of view movement locus is set for chips mounted at a distance from each other on a substrate. 15 is a schematic diagram showing a state in which the observation object in FIG. 14 is moved along a field of view movement locus set thereon. FIG. 10 is a user interface screen of a magnified image observation program for operating the magnification observation device. 10 is a user interface screen of a magnified image observation program for operating the magnification observation device. 10 is a user interface screen of a magnified image observation program for operating the magnification observation device. FIG. 19A shows a wide-area image of an object to be observed, and FIG. 19B is a schematic diagram showing a state in which the contours of FIG. 19A are extracted. FIG. 20A is a schematic diagram showing the observation field while the XY stage is being moved, and FIG. 20B is a schematic diagram showing the observation field while the XY stage is being further moved. 21A is an image showing the observation field before switching the objective lens unit, FIG. 21B is the observation field after switching the objective lens unit, and FIG. 21C is an image showing the observation field after correcting the deviation of the field center from the state of FIG. 21B. 10A and 10B are schematic diagrams showing how a field of view movement trajectory is offset by a field of view deviation correction offset function. FIG. 23 is a schematic diagram showing how the field of view movement locus is offset in accordance with the position corresponding to the observation field of view. FIG. 10 is a schematic diagram showing a state in which a position away from the visual field movement trajectory is specified and the route tracing mode is cancelled; FIG. 25 is a schematic diagram showing a state in which the route trace mode in FIG. 24 is cancelled and a free mode is entered. FIG. 10 is a schematic diagram showing how the system returns to the route tracing mode after specifying a position close to the visual field movement trajectory. FIG. 10 is a schematic diagram of a user interface screen showing an example of a release condition setting section. Figure 28A is an image showing a state in which a field of view movement trajectory is displayed superimposed on a low-magnification image displayed in the entire navigation display area, and Figure 28B is an image showing a state in which a low-magnification image is displayed in part of the navigation display area and a field of view movement trajectory is displayed superimposed on it. FIG. 29A is a schematic diagram showing the route display area when the route guide function is being executed, and FIG. 29B is a schematic diagram showing the route display area when the route guide function is stopped. FIG. 10 is a schematic diagram showing a magnification observation device according to a second embodiment. 31A to 31D are schematic diagrams showing how the lighting direction changes with the search lighting function. FIG. 32A is a schematic diagram showing an example of acquiring three-dimensional information of an observation object from a low-magnification image, and FIG. 32B is a schematic diagram showing an example of estimating the inclination of a plane of an observation object. FIG. 10 is a schematic diagram showing an example of a profile obtained by acquiring a 3D shape from a low-magnification image. FIG. 10 is a schematic diagram showing the route guide function when XY movement is stopped. Figure 35A is a schematic diagram showing a field of view movement trajectory and reference point specified in three-dimensional space, Figure 35B is a schematic diagram showing the reference point projected onto the XY plane, and Figure 35C is a schematic diagram showing a field of view movement trajectory defined by the projected reference point. FIG. 36A is a plane defined in three-dimensional space, and FIG. 36B is a schematic diagram showing the state in which the visual field movement trajectory within this plane is projected onto the XY plane. FIG. 37A is a schematic diagram showing a polygonal view movement trajectory defined in three-dimensional space, and FIG. 37B is a schematic diagram showing the state in which this polygonal view movement trajectory is projected onto the XY plane. FIG. 10 is a schematic diagram showing how the visual field movement direction is determined when visual field movement trajectories intersect. FIG. 10 is a schematic diagram showing how a route guide function is executed on visual field movement trajectories that intersect at a twist position. 10 is a schematic diagram showing how a route guide function is executed for a visual field movement trajectory on which adjacent registration points are set. FIG. 10A and 10B are schematic diagrams showing an example of an operation when moving in the Z direction while a route guide function is being executed. 10A and 10B are schematic diagrams showing another example of the operation when moving in the Z direction while the route guide function is being executed. 43A is a schematic diagram showing a circular object to be observed, FIG. 43B is a schematic diagram showing a state in which the entire view of FIG. 43A has been acquired, and FIG. 43C is a schematic diagram showing a state in which a field of view movement trajectory has been set for FIG. 43B.

The following describes embodiments of the present invention with reference to the drawings. However, the embodiments described below exemplify a magnification observation device, a magnification image observation method, a magnification image observation program, a computer-readable recording medium, and a device storing the same, which embody the technical concepts of the present invention. The present invention does not limit the magnification observation device, the magnification image observation method, the magnification image observation program, the computer-readable recording medium, or the device storing the same to those described below. Furthermore, this specification does not limit the components set forth in the claims to those described in the embodiments. The dimensions, materials, shapes, and relative positions of components described in the embodiments, unless otherwise specified, are not intended to limit the scope of the present invention and are merely illustrative examples. Note that the size and relative positions of components shown in the drawings may be exaggerated for clarity. Furthermore, in the following description, the same names and symbols indicate components that are identical or of similar quality, and detailed description will be omitted as appropriate. Furthermore, the elements constituting the present invention may be configured with the same component, with a single component fulfilling multiple functions, or conversely, the function of a single component may be shared by multiple components.

The magnification observation device used in the embodiments of the present invention is connected to computers, printers, external storage devices, and other peripheral devices connected thereto for operation, control, display, and other processing, via serial or parallel connections such as IEEE 1394, RS-232x, RS-422, or USB, or via networks such as 10BASE-T, 100BASE-TX, or 1000BASE-T, and is electrically, magnetically, or optically connected for communication. The connection is not limited to a physical connection using a wire; it can also be a wireless connection using radio waves, infrared light, or optical communication such as a wireless LAN such as IEEE 802.x or Bluetooth (registered trademark). Furthermore, storage media for data exchange and setting storage can include memory cards, magnetic disks, optical disks, magneto-optical disks, and semiconductor memory. Note that in this specification, the terms "magnification observation device" and "magnified image observation method" refer not only to the magnification observation device itself, but also to a magnification observation system that combines it with peripheral devices such as a computer and external storage device.

Furthermore, in this specification, the term "magnification observation device" is not limited to the system itself that performs magnification observation, nor to devices or methods that perform input/output, display, calculation, communication, and other processing related to imaging using hardware. Devices and methods that implement processing using software are also within the scope of the present invention. For example, devices and systems that incorporate software, programs, plug-ins, objects, libraries, applets, compilers, modules, macros that run on specific programs, etc. into general-purpose circuits or computers to enable imaging itself or related processing also fall under the category of the magnification observation device of the present invention. Furthermore, in this specification, the term "computer" includes not only general-purpose and dedicated electronic computers, but also workstations, terminals, and other electronic devices. Furthermore, in this specification, the term "program" is not limited to a standalone program, but can also be used in the following ways: as part of a specific computer program, software, service, etc.; ...
[Embodiment 1]

A magnification observation device 100 according to a first embodiment of the present invention will be described below with reference to FIGS. 1 and 2. As shown in FIG. 1, the magnification observation device 100 is broadly divided into an imaging system 1 and a control system 2. The imaging system 1 includes an illumination unit 60 for illuminating an observation object WK, such as a sample, workpiece, or other subject, and a head unit 4 for capturing an image of the observation object WK illuminated by the illumination unit 60. As shown in FIG. 2, the head unit 4 includes a camera unit 10 including an image sensor 12 and a microscope lens unit 20 detachably attached to the tip of the camera unit 10. The microscope lens unit 20 constitutes an imaging optical system (lens optical system) composed of multiple optical lenses. Here, the microscope lens unit 20 includes an objective lens unit 25. The head unit 4 also functions as an imaging means for receiving reflected or transmitted light of the illumination light.
(Camera unit 10)

As shown in FIG. 2, the camera unit 10 includes an image sensor 12 that electrically reads reflected light incident via the imaging optical system 11 from the observation object WK illuminated by the illumination unit 60. In this example, the image sensor 12 uses a CMOS, but other light-receiving elements such as a CCD can also be used. The objective lens unit 25 is positioned facing the observation object on the stage unit 30. The camera unit 10 also captures an image of the observation object formed via the objective lens unit 25 and generates image data representing this image.

The imaging system 1 also includes a stage unit 30 on which the observation object WK is placed, an upper Z stage as a first focus adjustment unit that adjusts the focus by changing the relative distance in the optical axis direction between this stage unit 30 and the head unit 4, and an upper Z elevator 16 for driving this upper Z stage. Light is incident on the observation object WK placed on this stage unit 30 via the imaging optical system 11, and reflected light reflected by the observation object WK or transmitted light irradiated from the bottom side of the observation object WK is electrically read by the image sensor 12 of the camera unit 10.

The control system 2 further includes a main body 50 having a display 70 that displays an enlarged image captured by the camera 10. The camera 10 is connected to the main body 50 via the cable 3. The display 70 displays an image of the observation field including the object to be observed based on image data generated by the camera 10. In the example of Figure 1, the display 70 is provided integrally with the main body 50, but the display can also be a separate component from the main body. For example, the main body 50 can include a display connection interface that connects the display 70 to the main body 50, in addition to a display control unit 52 that generates the content to be displayed on the display 70.

The main body 50 also includes a processor 51. As shown in the block diagram of Figure 2, the processor 51 performs multiple functions (details will be described later). The main body 50 can be a general-purpose computer with a dedicated program installed, or a specially designed device. In this example, a general-purpose computer with a magnified image observation program for operating the magnification observation device installed is used as the main body. The main body 50 includes a processor 51, a display control unit 52, a storage unit 53, an interface 54, an operation unit 55, and a memory unit 56.

The cable unit 3 includes an electrical cable for transmitting image information obtained by the image sensor 12 of the camera unit 10 to the body unit 50, as well as an optical cable 3b for transmitting illumination light from the body unit 50 to the head unit 4. The cable unit 3 can be integrated with the electrical cable and the optical cable 3b, or these can be provided separately.
(Display unit 70)

The display unit 70 can be a monitor such as an LCD display, organic EL display, or CRT. The main body unit 50 is also connected to an operation unit 55 that enables the user to perform various operations. The operation unit 55 is an input device such as a console or mouse. In this example, the display unit 70 and operation unit 55 can be integrated into the main body unit 50 or can be external components. Furthermore, if the display unit 70 is configured as a touch panel, the display unit 70 and operation unit 55 can be configured as a single unit.

The operation unit 55 is connected to the main body 50 or a computer via a wired or wireless connection, or is fixed to the computer. Typical operation units 55 include various pointing devices, such as a mouse, keyboard, slide pad, TrackPoint, tablet, joystick, console, jog dial, digitizer, light pen, numeric keypad, touchpad, and AccuPoint. These operation units 55 can be used not only to operate the magnification observation operation program, but also to operate the magnification observation device itself and its peripheral devices. Furthermore, a touch screen or touch panel can be used for the display itself that displays the interface screen, allowing the user to input and operate the screen by directly touching it with their hands, or voice input or other existing input means can be used, or both. In the example of FIG. 1 , the operation unit 55 is composed of a mouse, keyboard, and joystick 55b.
(Illumination unit 60)

The illumination unit 60 generates illumination light that illuminates the observation object WK that is imaged on the image sensor 12. A schematic configuration of the illumination unit 60 is shown in Figure 3A. The illumination unit 60 is equipped with an illumination control unit 66. The illumination control unit 66 controls the illumination light according to set illumination conditions. The illumination conditions adjust the brightness of the illumination light emitted from the illumination unit 60. For example, the brightness of the illumination light can be adjusted by adjusting the irradiation time or irradiation intensity of the illumination unit 60. The illumination unit 60 may include an LED as a light source. The light emission of the light source may be controlled by the illumination control unit 66, or the brightness of the irradiated light may be controlled by a light-blocking member such as a shutter.

The illumination light source 65 is built into the main body 50, and illumination light is transmitted to the illumination unit 60 in the head 4 via the optical cable 3b. The illumination unit 60 can either be built into the head 4 or be a separate unit that can be detached from the head 4. Furthermore, illumination methods such as epi-illumination and transmitted illumination can be used as appropriate. Epi-illumination is an illumination method in which illumination light falls from above the object being observed, and includes ring illumination and coaxial illumination.

Coaxial lighting can also be equipped with oblique lighting. An example of a coaxial lighting and ring lighting with such oblique lighting functionality is shown in the schematic diagram of Figure 3B. The lighting unit 60 shown in this figure comprises a circular coaxial epi-illumination unit 62 located on the inner ring side and a similarly circular ring lighting unit 63 located on the outer ring side. Both the coaxial epi-illumination unit 62 and the ring lighting unit 63 are divided into sections along the circumference, and by switching the lighting of each divided block, it is possible to provide oblique lighting with different lighting directions.

In this way, the illumination unit 60 can switch the illumination direction toward the observation object. The illumination unit 60 can also irradiate with a first illumination pattern that sequentially switches the illumination direction and irradiates with light, and a second illumination pattern that irradiates with light in a selected illumination direction. Such switching of illumination patterns is performed by the illumination control unit 66.
(Lighting control unit 66)

When the observation field of view is moved along the field of view movement trajectory, the lighting control unit 66 causes the lighting unit 60 to illuminate with a first lighting pattern. This allows the lighting direction to be automatically changed while the observation field of view is moving, making it possible to observe the surface condition of the object under observation with different lighting, making it easier to find scratches, chips, etc. Furthermore, when the lighting control unit 66 stops the movement of the observation field of view by the field of view movement mechanism 5, it analyzes multiple image data with different lighting directions stored in the buffer memory 57, selects the image with the lighting direction that makes the flaw most clear, and causes the lighting unit 60 to illuminate with a second lighting pattern fixed to that lighting direction. As a result, when the movement of the observation field of view is stopped, image data with the lighting direction that makes the flaw most visible is displayed, enabling observation that is ideal for flaw detection.

The illumination unit 60 shown in FIG. 1 includes a coaxial incident illumination unit 62 (see FIG. 3A) for illuminating the observation object WK with coaxial incident light, and a ring illumination unit 63 for illuminating ring-shaped illumination light from a ring-shaped light source. These illumination units are connected to the main body 50 via the optical cable 3b. The main body 50 has a connector for connecting the optical cable 3b and a built-in illumination light source 65 for transmitting light to the optical cable 3b via the connector (see FIG. 3A). The ring illumination unit 63 can switch between full-circle illumination and oblique illumination. To achieve this, the ring illumination unit 63 can be configured with multiple LEDs arranged in a ring shape and some of the LEDs can be turned on and off, or with a turret-type mask that blocks part of the illumination light. The illumination control unit 66 controls the illumination and switching of these illumination units.

As shown in the schematic cross-sectional view of Figure 3A, the illumination unit 60 has a built-in light source on the control system 2 (e.g., main body 50) side, and employs a configuration in which illumination light is transmitted to the head unit 4 on the imaging system 1 side via an optical fiber or the like. The illumination unit 60 includes a coaxial epi-illumination unit 62 and a ring illumination unit 63, and the coaxial epi-illumination is effective for viewing unevenness in mirror-finished workpieces such as silicon wafers and LCD panels. The lighting of the illumination unit 60 is controlled by an illumination control unit 66.
(Illumination light source 65)

The illumination light source 65 can be a semiconductor light-emitting element such as a light-emitting diode (LED) or a semiconductor laser (LD). LEDs, in particular, have excellent ON/OFF response, which improves measurement throughput. They also have long life, low power consumption, low heat generation, and resistance to mechanical shock. Alternatively, a light source can be used that utilizes wavelength conversion materials, such as phosphors, that are excited by the ultraviolet or visible light of the source light. Furthermore, LEDs that can emit ultraviolet or infrared light in addition to visible light can also be used as the light source. For example, infrared observation is useful for analyzing defective products and tissue distribution in biological tissue. The illumination light source is not limited to semiconductor light-emitting elements. White light sources that emit white light over a wide wavelength range, such as halogen lamps, xenon lamps, and HID lamps, can also be used. Light sources that can emit both visible and infrared light are also acceptable. Halogen lamps are particularly preferable due to their wide emission wavelength range. In addition to using a single light source, multiple light sources can be used, and they can be lit simultaneously to produce a mixed color illumination, or they can be switched between different colors.

The illumination light source is not limited to being built into the main body. For example, it can be provided in the stage or microscope lens. That is, a transmitted illumination light source may be provided on the stage side as an illumination light source, and an illumination light source for coaxial epi-illumination or ring illumination may be provided on the microscope lens side. This configuration eliminates the need to transmit illumination light from the main body side to the head side using optical fibers or the like, thereby offering the advantage of simplifying the configuration by reducing the number of cables drawn to the outside. Furthermore, inside the head side, light from the illumination light source may be branched using optical fibers, or a semiconductor light-emitting element such as a high-intensity LED may be provided for direct illumination. In particular, LEDs are smaller, generate less heat, have a longer life, and are maintenance-free compared to conventional halogen lamps and the like.
(Field of view movement mechanism 5)

The magnification observation device 100 also includes a field-of-view movement mechanism 5 that moves the observation field displayed on the display unit 70. The field-of-view movement mechanism 5 changes the relative position of the objective lens unit 25 and the stage unit 30 to change the position of the optical axis AX of the objective lens unit 25 on the mounting surface of the stage unit 30. When the field-of-view movement mechanism 5 moves the relative position of the objective lens unit 25 and the stage unit 30, an updated image can be displayed on the display unit 70 in the observation field after the movement. In the example of Figure 1, an XY stage that moves the stage unit 30 is used as the field-of-view movement mechanism 5. However, the present invention may also use a field-of-view movement mechanism that moves the objective lens unit instead of, or in addition to, this. It is sufficient for the field-of-view movement mechanism to be able to move the relative position of the objective lens unit and the stage unit as viewed from the optical axis side, i.e., the observation field within the XY plane.

The XY stage is an electrically powered stage that can move the mounting surface of the stage unit 30 in the X-axis and Y-axis directions. The field of view movement mechanism 5 can also be equipped with a θ stage that can rotate the stage unit 30.

Furthermore, in addition to being able to move the stage unit 30 within the XY plane using the field of view movement mechanism 5, the stage unit 30 can also be moved in the height direction, i.e., in the Z direction, using the lower stage elevator 35.

The operation of the lower stage elevator 35 will now be described. The main body 50 changes the relative distance in the optical axis direction between the stage 30 and the head 4, which has the imaging optical system 11 and imaging element 12, in this case the height in the z direction, by inputting control data related to the control of the stepping motor 37 to the motor control circuit 36. Specifically, the main body 50 controls the rotation of the stepping motor 37 by inputting control data necessary for control of the lower stage elevator 35 to the motor control circuit 36, thereby raising and lowering the height z (position in the z direction) of the stage 30. The stepping motor 37 generates a rotation signal corresponding to the rotation. Based on the rotation signal input via the motor control circuit 36, the main body 50 stores the height z of the stage 30 as information related to the relative distance in the optical axis direction between the stage 30 and the imaging optical system 11. The stage 30 functions as an observation positioning device that positions the observation position relative to the observation object WK.

Furthermore, in this embodiment, changing the height of the stage unit 30 not only changes the relative distance between the stage unit 30 and the imaging optical system 11 in the optical axis direction, but also changes the height of the imaging optical system, i.e., the height of the head unit 4. The head unit 4 is connected to the main body unit 50 by the cable unit 3. As a result, data acquired by the head unit 4 is sent to the main body unit 50 via the cable unit 3, and the necessary processing can be performed on the main body unit 50 side. Note that the stage unit may be provided on the microscope main body, or on the head unit as a separate component from the main body, or an imaging unit without a stage may be provided on the head unit. An imaging unit without a stage can be attached to a mounting stand or handheld by the user.

The image sensor 12 can electrically read the amount of received light for each pixel arranged two-dimensionally in the x and y directions. The image of the object of observation WK formed on the image sensor 12 is converted into an electrical signal according to the amount of received light at each pixel of the image sensor 12, and then further converted into digital data by the image sensor control circuit 13. The main body 50 stores the digital data converted by the image sensor control circuit 13 as received light data D in the storage unit 53 along with pixel arrangement information (x, y) as two-dimensional position information of the object of observation WK in a plane (x, y directions in Figure 2) approximately perpendicular to the optical axis direction (z direction in Figure 2). Here, the plane approximately perpendicular to the optical axis direction does not necessarily have to be a plane strictly at a 90° angle with the optical axis; it can be any observation plane within a range of inclination that allows the shape of the object of observation WK to be recognized with the imaging optical system and the resolution of the image sensor 12.

In the above explanation, an example of the stage unit 30 has been given in which the observation object WK is placed on the stage unit 30, but it is also possible to use a configuration in which, for example, an arm is attached instead of the stage unit and the observation object WK is fixed to the tip of the arm. Furthermore, the head unit 4 is not only used by being attached to the camera attachment unit 43, but can also be detachable and placed at a desired position and angle by being held by hand, etc.
(Moving direction instruction section 55a)

The control system 2 includes an operation unit 55. The operation unit 55 is an input device connected to the main body 50. The operation unit 55 functions as a movement direction indicator 55a that accepts user input indicating the movement direction of the observation field on the display unit 70. The movement direction of the field of view movement mechanism 5 is indicated according to the direction input from the operation unit 55. The operation unit 55 can be configured with a joystick, touchpad, mouse, keyboard (arrow keys or specific keys), etc. In particular, by using a joystick 55b as the operation unit 55, the user can intuitively indicate the movement direction of the observation field by tilting the joystick 55b. For example, this makes it easy to indicate forward, reverse, clockwise, counterclockwise, up, down, left, right, etc. The movement speed can also be specified by the angle at which the joystick 55b is tilted from a vertical position. Specifically, as shown in Figure 4, the movement speed increases as the tilt angle α in a side view increases, and decreases as the tilt angle α decreases. As shown in Figure 5, the tilt angle β of the joystick 55b in a planar view indicates the direction of movement of the observation field, but when the route guide function described below is in operation, this tilt angle β in a planar view can also be used to indicate the movement speed.

The direction of the user input received by the operation unit 55 and the direction of movement of the stage unit 30 or the objective lens unit 25 by the field of view movement mechanism 5 differ depending on the manner in which the operation unit 55 receives the user input and the object to be moved by the field of view movement mechanism 5. For example, if the operation unit 55 is a joystick 55b and the field of view movement mechanism 5 is an XY stage that moves the stage unit 30, the observation field of view is moved in the direction input by the user by moving the stage unit 30 on the XY stage in the direction opposite to the direction input by the operation unit 55. Furthermore, if the field of view movement mechanism 5 moves the objective lens unit 25, the direction of the user input and the direction of movement of the XY stage coincide.
In this example, an XY stage for moving the stage unit 30 is used as the visual field moving mechanism 5, and a joystick 55b is used as the movement direction instruction unit 55a.

Furthermore, when the lever of the joystick 55b is pressed, the stage unit 30 moves to the origin or initial position. Furthermore, when the coordinates of the destination of the stage unit 30 are specified, the route tracing mode is canceled and the stage unit 30 follows those coordinates.

Furthermore, in addition to being able to move in the height direction, i.e., the Z direction, using the lower stage elevator 35, the stage unit 30 can also be moved within a plane. Specifically, it is equipped with an XY stage that can move in the X-axis and Y-axis directions. It can also be equipped with a rotatable stage (θ stage) that rotates the stage unit 30.

In this example, both the upper Z elevator 16 and the lower stage elevator 35 are electrically driven. However, in the present invention, it is sufficient to obtain height information for the objective lens unit 25 and the stage unit 30, and it is not necessary for both the upper Z elevator and the lower stage elevator to be electrically driven. For example, either the upper Z elevator or the lower stage elevator may be configured to be driven manually.

As shown in the block diagram of Figure 2, the main body 50 includes a processor 51, a display control unit 52, a storage unit 53, an interface 54, an operation unit 55, and a memory unit 56. This magnification observation device 100 captures an observation image using an image sensor 12 that electrically reads reflected or transmitted light from an observation object WK fixed to a stage unit 30 that enters via an imaging optical system 11, and displays the image on a display unit 70.

The storage unit 53 functions as a memory unit, for example, by saving image data displayed on the display unit 70 by the display control unit 52 as a moving image. The interface 54 is a connection unit that allows the main body unit 50 to communicate data with the head unit 4, the lower stage elevator 35, etc. The memory unit 56 is composed of RAM, ROM, etc. This memory unit 56 includes a buffer memory 57 that sequentially stores image data captured by the camera unit 10 in different illumination directions while the field of view movement mechanism 5 is moving. The operation unit 55 is also a component for setting imaging conditions for setting conditions for capturing images with the camera unit 10, as well as for performing various other necessary settings and operations.
(Display control unit 52)

The display control unit 52 outputs image data generated by the camera unit 10 to the display unit 70. The display unit 70 displays image data of the observation field output from the display control unit 52. This display control unit 52 can be configured with a GPU or the like. The example in FIG. 2 illustrates an example in which the display control unit 52 is configured as a separate member from the processor unit. Such a display control unit 52 is configured with, for example, a GPU. However, the present invention is not limited to this configuration, and the display control unit 52 may be incorporated into the processor unit. For example, the display control unit 52 may be integrated into a CPU or MPU that constitutes a processor.
(Processor unit 51)

The processor unit 51 includes a trajectory instruction unit 81 for instructing trajectory information regarding the direction of movement of the observation field for image data displayed on the display unit 70; a trajectory calculation unit 82 for calculating a field of view movement trajectory for moving the observation field based on the trajectory information instructed by the trajectory instruction unit 81; a movement control unit 83 for controlling movement of the field of view movement mechanism 5 along the field of view movement trajectory calculated by the trajectory calculation unit 82 in accordance with the movement direction instruction from the movement direction instruction unit 55a; an image processing unit 84 for calculating the height in the optical axis direction of the observation object WK corresponding to the set region based on focal length information stored in the storage unit 53 for part or all of the observation object WK corresponding to the set region; and a reference coordinate acquisition unit 85. The reference coordinate acquisition unit 85 acquires the coordinate position of the center of the observation image displayed in the image display area 111 as a reference position and registers it as the coordinate of the reference point. Furthermore, as will be described later, if the field of view movement trajectory also includes z-coordinate information, the reference coordinate acquisition unit 85 may acquire not only x-y coordinates but also z-coordinates when registering the reference point. This magnification observation device 100 can calculate the average height (depth) in the optical axis direction of the observation object WK corresponding to the specified area using the image sensor 12.

The processor unit 51 can be configured with a general-purpose CPU, MPU, SoC, or a gate array such as an ASIC or FPGA customized for a specific application. While this example shows a configuration in which a single CPU serves as the processor unit and realizes multiple functions described below, the present invention is not limited to this configuration, and the processor unit may be configured with multiple CPUs. For example, the processor unit may be configured with a so-called multi-core MPU. In this case, each function may be realized by multiple cores, or different functions may be assigned to each core for execution. Furthermore, the processor unit may be configured with a combination of a CPU and a GPU. In this case, the GPU not only performs the functions of the display control unit 52 described above, but may also be configured to execute some or all of the functions assigned to the processor unit.
(Route guide function)

This magnification observation device 100 is equipped with a route guide function that makes it easy to move the observation field along a pre-set route. For example, consider an application in which the observation object is a cylindrical workpiece WK2 as shown in Figure 6, and the magnification observation device is used to observe burrs and scratches on the circumference of its end face. In this case, it is necessary to move the stage unit 30 so that the observation field displayed on the display unit 70 moves along the arc of the edge of the observation object.

When observing with a digital microscope, the magnification is high, making it somewhat difficult to move the observation field to the intended position. When manually moving the stage, it is common to move the X and Y axes separately. For example, separate knobs are provided for moving the stage in the X and Y directions, and the user operates each knob to move in the desired direction. This method requires separate operation for movement in the X and Y directions, making it difficult to move to the desired position. For example, it was necessary to observe by alternately moving the X and Y axes along the arc of the cylindrical workpiece WK2 shown in Figure 6.

However, with the evolution of digital microscopes, the use of motorized XY stages has been increasing in recent years. This makes it possible to move freely on the XY plane by operating the on-screen mouse or a joystick as shown in Figure 7.

However, when using mouse operations such as dragging or double-clicking, you need to perform the operation continuously to move long distances.

Furthermore, even when moving continuously in a specified direction using a joystick or similar, the field of view does not move strictly along the sample, but rather moves in a zigzag pattern along the desired route, as shown in Figure 6. While the observation field of view moves in a zigzag pattern, the user must continue to concentrate on the operation to ensure that it does not stray from the desired route, which can be very stressful.

On the other hand, you can also use the image stitching function to take photos in advance and check them later. However, in this case, it takes a lot of time from setting the shooting range to completing the shoot, and for users who just want to check for the presence or absence of burrs, the effort of taking photos and saving them each time can be a significant burden.

Therefore, the magnification observation device 100 according to this embodiment is equipped with a route tracing mode in which a route for moving the observation field is set in advance, and when the observation field is actually specified using the movement direction indicator 55a, the field of view movement mechanism 5 is controlled so that the observation field changes along the set route. By executing the route tracing mode, the user can move the observation field along the desired route, even if it is a complex path, simply by specifying a general direction using the movement direction indicator 55a. This allows the user to concentrate on the observation without being distracted by moving the stage unit 30. In the example shown in Figure 7 above, an arc-shaped route RT is set in advance to follow the cylindrical end face, as shown in Figure 8. Then, by simply tilting the joystick 55b, which serves as the movement direction indicator 55a, roughly to the right, the observation field can be moved along the arc. In Figure 8, the arrow DJ1 indicates the direction in which the joystick 55b is tilted. Normally, the observation field of view cannot be moved in an arc unless the tilt direction of the joystick 55b is gradually changed from right to upward along the tangent direction DS1 of the arc-shaped route. However, by executing the route tracing mode, it is possible to move the stage unit 30 in an arc along the route RT, i.e., in the direction of DS1, even while keeping the joystick 55b tilted in a fixed direction DJ1, as shown in FIG. 8. In other words, it is possible to move the optical axis AX of the objective lens unit 25 on the mounting surface of the stage unit 30, bending along the circumference as shown in FIG. 8, as shown in DS1, without gradually changing the tilt direction DJ1 of the joystick 55b according to the position of the arc. In this way, the route tracing mode allows movement control in which the tilt direction DJ1 of the joystick 55b and the actual movement direction DS1 of the stage unit 30 are deviated.

Furthermore, forward and reverse directions may be set in advance for the field of view movement trajectory and stored in memory unit 56 together with the field of view movement trajectory. In this case, movement is controlled so that when β in Figure 5 is tilted at an angle between 1° and 179°, it moves in the forward direction, and when it is tilted at an angle between 181° and 359°, it moves in the reverse direction. Note that the direction in which it moves when tilted at angles of 0° and 180° can be set as appropriate. The correspondence between the tilt angle of angle β and the forward and reverse directions is not limited to this example and may be set as appropriate.

Such operations are not limited to the joystick 55b. For example, virtual or physical movement buttons 55c, as shown in Figure 9, can be prepared, and instructions can be given by operating the buttons to move forward or backward, or to the right or down, up or down, or left or right. Furthermore, if the forward and reverse directions are preset for the field of view movement trajectory and stored in the memory unit 56 along with the field of view movement trajectory, buttons for specifying two movement directions, such as a forward button and a back button, are sufficient. In this case, pressing the forward button controls the movement mechanism so that the observation field moves forward, and pressing the back button controls the movement mechanism so that the observation field moves in the reverse direction. Furthermore, if a mouse with a wheel button is used as the movement direction instruction unit 55a, the direction of movement of the observation field can be specified, like with a joystick, by moving the mouse cursor MC, which indicates the direction of field movement, around an icon IC, which is displayed by clicking the mouse wheel, as shown in Figure 10.

As described above, the field of view movement trajectory calculated by the trajectory calculation unit 82 can include a curve. This has the advantage of allowing curved movement of the observation field, which has been troublesome in the past, to be easily performed. Furthermore, the field of view movement trajectory calculated by the trajectory calculation unit 82 can also include a bent portion. This has the advantage of allowing non-linear, bent movement of the observation field, which has been troublesome in the past, to be easily performed.
(Movement control unit 83)

The movement control unit 83 can switch between route trace mode and free mode as a control method for the field of view movement mechanism 5. In route trace mode, the movement direction indication unit 55a controls the field of view movement mechanism 5 so that it follows the field of view movement trajectory calculated by the trajectory calculation unit 82. In free mode, the field of view movement mechanism 5 is controlled in the movement direction indicated by the movement direction indication unit 55a or to a specified coordinate position, regardless of the field of view movement trajectory. This makes it possible to easily move the observation field of view along the field of view movement trajectory in route trace mode, while also allowing the user to freely move the observation field of view in free mode.

The movement control unit 83 realizes smooth movement of the field of view movement function of the stage unit 30, etc., by anticipating changes in the field of view movement trajectory in advance, so that the observation field of view moves smoothly along the field of view movement trajectory in accordance with the movement direction indicated by the movement direction indication unit 55a, such as the joystick 55b. Specifically, as shown in FIG. 11A, the movement control unit 83 calculates a predicted arrival point from the movement direction DS1 and movement speed of the stage unit 30, etc., as well as the movement direction DJ1 indicated by the joystick 55b, etc., i.e., the polling interval for monitoring movement commands. Then, as shown in FIG. 11B, it searches for the point closest to the calculated predicted point on the field of view movement trajectory, corrects the movement direction DS1 and movement speed of the stage unit 30, etc., and moves it toward the closest point. The stage is moved based on the calculated DS1, and the above process is repeated when the polling period has elapsed. This achieves smooth movement even when the field of view movement trajectory is curved. Additionally, if forward and reverse directions are set in advance for the visual field movement trajectory, and the joystick tilt angle β is associated with these directions, the movement speed may change according to the angle of β. In this case, for example, the movement mechanism may be controlled so that the forward movement speed increases as the angle of β approaches 90°, and the reverse movement speed increases as the angle approaches 270°.

Here, a method for observing an enlarged image that realizes the route guide function will be described based on the flowchart in Figure 12. First, in step S1201, the user is prompted to specify trajectory information related to the direction of movement of the observation field. For example, the trajectory specification unit 81 accepts the user's specification of a reference point as trajectory information.

Next, in step S1202, the trajectory calculation unit 82 calculates a field of view movement trajectory in accordance with the trajectory information. For example, an interpolated route is set based on a reference point specified by the user using the trajectory specification unit 81. Then, in step S1203, a user instruction for the movement direction of the stage unit 30 is accepted. Furthermore, in step S1204, the movement direction of the stage unit 30 is determined based on the specified movement direction and the interpolated route. Finally, in step S1205, the stage unit 30 is controlled based on the determined movement direction.
(Trajectory instruction section 81, trajectory calculation section 82)

The trajectory instruction unit 81 specifies trajectory information related to the direction of movement of the observation field. Based on the trajectory information specified by the trajectory instruction unit 81, the trajectory calculation unit 82 sets a trajectory, i.e., a route, along which the observation field will move. Here, trajectory information can include, for example, multiple reference points. The trajectory calculation unit 82 calculates a route that passes through the multiple reference points specified by the trajectory instruction unit 81. For example, if the route is circular, three points on the circumference are specified as reference points. Furthermore, if the route is a combination of straight line segments, the start and end points of the line segments, or bend points, are specified as reference points. Note that the trajectory instruction unit 81 and the trajectory calculation unit 82 may be separate components or may be integrated.

The trajectory indication unit 81 and trajectory calculation unit 82 can also approximate a route using a pre-prepared geometric shape. That is, by selecting a geometric shape in advance using the trajectory indication unit 81 and then specifying a point that passes through this geometric shape as a reference point, the trajectory calculation unit 82 can calculate a route. Geometric shapes include circles, ellipses, rectangles, polygons, stars, straight lines, line segments, and curves such as arcs. There are no particular restrictions on geometric shapes as long as they can be expressed on a two-dimensional plane. Circles, ellipses, Bezier curves, etc. can be specified by specifying three or more points. Polygons such as rectangles can be created by specifying vertices.

The geometric shape can be selected in the geometric shape selection section, and the reference point can be specified in the reference point specification section. For example, if a straight line or line segment is selected as the geometric shape GS1, as shown in Figure 13A, by specifying two points on the display unit 70 as reference points, the straight line or line segment passing through these two points can be calculated as the route. Note that for ease of explanation, this example shows the state in which the head unit 4 has been moved relative to the stage unit 30.

As shown in Figure 13B, if a circle is selected as the geometric shape GS2, three reference points can be specified to calculate the circle that passes through these points as the route. Similarly, as shown in Figure 13C, if a rectangle is selected as the geometric shape GS3, three vertices or four corners can be specified as reference points to calculate the route of the rectangle defined based on these points. Note that if the four corners are 90°, the rectangle can also be defined by three points. Furthermore, geometric shapes are not limited to simple ones such as circles or rectangles; complex shapes can also be specified. For example, any shape can be used, such as the union or intersection of multiple shapes, a series of line segments including bent portions, or a shape tracing a path indicated with a pointing device. For example, Figure 13D shows an example of a partially missing circle as the geometric shape GS4, Figure 13E shows a star as the geometric shape GS5, and Figure 13F shows an example of a shape composed of multiple continuous line segments as the geometric shape GS6.

Furthermore, the geometric shape does not need to be a closed figure such as a circle or a rectangle, and can also be a straight line, a curve, etc. For example, consider an example of observing each chip CP of an observation object in which multiple chips CP are discretely mounted on a substrate CB, as shown in Figure 14. In this case, by setting a reference point at the position of each chip CP according to the arrangement of the chips CP, a broken line route RT as shown in Figure 15 is set as the field of view movement trajectory.
(Specific example of the trajectory instruction unit 81)

Specific examples of specifying trajectory information related to the direction of movement of the observation field are described with reference to Figures 16 to 18. These figures show an example in which the trajectory instruction section 81 is implemented on the trajectory instruction screen 110, which is a user interface screen for a magnified image observation program that operates the magnified observation device and is installed and executed on the main unit 50 of Figure 1. Such a user interface screen is displayed on the display unit 70 of the magnified observation device 100 or on the monitor of an externally connected computer. The user performs various settings and operations for the magnified observation device 100 from the screen displayed on the display unit 70. This magnified image observation program is built into the main unit 50. It goes without saying that the layout, shape, display method, size, color scheme, and pattern of each input field and button in these program user interface screen examples can be modified as needed. Changing the design can also improve visibility, facilitate evaluation and judgment, and create a layout that is easier to operate. For example, modifications can be made as needed, such as displaying a detailed settings screen in a separate window or displaying multiple screens on the same screen. Furthermore, on the user interface screens of these programs, the ON/OFF operation of virtually provided buttons and input fields, the input of numerical values and commands, etc., is performed by the operation unit 55. Here, imaging conditions, etc., are set using an input device connected to a computer in which the programs are installed. In this specification, "press" includes not only physically touching a button to operate it, but also clicking or selecting it using the input unit to virtually press it. The input/output devices constituting the operation unit 55, etc., are connected to the computer via wired or wireless connections, or are fixed to the computer, etc. Typical input units include various pointing devices such as a mouse, keyboard, slide pad, trackpoint, tablet, joystick, console, jog dial, digitizer, light pen, numeric keypad, touchpad, and AccuPoint. Furthermore, these input/output devices are not limited to operating programs but can also be used to operate hardware such as the magnifying observation device 100. Furthermore, the display unit 70 that displays the interface screen may itself use a touch screen or touch panel, allowing the user to input or operate by directly touching the screen with their hand, or voice input or other existing input means may be used, or these may be used in combination.

The trajectory instruction screen 110 of the magnified image observation program shown in Figures 16 to 18 has an image display area 111 on the left and an operation area 112 on the right. Image data of the observation field is displayed in the image display area 111. The operation area 112 displays buttons for making settings necessary to execute the route guide function. In this example, the items that the user must set are arranged from top to bottom in the operation area 112, and the user can use the route guide function by making settings in the order shown in the operation area 112. In this way, the operation area 112 serves a guidance function that guides even those who are not familiar with operating a magnified observation device to easily perform the necessary settings. Specifically, the operation area 112 has a route guide setting section 113 on the top row, a route display area 120 on the middle row, and a route guide execution display field 122 and a route guide start button 124 on the bottom row.

The route guide setting unit 113 has components for setting the geometric shape of the route. Specifically, the route guide setting unit 113 has a geometric shape selection button 114, a "Register" button 115, an "Undo" button 116, a "Reset" button 117, and the like. The geometric shape selection button 114 is a component for selecting the geometric shape of the route. In this example, three geometric shapes are available: circle, ellipse, and polygon, and the user can select one of them. The number of reference points that must be specified varies depending on the geometric shape selected with the geometric shape selection button 114. For example, as mentioned above, if the geometric shape is a circle, the circle will pass through the three specified reference points. If the geometric shape is a polygon, the line will be a straight line connecting the specified reference points.

The "Register" button 115 is used to register a specified point as a reference point. The "Undo" button 116 is used to delete the most recently specified reference point. The "Reset" button 117 is used to delete all specified points.

This example shows the state in which a circle has been selected with the geometric shape selection button 114. When a circle is selected, the route guide setting unit 113 shows a schematic diagram indicating that three points must be specified to define the circle. The user moves the observation field displayed in the image display area 111 to sequentially specify three points. Specifically, the user moves the field of view movement mechanism 5 to adjust the desired position of the observation object to be at the center of the observation field. Once positioning is complete, the user presses the "Register" button 115. The reference coordinate acquisition unit 85 shown in FIG. 2 acquires the coordinate position of the center of the observation image displayed in the image display area 111 as the reference position and registers it as the coordinate of the reference point. At the same time, the observation field including the registered reference position is displayed in the route display area 120 in a reduced size. Furthermore, as will be described later, if the field of view movement trajectory also includes z-coordinate information, the reference coordinate acquisition unit 85 may acquire not only x-y coordinates but also z-coordinates when registering the reference point.

The route display area 120 is a component for displaying the specified reference points and the set visual field movement trajectory. This route display area 120 can change the display scale each time a reference point is specified so that all of the specified reference points fit within the image.

The route display area 120 can also display a wide-area image showing the entire object being observed. The wide-area image is image data captured with the objective lens unit 25 at a low magnification to widen the observation field. In the wide-area image, a rectangular portion of the observation field registered as a reference point is displayed. As the user sequentially specifies reference positions in the image display area 111, a rectangular image of the observation field including the registered reference position is displayed in the corresponding portion of the wide-area image displayed in the route display area 120. Figure 17 shows the state after a second reference position has been registered following Figure 16. Figure 18 shows the state after a third reference position has been registered. In this state, a circle passing through the three reference positions is established, and the field movement trajectory is determined. Furthermore, the display magnification in the route display area 120 is automatically adjusted to display all registered reference positions. In the example of Figure 18, the display magnification of Figure 17 is insufficient to display all three reference positions, so the display magnification is automatically adjusted to display a circle including all three reference positions by lowering the display magnification. In this way, being able to see the entire view movement trajectory and the specified reference position in one image makes it easier for users to visually grasp the correspondence between the reference position they specified and the view movement trajectory.

The route guide execution display field 122 displays a message to notify the user that route tracing mode has been selected. For example, while route guidance is being set or executed, "Route guidance in progress" is displayed. When route tracing mode is canceled, messages such as "Route guidance canceled" or "Guidance will begin when you move onto the route" are displayed, and the user can be shown the steps to return to the route guidance function.

The route guide start button 124 is a component for executing the route guide function. The route guide start button 124 cannot be pressed until the settings required to execute the route guide function have been completed. Specifically, since the number of reference points that need to be specified varies depending on the selected geometric shape, the route guide start button 124 is grayed out until the required number of reference points have been specified. The user is prompted to perform settings in order from the top of the operation column, and once all the setting work has been completed, the route guide start button 124 becomes selectable. When the route guide start button 124 is pressed in this state, route guidance begins and the movement of the stage unit 30 is restricted.
(Root completion)

The trajectory calculation unit 82 calculates the field of view movement trajectory based on the trajectory information input from the trajectory specification unit 81. The trajectory calculation unit 82 interpolates the coordinates between multiple reference points and completes the field of view movement trajectory as an interpolated route. Furthermore, the trajectory calculation unit 82 changes the interpolation algorithm based on the selected geometric shape. Note that trajectory information is not limited to multiple reference points; other specification methods can also be used as appropriate. For example, trajectory information specifying a circular trajectory can be specified by the center coordinates and radius of the circle.

Up to this point, we have mainly discussed methods for specifying circular trajectories. Next, we will explain the case of specifying a linear field of view movement trajectory with bends, based on Figures 14 and 15. First, if a straight line is selected as the geometric shape GS2 in Figure 13B, by specifying two or more points as reference points, a straight line passing through these points can be calculated as the field of view movement trajectory. In Figure 14, I to IV indicate the observation field and the order in which they are specified as reference points. If a straight line is specified as the geometric shape GS2, when a reference point is specified, the order in which the reference points are specified may be stored in memory unit 56 along with the coordinates of the reference points. In this case, the field of view movement trajectory is calculated so that the intervals between the reference points are interpolated in the order in which they are specified. That is, if the reference points are specified in the order I to IV in Figure 14, the field of view movement trajectory shown in Figure 15 is calculated. Furthermore, if the forward and reverse directions described above are stored in association with the field of view movement trajectory, the direction from the earliest to the latest reference point may be set as the forward direction, and the reverse direction may be set as the reverse direction. That is, in Figure 15, the direction from I to IV is set as the forward direction, and the direction from IV to I is set as the reverse direction.

Furthermore, the trajectory information for specifying the direction of movement of the observation field of view is not limited to the method of specifying a reference point on the screen of the display unit 70 as described above; other methods can also be used, such as directly entering numerical coordinate positions or specifying the field of view movement trajectory using a formula.

Note that when placing an object to be observed in a fixed position on the stage unit 30, for example, when using a positioning jig to place the same sample in the same position on the stage unit 30 each time, the field of view movement trajectory can be set once to match the jig's design values, and route guidance can then be performed under the same conditions. In other words, there is no need to move the stage unit 30 each time, register a reference position, and set the field of view movement trajectory. Furthermore, by storing the field of view movement trajectory settings in the memory unit 56, the stored settings can be recalled to reproduce the route. In this case, for example, the coordinates of each reference point acquired by the reference coordinate acquisition unit 85 and the geometric shape selected and accepted by the trajectory designation unit 81 are stored as setting information for the field of view movement trajectory. When recalling the stored settings to reproduce the route, the route is reproduced based on the coordinates and geometric shape of each reference point. Furthermore, if the lens and magnification used when the reference point was specified are also stored, the field of view movement trajectory is reproduced based on this information and information such as the current lens and magnification, and also performs field of view misalignment correction, as described below.

In addition to being specified by a geometric shape, the field of view movement trajectory can also be extracted automatically from an image. For example, a wide-area image of the observation object WK3 as shown in Figure 19A is captured, and a contour PL is extracted by edge extraction as shown in Figure 19B. If it is set to move along all or part of this contour PL, the trajectory instruction unit 81 or trajectory calculation unit 82 can automatically obtain trajectory information and acquire the field of view movement trajectory, saving the user the effort of manually specifying trajectory information each time. Edge extraction can, for example, connect extraction points to obtain a contour, or the line connecting the extraction points can be approximated with a straight line or curve.

In addition to displaying the entire image of the object being observed on one screen as shown in Figure 19A and extracting the shape of the object being observed to set the field of view movement trajectory, it is also possible to configure the system to display a portion of the object being observed while moving the observation field and sequentially extracting the contours to set the field of view movement trajectory. For example, as shown in Figures 20A and 20B, the observation field of view can be updated while moving the XY stage, and the contour can be obtained in real time from the image of the object being observed WK4, automatically detecting the field of view movement trajectory. Such automatic acquisition of the field of view movement trajectory can be performed by the trajectory instruction unit 81 or the trajectory calculation unit 82.

After setting the field of view movement trajectory as described above, when route trace mode is executed, the user can move the observation field of view along the field of view movement trajectory even with a rough instruction, without having to specify the movement direction of the field of view movement mechanism 5 in detail using the movement direction instruction unit 55a such as the joystick 55b. That is, in route trace mode, the movement direction specified by the movement direction instruction unit 55a is compared with the direction of the field of view movement trajectory, and if it is within a specified range of the field of view movement trajectory, it is determined that an instruction to move along the field of view movement direction has been made, and the field of view movement mechanism 5 is moved. As a result, even while the observation field of view is being moved, the center of the observation field of view is always moved to be on the field of view movement trajectory, allowing the user to always observe the object of observation in a position that is easy to see, even while moving the observation field of view.

In this way, simply tilting the joystick 55b in a general direction allows the observation field of view to move along the specified route for easy viewing. The tilt direction of the joystick 55b, i.e., the tilt angle β, is specified, for example, in the plan view shown in Figure 5, within the range of 0° to less than 360°. When using this joystick 55b to move along the route RT of the field of view movement trajectory in the straight section from P3 to P4 as shown in Figure 15 above, movement in the field of view movement direction continues as long as the joystick 55b is tilted within a range of ±90° relative to the field of view movement direction, i.e., in the area in the direction of travel defined by a line perpendicular to the field of view movement direction. On the other hand, if it is not within this range, i.e., if the joystick 55b is tilted in a direction roughly opposite to the direction of travel or is not tilted, movement of the observation field of view stops. Furthermore, if the field of view movement trajectory is a curve such as a circular arc, the tangent at the current position on the field of view movement trajectory is compared with the movement direction, and if it is within the specified angle range, route guidance continues.

On the other hand, when the movement direction instruction by the movement direction instruction unit 55a is made at a position where the field of view movement trajectory is bent, such as position P2 in Figure 15, the bisector that bisects the angle of the field of view movement trajectory formed by the bent position determines whether the observation field of view can be moved. That is, in the example of Figure 15, if a movement direction is instructed at an angle above the bisector indicated by the dashed line, the observation field of view is moved in the direction from 2 to 3. On the other hand, if a movement direction is instructed at an angle below the bisector, the movement of the observation field of view is stopped.

The movement speed of the observation field of view may also be changed by the tilt direction of the joystick 55b. For example, the movement speed of the observation field of view, for example, the movement speed of the stage unit 30, may be increased as the tilt direction of the joystick 55b is closer to the field of view movement trajectory, i.e., the smaller the angular difference between the two, and the slower the movement speed may be, and the movement speed may be increased as the angular difference is greater. Also, as described above, when the movement speed of the observation field of view, for example, the movement speed of the stage unit 30, is changed according to the vertical tilt angle α of the joystick 55b, the movement speed of the stage may be changed by combining the tilt direction of the joystick 55b in a planar view (tilt angle β) and the tilt angle α in the vertical direction.

The memory unit 56 may also store the lens type, magnification, and movement speed settings in association with each other. In this case, the movement speed of the observation field of view can be changed depending on the lens type and magnification used during observation. The higher the magnification, the greater the movement amount of the observation field relative to the movement amount of the field of view movement mechanism 5. Therefore, it is desirable to change the movement speed setting depending on the lens type and magnification. In this case, a consistent operational feel can be obtained regardless of the selected lens type and magnification. The movement speed setting may also be changed depending on the aspect ratio of the observation field of view. For example, if the observation field of view is vertically long, horizontal movement will be perceived as relatively fast. Therefore, the vertical and horizontal movement speeds of the observation field of view can be set to be constant depending on the aspect ratio.
(Field of view offset correction function)

It is also possible to change the display magnification in the image display area 111 during observation. To ensure that the device continues to operate according to the set route even after the magnification is changed, the magnification observation device 100 is equipped with a field of view shift correction offset function that automatically corrects field of view shift when the magnification is changed. This is described in detail below.

When switching the objective lens unit 25 or other components to change the magnification, the center position of the observation field may shift. For example, in the observation field shown in Figure 21A, if the multiple objective lens units 25 provided on a rotating revolver are mechanically switched by rotating the revolver to increase the display magnification, the field of view center CS, indicated by the intersection of the crisscrossing grid lines, will be displayed shifted, as shown in Figure 21B. For this reason, a known technique is to calculate and store the amount of shift in the field of view center CS in advance, and then move the XYZ stage by the amount of shift when switching magnification to correct the field of view shift. For example, the field of view movement mechanism 5 is automatically moved from the state shown in Figure 21B to the position corresponding to the field of view center CS previously shown in Figure 21A, as shown in Figure 21C.

However, if the route set in route tracing mode is managed using stage coordinates, moving the XY stage due to such a correction will result in the field of view after switching being on the route, but being treated as being off the route in stage coordinates. Furthermore, the route set in route tracing mode contains information not only on the XY plane but also in the Z direction. Conventional field of view deviation correction functions only perform correction in the XY plane, and therefore cannot correct deviations in the height direction. Therefore, in the magnification observation device 100 according to this embodiment, when deviation correction is performed, the route itself is offset by the deviation correction amount, as shown in Figure 22, to ensure that the route after deviation correction is consistent. This field of view deviation correction offset function can be performed not only on the XY plane, but also in the Z direction, which is the height direction. Such a field of view deviation correction offset function can be performed by the trajectory calculation unit 82.

Note that misalignment of the field of view center when switching magnification is not limited to when switching the objective lens unit 25 using a revolver or when physically replacing the objective lens unit 25, but can also occur when zooming in or out using a zoom optical system. The magnification observation device 100 according to this embodiment does not limit the use of the field of view deviation correction offset function to when switching the objective lens unit 25, but can be used in any situation where field of view deviation occurs.

The route offset function can also be used for other purposes. For example, if it is difficult to always place a sample in the same position on a previously set route, the entire route can be offset by specifying that any point on the route corresponds to the current field of view. For example, by specifying a position on the field of view movement trajectory that corresponds to the observation field of view, such as RA in Figure 23, the entire field of view movement trajectory will be offset according to that specification, as shown at RB.

As described above, the magnifying observation device 100 has a route tracing mode in which the route guide function is executed, and a free mode in which the route tracing mode is canceled. In the route tracing mode, as described above, the movement control unit 83 controls the movement direction instructing unit 55a to operate the field of view movement mechanism 5 so as to follow the field of view movement trajectory calculated by the trajectory calculating unit 82. On the other hand, in the free mode, the movement control unit 83 controls the field of view movement mechanism 5 to operate in the movement direction instructed by the movement direction instructing unit 55a, regardless of the field of view movement trajectory. With this configuration, the route tracing mode easily realizes movement of the observation field of view along the field of view movement trajectory, while also accommodating the user's free movement of the observation field of view.
(Route guide release unit 55d)

The magnification observation device 100 may also be provided with a route guide canceling unit 55d. The route guide canceling unit 55d is a component for canceling the route trace mode while it is running. The route guide canceling unit 55d may detect a condition for canceling the route trace mode and automatically transition from the route guide mode to the free mode, or may be configured to accept an explicit instruction from the user to cancel the route trace mode. The route guide canceling unit 55d may automatically cancel the route guide when, for example, the direction or position of movement of the field of view movement mechanism 5 differs from the field of view movement trajectory calculated by the trajectory calculation unit 82 by a predetermined value or more. If the destination is specified by coordinates, the predetermined value may differ for each magnification. Here, a case where the movement position is used as the cancellation condition is explained with reference to Figures 24 to 26. The example in Figure 24 illustrates a case where a circle is specified as the movement position PT using a pointing device such as a mouse or touch panel as the movement position to which the observation field of view is moved relative to the route RT of the field of view movement trajectory set in an arc shape. When a new movement position PT is specified, the route guide cancellation unit 55d calculates the distance between the specified movement position and the field of view movement trajectory. If this distance is equal to or greater than a predetermined value, the cancellation condition is met and the route trace mode is canceled, as shown in FIG. 25. That is, the field of view movement mechanism 5 controls the stage unit 30 to move away from the field of view movement trajectory and to the specified movement position. Alternatively, the route trace mode may be canceled when an area other than the area displayed as the observation field of view is specified. Note that, as shown in FIG. 26, it is also possible to return to route trace mode by again specifying a position closer to the field of view movement trajectory (details will be described later).

The movement direction may also be used as another cancellation condition. In this case, the route guide cancellation unit 55d determines whether the movement direction of the observation field of view indicated by the movement direction indication unit 55a is within a predetermined angle range with respect to the field of view movement trajectory. If it is within the angle range, the route tracing mode continues. If it is outside the angle range, the route tracing mode is canceled and the mode transitions to free mode. Here, the movement direction of the observation field of view is the inclination angle β indicated by the movement direction indication unit 55a, such as the joystick 55b. Furthermore, if the field of view movement trajectory is a straight line, the angle between the movement direction of the observation field of view and the field of view movement trajectory is the angular difference between this straight line and the movement direction of the observation field of view. Furthermore, if the field of view movement trajectory is a curved line, the angle is the angular difference between the tangent direction of the curve and the movement direction of the observation field of view.
(Cancellation condition setting unit)

The user may also be able to set the conditions for canceling this route tracing mode. For example, a cancellation condition setting screen 130 such as that shown in FIG. 27 may be provided as a cancellation condition setting section for setting the conditions. In this example, the user can select either the distance between the movement position and the visual field movement trajectory or the angle difference between the movement direction and the visual field movement trajectory as the cancellation condition, and can also set the range of the selected distance or angle.

Furthermore, without providing a physical route guide canceller, the movement control unit 83 may automatically switch route tracing mode ON/OFF. For example, if the movement direction instruction given by the movement direction instruction unit 55a for the observation field of view of the image data is within a predetermined angle range with respect to the direction in which the observation field of view should be moved along the field of view movement trajectory calculated by the trajectory calculation unit 82, the route tracing mode will continue, but if it is outside the predetermined angle range, the system will switch from route tracing mode to free mode.

In route tracing mode, when a direction of movement is specified, the mode is controlled to stay on the route. On the other hand, when a destination is specified by coordinates, such as by dragging the mouse, specifying coordinates, or moving the origin, the mode is controlled to turn off. In this case, the mode does not turn off immediately, but rather when the user moves a certain distance away from the route. This certain distance can be set, for example, to 30% of the observation field of view. Even if the mode is turned off after moving away from the route, it will automatically turn back on if the user approaches the route and moves again. Therefore, for example, if you discover something interesting while moving the observation field of view along a route, you can drag the mouse in that direction, confirm it, and then push the joystick back in the direction of the route to naturally resume movement along the route.
(Route guide ON/OFF display function)

Because movement of the observation field of view in the X and Y directions is restricted while the route guide function is active, it is preferable to notify the user that route tracing mode is active. Similarly, it is desirable to notify the user when route tracing mode is canceled. Therefore, the magnification observation device 100 according to this embodiment is equipped with a route guide ON/OFF display function that indicates whether route guidance is ON or OFF. Specifically, the display mode of the field of view movement trajectory on the display unit 70 is changed depending on whether the route tracing mode is ON or OFF. For example, the field of view movement trajectory is overlaid directly on the image being observed, displayed in the image display area 111. This allows the user to predict the direction of travel while operating the device. The display mode may also be changed not only in the image display area 111 but also in the route display area 120. Furthermore, as shown in Figures 28A and 28B, a navigation display screen 140 that displays low-magnification images may be separately provided on the display unit 70, and the display mode of the route corresponding to the field of view movement trajectory may be changed on this navigation display screen 140. Figure 28A shows the state in which the field of view movement trajectory is displayed superimposed on a low-magnification image displayed across the entire navigation display screen 140, while Figure 28B shows the state in which a field of view wider than the image obtained is displayed on the navigation display screen 140, with the field of view movement trajectory displayed even in areas where there is no image data.

By providing such a route guide ON/OFF display function, which indicates whether the route guide is ON or OFF, the user can visually recognize that the route trace mode has been canceled. For example, when the route trace mode in FIG. 24 is canceled and the free mode is entered, as shown in FIG. 25, the route RT of the visual field movement trajectory, which was displayed as a solid line in the image display area 111, changes to a route RT' displayed as a dashed line. Additionally, as shown in FIG. 29A, the circular visual field movement trajectory displayed in the route display area 120 also changes from a solid line route RT to a dashed line route RT' as shown in FIG. 29B. Additionally, a message such as "Guidance will begin when you move onto the route" may be displayed. This allows the user to be guided through the procedure for resuming the route trace mode. Furthermore, the display of the visual field movement trajectory is not limited to switching between solid and dashed lines as in this example; any other method can be used, such as changing the display color, changing from a thick line to a thin line, or graying out the trajectory. Such changes in the display mode of the visual field movement trajectory may be performed by the route guide cancellation unit 55d or by the trajectory calculation unit 82.

Furthermore, in free mode, if the observation field of view approaches the field of view movement trajectory, the system can be returned to route tracing mode. That is, even after switching to free mode, the route guide cancellation unit 55d continues to determine whether the cancellation conditions are met, and if it detects that the cancellation conditions are no longer met, it returns to route tracing mode. For example, if it detects that the movement position approaches the field of view movement trajectory from the state shown in FIG. 25, as shown in FIG. 26, the system executes route tracing mode again. In response to this, the display mode of the field of view movement trajectory also changes from free mode to route tracing mode, for example, from a dashed line to a solid line. For example, while sequentially observing target portions of an observation object in route tracing mode, if the user temporarily discovers a site of interest and moves the observation field off the route, even if the user wishes to return to the original observation field after finishing the temporary observation and continue observation, by moving the observation field closer to the field of view movement trajectory, the user can easily move the observation field along the field of view movement trajectory again using the movement direction indicator 55a.

In the above example, we have described a configuration in which the route guide canceling unit 55d automatically switches between canceling and returning to route tracing mode. However, the present invention is not limited to this configuration, and the route tracing mode may also be explicitly switched on and off. For example, a component such as a mode selector switch could be provided as the route guide canceling unit, allowing the user to explicitly switch route guidance on and off.

In the above example, the instruction to cancel route tracing mode can be accepted whether the observation field is moving or is paused.

In the above example, the route guide canceling unit 55d is configured to determine whether the canceling conditions for canceling the route guide mode are met, but the canceling conditions may be determined by another component, such as the trajectory calculation unit 82 or the movement control unit 83. In addition, when the functions of the route guide canceling unit 55d and the trajectory calculation unit 82 are realized by a processor unit, it is also possible to have a common component execute the determination of the canceling conditions and the calculation of the field of view movement trajectory.
[Embodiment 2]

In the above example, a pointing device such as a mouse is used as the route guide release unit 55d. In other words, the route guide release unit 55d is provided separately from the movement direction instruction unit 55a such as the joystick 55b. In this way, the movement direction instruction unit 55a and the route guide release unit 55d may be configured as separate components, or the functions of the movement direction instruction unit 55a and the route guide release unit 55d may be realized by a common component. For example, in the magnification observation device 200 according to the second embodiment shown in FIG. 30 , the movement direction instruction unit 55a and the route guide release unit 55d are realized by a single operation unit 55. For example, a mouse may be used as the operation unit 55, and the function of the movement direction instruction unit 55a may be realized by a drag operation of the mouse, and the function of the route guide release unit 55d may be realized by input from the mouse. Similarly, the functions of the movement direction instruction unit 55a and the route guide release unit 55d may be realized by a keyboard, or the functions of the movement direction instruction unit 55a and the route guide release unit 55d may be realized by an input device combining a mouse, a keyboard, or the like.
(search writing)

Furthermore, during route tracing mode, a searchlighting function can be implemented that automatically changes the lighting direction while the observation field of view is moving along a set route. For example, as shown in Figures 31A to 31D, the one-sided lighting of the illumination unit 60 can be switched between four directions in sequence. This allows the user to notice flaws and other defects that are only visible when illuminated in a specific direction while moving the stage unit 30. In other words, in the past, it was difficult to accurately move along the entire circumference, and the user had to try out lighting patterns in all directions, which required time and effort. This forced users to resort to simple measures such as lowering the magnification and checking only a portion, which resulted in the risk of overlooking something. In contrast, the magnification observation device 100 of this embodiment combines the route guide function with the searchlighting function, thereby achieving observations with significantly reduced user burden.

The examples in Figures 31A to 31D illustrate an example in which the ring illumination unit 63 or coaxial illumination of the illumination unit 60 shown in Figure 3B is divided into four blocks in the vertical and horizontal directions, and the lighting pattern of each divided illumination block is sequentially switched clockwise. By rotating the illumination direction in a fixed direction in this way, the illumination changes relatively smoothly and the visibility of scratches can be changed so that they gradually become easier to see or gradually become less visible, making it easier for the user to search for scratches, etc. Of course, the illumination direction can also be counterclockwise, and the division of the illumination blocks is not limited to four, but can also be two, three, five or more blocks.

The searchlighting function also continuously switches the lighting direction. When a user finds what appears to be a scratch, they can specify the location of the scratch on the display screen of the display unit 70, and the lighting direction automatically switches to the lighting direction that best visualizes the specified location. For example, if the user clicks on a location of interest in the image display area 111 with a pointing device such as a mouse while the observation field is moving and the lighting direction is rotating, the lighting rotation stops and the lighting direction automatically switches to the lighting direction that best visualizes the scratch. The appropriate lighting direction is selected, for example, by the image processing unit. For example, while the searchlighting function is running, image data for which the lighting direction is being switched—in this case, four pieces of image data corresponding to the lighting directions of up, down, left, and right—are stored in the buffer memory 57. When a mouse click is detected, the lighting direction switching stops and the image data for each of the most recent lighting directions stored in the buffer memory 57 is analyzed. For example, image data with a distribution that makes the scratch more visible, based on a brightness histogram or the like, is selected, and the lighting direction used in that image data is selected. The display on the display unit 70 displays a live image fixed in that lighting direction. For example, in the examples of Figures 31A to 31D, the lighting direction corresponding to the image in Figure 31D is selected. This allows for a display mode that makes scratches easier to see.

When the display magnification is high, the entire image screen may not be in focus. Therefore, multiple images can be captured while moving the Z stage, and the in-focus pixels can be combined to generate an all-in-focus image. In particular, this combination process can be performed when the XY stage stops. By automatically combining all-in-focus images when the stage unit 30 stops, the user can move along a route and simply stop the stage unit 30 when an object of interest comes into view, generating an all-in-focus image of that location for review. Furthermore, combined images can be created by switching shooting conditions such as lighting brightness and exposure time. For example, images with different lighting directions, known as Differential Phase Contrast, can be combined to more clearly highlight fine irregularities. Furthermore, technology such as switching lighting directions to combine pixels without halation, or HDR, can also be used in combination.

When inspecting parts for burrs or chips, it is extremely important to ensure nothing is overlooked. Therefore, while observing the entire circumference of the part as a route, it is possible to continue recording video as a record of the inspection.

Focusing can also be continued during XY movement by obtaining the 3D shape of the sample in advance from a low-magnification image as shown in Figure 32A, or by estimating the tilt of the sample plane within the observation field as shown in Figure 32B. Note that when obtaining the 3D shape from a low-magnification image, the focus may not always be achieved with high precision.

Figure 33 shows an example of a profile in which a 3D shape is obtained from a low-magnification image. In this figure, the true shape of the object being observed is shown by a dashed line, and the 3D shape obtained from the low-magnification image is shown by a solid line. As shown on the left side of the figure, the edges of the field of view are detected as raised due to the influence of the mirror curvature of the lens. Furthermore, in areas that are bent, such as rising, the 3D shape is detected as dull.

Therefore, autofocus can also be used when the XY movement of the stage unit 30 is stopped. By offsetting the entire 3D shape using the Z coordinate after autofocus is performed, it is possible to reduce focus shifts when movement starts. This also makes it possible to deal with cases where the 3D shape is distorted due to the effects of lens field curvature, etc.
(Autofocus processing)

In image autofocus processing, a still image is captured at the in-focus position while moving the Z position. In this case, the image capture process is required for the range searched in the vertical direction. Typically, several dozen images are captured and combined at even intervals. Autofocus processing can be performed manually by the user using a dedicated execution button, or it can be performed automatically. For example, autofocus can be performed automatically when the movement of the observation field of view has stopped. The movement of the observation field of view can be determined to have stopped when the input of an operation signal from the field of view movement mechanism 5 is no longer received, or when a change in the image is detected and the amount of change falls below a predetermined value. For example, if the image does not change for a certain period of time (e.g., a few seconds) in live video, it can be determined that the movement of the observation field of view has stopped, autofocus can be performed, and a focused still image (autofocus image) can be displayed. This ensures that a focused still image is always displayed even when the user has stopped moving the field of view, making it easier to check details.

In addition, in a method for estimating the tilt of the sample plane within the observation field, the Z stage is operated each time the XY movement stops, and the tilt in the field of view where it has stopped is continuously determined. This is shown in Figure 34. As shown on the left side of the figure, the stage unit 30 is moved in the Z direction at the starting position to estimate the plane. Then, as shown in the center of the figure, the plane is estimated when the XY movement stops. Accuracy is improved by using these two points in conjunction with the height information measured earlier. Furthermore, as shown on the right side of the figure, the process of estimating the plane is repeated when the stage unit 30 is stopped again, further improving accuracy.

In this way, for samples of flat but tilted objects, the accuracy of the estimated plane improves with repeated movement and stopping. It goes without saying that the above functions can also be used in combination, such as by sequentially switching the lighting during XY movement along a route and recording the process.

Next, we will explain in detail how to set the route of the visual field movement trajectory. It is possible to set multiple visual field movement trajectories on the stage unit 30. In this case, when moving between multiple routes, the route trace mode is temporarily turned OFF.

When setting a route, the reference coordinate acquisition unit 85 may be configured to register not only XY coordinates but also Z coordinates, i.e., information regarding the relative distance between the focal position of the objective lens unit 25 and the object being observed. This makes it possible to move along the desired observation position even in cases where the object being observed has an inclined surface. For example, when setting the circular field of view movement trajectory described above, the reference coordinate acquisition unit 85 can also acquire Z-direction information when specifying three reference points, thereby calculating the tilt of the plane from the information on the three reference points. In other words, a 3D field of view movement trajectory can be calculated. Therefore, by setting the field of view movement trajectory within the calculated plane, it is possible to realize the route guide function along the inclined plane in a focused state. In other words, because the field of view movement trajectory retains information regarding the relative distance between the focal position of the objective lens unit 25 and the surface of the object being observed, the movement mechanism can be controlled so that the observation field of view moves along the 3D field of view movement trajectory.

Routes in three-dimensional space can also be managed as a point cloud. They can also be managed as a figure projected onto the XY plane. For example, to set a circular view movement trajectory when viewed from directly above, three points specified in three-dimensional space as shown in Figure 35A are projected onto the XY plane as shown in Figure 35B, and a circle is defined on the XY plane based on the three projected points as shown in Figure 35C. This makes it possible to handle the view movement trajectory as three-dimensional data mapped onto the XY plane, simplifying calculation processing.

The XY stage simply moves the stage unit 30 along the circle on the XY plane calculated as described above. The Z stage can move along a specified route in three-dimensional space as shown in Figure 36B by constantly calculating the Z coordinate corresponding to the XY coordinate of movement based on the plane calculated in three-dimensional space as shown in Figure 36A. By knowing the equation that defines the inclined plane calculated in three-dimensional space as shown in Figures 36A and 36B, the Z coordinate can be determined by projecting the XY plane onto this inclined plane.

In the case of polygons, the XY stage can be operated by projecting them onto the XY plane in a similar manner. In this case, by calculating the Z coordinate based on the slope between points as shown in Figures 37A and 37B, the user can move along the intended route even while moving between points.

Furthermore, even when routes intersect on the XY plane, the direction to proceed can be determined based on the direction of the joystick 55b, etc. For example, in the example shown in Figure 38, if the joystick 55b is tilted within a range of ±90° relative to the direction of travel, the robot will proceed straight; if it is tilted between +90° and +180°, the robot will turn left (upward in the figure); and if it is tilted between -90° and -180°, the robot will turn right (downward in the figure).

It is also possible to automatically determine which direction to move based on the shape unit used when the coordinate was registered, such as a line segment or circle. When the Z coordinate is also being tracked, it is preferable to determine the movement direction based on the shape unit. For example, if the coordinates intersect at a twisted position, as shown in Figure 39, an unintended Z-axis rise may occur, causing stress for the user. If the user operates the Z stage while the Z coordinate is being tracked, the route is offset to the position where the operation is completed. When the user moves the stage unit 30, it is highly likely that the object being observed is not in focus. For example, if the registration point is too close, as shown in Figure 40, the movement may cause the object to become out of focus. It is preferable that the process of offsetting the route to the Z coordinate operated by the user also be performed when the Z stage is moved while the XY stage is stopped. For example, as shown in the lower left of Figure 41, if the XY movement is temporarily stopped and manual movement is performed in the Z direction while the stage is automatically tracking along the route, when movement in the XYX directions is resumed, the stage will move along the offset route.

Furthermore, if managing Z stage operation while the XY stage is moving becomes cumbersome, you can exclude Z stage operation or stop Z tracking when the stage is operated. For example, as shown in the bottom left of Figure 42, if the stage is automatically tracking along the route in the X, Y, and Z directions, and then you manually move in the Z direction during XY movement, Z tracking will be stopped and the route guide function will only work for movement in the X and Y directions.

In addition, offsetting may be performed only when movement in the XY direction is stopped, without offsetting even if movement in the Z direction is performed while movement in the XY direction is in progress.

By offsetting the Z coordinate at the time the XY stage movement begins, even if Z tracking stops working and blur becomes noticeable, the user can simply stop the XY stage, adjust the focus, and then resume XY stage movement.

When the user explicitly attempts to move to a specific location, such as by clicking the mouse on the screen to move the XY stage, it is possible to temporarily allow movement off the route. In such cases, the process of turning off the movement mode, moving to the tip, turning the mode back on, and returning to the route is time-consuming. Therefore, dragging the screen allows movement off the route and temporarily turns the mode off. If the user then returns to the route using the joystick 55b or other devices, the mode automatically turns back on. When the mode is temporarily turned off, the user is notified by a preview or other display. In addition to dragging, other operations that can cause movement off the route include commands to move to the XY stage origin and move to specified coordinates. The determination of whether or not the user is on the route can also be managed with a certain degree of leeway. This prevents chattering between the mode on and off, even if a slight overrun occurs when returning to the route.
(Surface observation)

In the above example, we have described an example in which observation is performed along the contour of the object being observed. However, the present invention is applicable not only to linear observations like this, but also to planar observations. That is, it is possible to automatically set the interior of a predetermined area as a route. For example, this is effective in applications such as visually scanning the entire area of a circular object being observed, as shown in Figure 43A. First, an overall image is obtained for the observation field of view in Figure 43A. For example, as shown in Figure 43B, an image of the entire view is synthesized while moving the observation field of view. Alternatively, the observation field of view can be moved along the contour, or the magnification can be adjusted to a low level so that the entire view fits within the observation field of view. Once the contour of the object being observed has been obtained in this way, a field of view movement trajectory is set so that the entire inner surface of this contour can be observed. For example, as shown in Figure 43C, the trajectory calculation unit 82 automatically calculates a route for the outer periphery area based on the size of the current observation field of view.

As described above, the user can easily move the observation field along the field of view movement trajectory calculated by the trajectory calculation unit 82 without having to give detailed instructions for moving the observation field using the movement direction instruction unit 55a, thereby simplifying operation. In other words, the magnification observation device 100 sets a route in the trajectory calculation unit 82, interpolated based on multiple reference points using the trajectory instruction unit 81, so that when a movement direction instruction is received from the movement direction instruction unit 55a, the stage unit 30 can be controlled along the set route. Furthermore, when setting a route from a reference point, not only the XY plane but also the Z coordinate is used as the basis, ensuring consistently focused observation while the observation field is moving along the route. The set route is based on two-dimensional shapes such as circles, ellipses, rectangles, and polygons, and movement along the periphery of these shapes or moving sequentially within the shape is possible.

When an image of the observation field including the object of observation based on image data generated by the camera unit is displayed on the display unit, a composite image that has undergone image synthesis processing can be displayed. Examples of composite images that have undergone image synthesis processing include depth synthesis images, HDR images, unevenness-enhanced images, and halation-removed images. Here, we will explain composite images. The image processing unit 84 realizes a function for synthesizing multiple images (synthetic image mode). For example, for an image with a shallow depth of focus, it is possible to synthesize only the in-focus areas of multiple images taken while changing the focus position based on the focus information to obtain an image with a deep depth of focus (depth synthesis image). It is also possible to obtain images with enhanced resolution or an expanded dynamic range using so-called super-resolution technology. Examples of composite images generated by the image processing unit 84 in this way include depth synthesis images, 3D composite images, pixel-shifted images, super-resolution images, and HDR images. HDR images have a significantly higher dynamic range, i.e., a ratio between minimum and maximum light intensity, than conventional images. For example, standard computer monitors use 8-bit to 24-bit color as their standard color representation, allowing for 2.56 million to 16.77 million shades of gray. However, in reality, many more colors exist, and the human eye adjusts its pupil size to a standard brightness that it considers appropriate. Therefore, HDR images that contain more color information than the monitor's representation capabilities are used. Such HDR images can be obtained using known techniques, such as combining multiple images of the same object captured at the same position under different imaging conditions (typically, the exposure time of the image sensor). For example, a high-gradation HDR image can be created by combining multiple low-gradation images captured with different dynamic ranges in the luminance domain.

Furthermore, when the composite image is displayed on the display unit, the composite image may be displayed as a still image when the movement of the observation field by the field of view movement mechanism is stopped, and a live image may be displayed when the observation field of view is moved.

The magnification observation device, magnified image observation method, magnified image observation program, computer-readable recording medium, and device storing the program of the present invention can be suitably used in microscopes, reflective and transmissive digital microscopes, and other devices.

100, 200... Magnification observation device 1... Imaging system 2... Control system 3... Cable section; 3b... Optical cable 4... Head section 5... Field of view movement mechanism 10... Camera section 11... Imaging optical system 12... Imaging element; 13... Imaging element control circuit 16... Upper Z elevator 20... Microscope lens section 25... Objective lens section 30... Stage section 35... Lower stage elevator 36... Motor control circuit 37... Stepping motor 43... Camera mounting section 50... Main body section; 51... Processor section; 52... Display control section 53... Storage section; 54... Interface;
55... operation unit; 55a... movement direction instruction unit; 55b... joystick; 55c... movement button; 55d... route guide cancellation unit; 56... memory unit;
57...Buffer memory 60...Illumination unit 62...Coaxial incident illumination unit 63...Ring illumination unit 65...Illumination light source 66...Illumination control unit 70...Display unit 81...Trajectory instruction unit; 82...Trajectory calculation unit; 83...Movement control unit 84...Image processing unit 85...Reference coordinate acquisition unit 110...Trajectory instruction screen 111...Image display area 112...Operation area 113...Route guide setting unit 114...Geometric shape selection button 115..."Register" button 116..."Restore" button 117..."Reset" button 120...Route display area 122...Route guide execution Line display field 124... route guide start button 130... release condition setting screen 140... navigation display screen WK, WK4... observation object WK2... cylindrical workpiece WK3... observation object AX... optical axis α... tilt angle in side view β... tilt angle in plan view DJ1... tilt direction of joystick DS1... movement direction of stage part IC... icon MC... mouse cursors GS1 to GS6... geometric shape CB... substrate CP... chip RT... route PL... contour CS... field of view center PT... movement position RT'... route displayed by dashed line

Claims (23)

a stage portion for placing an object to be observed;
an objective lens unit arranged to face an observation object on the stage unit;
a camera unit that captures an image of an observation object formed through the objective lens unit and generates image data representing the image;
a display control unit that causes an image of an observation field including an observation object to be displayed on a display unit based on image data generated by the camera unit;
a field of view moving mechanism that moves the optical axis of the objective lens unit and the stage unit relatively to each other so that the position of the optical axis of the objective lens unit on the stage unit changes and the observation field of view output to the display unit by the display control unit moves;
a movement direction instructing unit that instructs a movement direction of the field of view moving mechanism in accordance with a user input indicating a movement direction of the observation field of view on the display unit;
a trajectory designation unit for designating a geometric shape and a plurality of reference points as trajectory information ;
a trajectory calculation unit that calculates a field of view movement trajectory that moves an observation field of view through a plurality of reference points by approximating the geometric shape based on the geometric shape specified by the trajectory specification unit and the plurality of reference points;
a movement control unit that moves the optical axis of the objective lens unit and the stage unit relative to each other using the field of view movement mechanism in accordance with a movement direction instruction from the movement direction instruction unit along the field of view movement trajectory calculated by the trajectory calculation unit, thereby moving the observation field of view output to the display unit by the display control unit ;
A magnification observation device comprising: The magnification observation device according to claim 1,
the trajectory instruction unit receives instructions of a plurality of trajectory reference points as the trajectory information using images of one or a plurality of observation fields moved by the field of view movement mechanism;
A magnification observation device, wherein the trajectory calculation unit is configured to calculate a field of view movement trajectory based on a plurality of trajectory reference points designated by the trajectory designation unit. The magnification observation device according to claim 1,
the trajectory instruction unit causes the display control unit to display images of a plurality of observation fields of view on the display unit by moving the field of view moving mechanism, and receives instructions of a plurality of trajectory reference points corresponding to each observation field of view as the trajectory information;
A magnification observation device, wherein the trajectory calculation unit is configured to calculate a field of view movement trajectory based on a plurality of trajectory reference points designated by the trajectory designation unit. The magnification observation device according to any one of claims 1 to 3,
The movement control unit
a route trace mode in which the movement direction instruction unit controls the visual field movement mechanism so as to follow the visual field movement trajectory calculated by the trajectory calculation unit;
a free mode in which the visual field movement mechanism is controlled in a movement direction instructed by the movement direction instructing unit, regardless of the visual field movement locus. The magnification observation device according to any one of claims 1 to 4, further comprising:
A magnification observation device comprising a route guide canceling unit that, during execution of a route trace mode in which the movement direction indicating unit controls the field of view movement mechanism to follow the field of view movement trajectory calculated by the trajectory calculating unit, cancels the route trace mode and sets the device to a free mode in which the field of view movement mechanism can be controlled in the movement direction indicated by the movement direction indicating unit, regardless of the field of view movement trajectory. The magnification observation device according to claim 4 or 5,
The movement control unit
a movement position to which the field of view movement mechanism is moved;
The visual field movement trajectory calculated by the trajectory calculation unit,
The magnification observation device is configured to switch from the route tracing mode to a free mode when the difference is equal to or greater than a predetermined value. The magnification observation device according to any one of claims 4 to 6,
The movement control unit is configured to:
The movement direction indicated by the movement direction indication unit is
In the direction in which the observation field of view should be moved along the field of view movement trajectory calculated by the trajectory calculation unit,
If the route is within a predetermined first angle, continue the route tracing mode along the first direction;
A magnification observation device configured to continue the route tracing mode along a second direction opposite to the first direction if the angle is outside a predetermined first angle range. The magnification observation device according to any one of claims 4 to 6,
The movement direction indicating unit is configured to indicate a first direction along the field of view movement trajectory and a second direction along the field of view movement trajectory that is different from the first direction. The magnification observation device according to claim 2 or 3,
the trajectory calculation unit calculates a visual field movement trajectory that passes through the plurality of trajectory reference points designated by the trajectory designation unit;
A magnification observation device, wherein the field of view movement trajectory calculated by the trajectory calculation unit includes a bent portion. The magnification observation device according to claim 2 or 3,
A magnification observation device, wherein the trajectory calculation unit is configured to calculate the field of view movement trajectory so as to interpolate a plurality of trajectory reference points designated by the trajectory designation unit with a geometric shape including a curve. The magnification observation device according to claim 9 or 10,
A magnification observation device in which the trajectory calculation unit is configured to calculate the trajectory of movement of the field of view using the coordinate position of the center of the observation field of view as a trajectory reference point. The magnification observation device according to any one of claims 1 to 11,
The field of view moving mechanism includes an electric XY stage that electrically moves the stage unit in the X and Y directions. The magnification observation device according to any one of claims 1 to 12,
the field of view moving mechanism changes the relative position between the optical axis of the objective lens unit and the stage unit, and changes the relative distance between the focal position of the objective lens unit and the stage unit;
the trajectory information indicated by the trajectory indication unit includes information regarding a plurality of reference points with different relative distances;
the trajectory calculation unit calculates a visual field movement trajectory on a plane passing through a plurality of reference points by approximating the geometric shape specified by the trajectory specification unit to the geometric shape;
A magnification observation device configured to move the observation field by changing the relative position of the optical axis of the objective lens unit and the stage unit using the field of view movement mechanism in accordance with the movement direction instruction from the movement direction instruction unit along the field of view movement trajectory calculated by the trajectory calculation unit, and to change the relative distance between the optical axis of the objective lens unit and the stage unit using the field of view movement mechanism . The magnification observation device according to claim 13,
the field-of-view movement mechanism includes an electric Z stage that electrically moves the stage unit in a Z direction,
A magnification observation device configured such that the electric Z stage automatically adjusts the focal length so that the focus of the image data displayed on the display unit by the display control unit is in focus according to the distance from the objective lens unit to the object to be observed. The magnification observation device according to any one of claims 1 to 14,
a joystick for specifying a direction and speed of movement of the observation field on the display unit;
the movement direction instruction unit receives a designation of a movement direction and a movement speed of the observation field of view on the display unit by the joystick,
The movement control unit controls the movement speed of the observation field by the field movement mechanism so that the greater the angle at which the joystick is tilted , the faster the movement speed of the observation field. The magnification observation device according to any one of claims 1 to 15, further comprising:
a field of view deviation correction unit that detects deviation of the center position of the observation field of view and automatically corrects the field of view deviation,
The magnification observation device is configured such that, when the field of view shift correction unit performs field of view shift correction, the trajectory calculation unit automatically offsets the field of view movement trajectory in accordance with the field of view shift correction. The magnification observation device according to claim 13, further comprising:
a field of view deviation correction unit that detects deviation of the center position of the observation field of view or the relative distance and automatically corrects the field of view deviation,
The magnification observation device is configured such that, when the field of view shift correction unit performs field of view shift correction, the trajectory calculation unit automatically offsets the field of view movement trajectory in accordance with the field of view shift correction. The magnification observation device according to any one of claims 1 to 17, further comprising:
a first illumination pattern that is capable of switching an illumination direction toward an observation object and that sequentially switches the illumination direction to irradiate the object;
a first illumination pattern that illuminates in a selected illumination direction; and a second illumination pattern that illuminates in a selected illumination direction.
A magnification observation device configured such that, when the observation field of view is moved along the field of view movement locus, the illumination unit irradiates with the first illumination pattern. The magnification observation device according to any one of claims 1 to 18,
the display control unit causes the display unit to display an image of the observation field including the observation object based on the image data generated by the camera unit;
When the movement of the observation field by the field movement mechanism is stopped, a composite image that has been image-combined based on the plurality of image data generated by the camera unit is displayed on the display unit;
A magnification observation device configured to display a live image based on image data generated by the camera unit when the observation field of view is moved by the field of view moving mechanism. The magnification observation device according to any one of claims 1 to 14 ,
a joystick for specifying a direction and speed of movement of the observation field on the display unit;
the trajectory calculation unit calculates a predicted arrival point from the time interval for monitoring the movement direction indicated by the joystick in addition to the movement direction and speed of the field of view movement mechanism ;
A magnification observation device that searches for a point on the trajectory that is closest to the calculated predicted point, corrects the direction and speed toward the closest point that has been searched, and controls the field of view movement mechanism to move. A magnified image observation method comprising the steps of: capturing an image of an observation object placed on a stage unit with a camera unit via an objective lens unit, displaying the image on a display unit; and relatively moving the optical axis of the objective lens unit and the stage unit with a field of view moving mechanism so that the position of the optical axis of the objective lens unit on the stage unit changes and the observation field of view output to the display unit moves,
a step in which a trajectory instruction unit prompts the user to specify a geometric shape and a plurality of reference points as trajectory information ;
a step of calculating , by a trajectory calculation unit, a field of view movement trajectory that moves the observation field of view through a plurality of reference points by approximating the geometric shape based on the geometric shape specified by the trajectory specification unit and the plurality of reference points;
a step of moving the optical axis of the objective lens unit and the stage unit relative to each other along the field of view movement trajectory calculated by the trajectory calculation unit, in accordance with a user input indicating a movement direction of the observation field of view on the display unit, and in accordance with a movement direction instruction by a movement direction instruction unit that instructs the movement direction of the field of view movement mechanism, thereby moving the observation field of view that is output to the display unit by a display control unit, and controlling the movement by the field of view movement mechanism by a movement control unit;
A method for observing a magnified image, comprising: a stage portion for placing an object to be observed;
an objective lens unit arranged to face an observation object on the stage unit;
a camera unit that captures an image of an observation object formed through the objective lens unit and generates image data representing the image;
a display unit that displays an image of an observation field including an observation object based on image data generated by the camera unit;
a field of view moving mechanism that moves the optical axis of the objective lens unit and the stage unit relatively to each other so that the position of the optical axis of the objective lens unit on the stage unit changes and the observation field of view output to the display unit moves;
a movement direction instructing unit that instructs a movement direction of the field of view moving mechanism in accordance with a user input indicating a movement direction of the observation field of view on the display unit;
A magnified image observation program for operating a magnification observation device comprising:
a trajectory designation function for designating a geometric shape and a plurality of reference points as trajectory information ;
a trajectory calculation function that calculates a field of view movement trajectory that moves an observation field of view through a plurality of reference points by approximating the geometric shape based on the geometric shape specified by the trajectory specification function and the plurality of reference points;
a movement control function that controls movement of the field of view movement mechanism, which moves the optical axis of the objective lens unit and the stage unit relatively along the field of view movement trajectory calculated by the trajectory calculation function, in accordance with a movement direction instruction by the movement direction instruction unit, thereby moving the observation field of view output to the display unit by the display control unit ; and
A magnified image observation program that enables this to be achieved on a computer. A computer-readable recording medium or storage device storing the program described in claim 22.