JP7687907B2 - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, IMAGE SHARPENING METHOD, AND PROGRAM - Google Patents

Description

The disclosure of this specification relates to an image processing device, an image processing system, an image sharpening method, and a program.

In recent years, the use of digital images acquired by microscopes has been increasing in pathological diagnosis and the inspection of industrial parts. However, when observing using digital images acquired by microscopes, compared to observation with the naked eye, blurring caused by the shallow depth of field of the optical system and slight optical inhomogeneity that occurs between the object point and the image plane are easily noticeable as reduced sharpness and contrast.

Image processing technology related to this problem is described, for example, in Patent Document 1. By using the unsharp mask processing described in Patent Document 1, high-frequency spatial frequency components can be added to the original image at any intensity to obtain a sharpened image.

JP 2001-053974 A

However, in the technology described in Patent Document 1, sharpening processing is performed uniformly on the entire image. As a result, in areas of the image with high sharpness, the sharpness is overemphasized, which can lead to deterioration of graininess (i.e., graininess noise becomes noticeable). In areas of the image with low sharpness, the sharpness is not sufficiently improved, resulting in blurring.

Note that while the above explanation has been given using a microscope image as an example, the adverse effects of uniform sharpening processing can occur with any digital image, not just microscope images.

In light of the above-mentioned circumstances, an object of one aspect of the present invention is to provide a technology that appropriately sharpens the entire image.

An image processing system according to one embodiment of the present invention comprises an image processing device including: a microscope device that scans a specimen to obtain a plurality of images; a classification unit that classifies a plurality of regions constituting the image into a plurality of classes; a sharpening processing unit that sharpens the plurality of regions with an intensity determined for each classified class; a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device; and a stitching unit that stitches together a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image that is wider than the field of view of the microscope device, wherein the sharpening processing unit changes the intensity of sharpening the plurality of regions in accordance with the display magnification of the sharpened image displayed on the display device, the intensity being higher as the display magnification is lower.
Another aspect of the present invention provides an image processing system comprising: a microscope device that scans a specimen to obtain a plurality of images; a classification unit that classifies a plurality of regions constituting the images into a plurality of classes; a sharpening processing unit that sharpens the plurality of regions with an intensity determined for each classified class; a display control unit that displays a sharpened image, which is an image sharpened by the sharpening processing unit, on a display device; and a stitching unit that stitches together a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image that is wider than the field of view of the microscope device, wherein the sharpening processing unit changes the intensity with which the plurality of regions are sharpened to an intensity corresponding to the display magnification of the sharpened image, which is determined based on contrast sensitivity characteristics.

An image sharpening method according to one embodiment of the present invention classifies, for each of a plurality of images obtained by scanning a specimen with a microscope device, a plurality of regions constituting the image into a plurality of classes, sharpens the plurality of regions with an intensity determined for each classified class, displays the sharpened images on a display device, and stitches together the plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device. Depending on the display magnification of the sharpened images displayed on the display device, the intensity of sharpening the plurality of regions is changed to a higher intensity the lower the display magnification.
A sharpening method according to another aspect of the present invention classifies, for each of a plurality of images obtained by scanning a specimen with a microscope device, a plurality of regions constituting the image into a plurality of classes, sharpens the plurality of regions with an intensity determined for each classified class, displays the sharpened images on a display device, and stitches together the plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device, and changes the intensity with which the plurality of regions are sharpened to an intensity corresponding to the display magnification of the sharpened image, which is determined based on contrast sensitivity characteristics.

A program according to one aspect of the present invention causes a computer to execute the following processes: for each of a plurality of images obtained by scanning a specimen with a microscope device , classify a plurality of regions constituting the image into a plurality of classes, sharpen the plurality of regions with an intensity determined for each classified class, display the sharpened images on a display device, stitch together the plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device, and change the intensity of sharpening the plurality of regions depending on the display magnification of the sharpened image displayed on the display device, with the intensity being higher the lower the display magnification.
A program according to another aspect of the present invention has a computer that classifies, for each of a plurality of images obtained by scanning a specimen with a microscope device, a plurality of regions constituting the image into a plurality of classes, sharpens the plurality of regions with an intensity determined for each classified class, displays the sharpened images on a display device, stitches together the plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device, and changes the intensity with which the plurality of regions are sharpened to an intensity corresponding to the display magnification of the sharpened image, which is determined based on the contrast sensitivity characteristic.

According to the above aspect, the entire image can be appropriately sharpened.

1 is a diagram illustrating an example of a configuration of an image processing system 1 according to an embodiment. FIG. 1 is a functional block diagram showing a functional configuration of an image processing device 100 according to an embodiment. 4 is an example of a flowchart of a process performed by the image processing device 100. 11A and 11B are diagrams for explaining a method of generating a sharpened image. 11 is a diagram for explaining the size of a region R. FIG. FIG. 1 is a functional block diagram showing a functional configuration of an image processing apparatus 101 according to a first embodiment. 4 is an example of a flowchart of a process performed by the image processing apparatus 101. FIG. 11 is a functional block diagram showing the functional configuration of an image processing apparatus 102 according to a second embodiment. FIG. 13 is a functional block diagram showing the functional configuration of an image processing device 103 according to a third embodiment. FIG. 13 is a functional block diagram showing the functional configuration of an image processing device 104 according to a fourth embodiment. FIG. 13 is a functional block diagram showing the functional configuration of an image processing device 105 according to a fifth embodiment. 1 is an example of a flowchart of a process performed by an image processing system including an image processing device 105. 10 is another example of a flowchart of the process performed by the image processing system including the image processing device 105. FIG. 13 is a functional block diagram showing the functional configuration of an image processing device 106 according to a sixth embodiment. FIG. 13 is a functional block diagram showing the functional configuration of an image processing device 107 according to a seventh embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer 200 for implementing the image processing apparatus.

Fig. 1 is a diagram illustrating an example of the configuration of an image processing system 1 according to one embodiment. First, the configuration of the image processing system 1 shown in Fig. 1 will be described with reference to Fig. 1. As shown in Fig. 1, the image processing system 1 includes an image processing device 100, one or more image capture devices 10, and one or more client terminals 20, which are connected to each other via a network 30.

The type of network is not particularly limited. The network may be, for example, a public line such as the Internet, a dedicated line, or a LAN (Local Area Network). The image processing device 100, the photographing device 10, and the client terminal 20 may be connected by wire or wirelessly.

The image processing device 100 is a device that performs processing to enhance the sharpness of an input image and outputs a sharpened image (hereinafter referred to as a sharpened image). The image processing device 100 does not sharpen the entire image uniformly, but rather classifies the regions that make up the image into classes and sharpens each region with an intensity according to the class. This makes it possible to generate a sharpened image in which the entire image is appropriately sharpened, even if the input image contains a mixture of regions with high and low sharpness. Details of the image processing device 100 will be described later.

The imaging device 10 is a device that photographs a specimen and acquires an image to be input to the image processing device 100. The image of the specimen generated by the imaging device 10 is output from the imaging device 10 to the image processing device 100 via the network 30, for example. However, the image generated by the imaging device 10 may also be input to the image processing device 100 via another device or recording medium.

The imaging device 10 may be any device that captures an image of a specimen and generates a digital image. The imaging device 10 may include, for example, a microscope device 11. The imaging device 10 may also include an observation device 12 that is placed in an incubator and captures time-lapse images of the specimen in the culture container. The imaging device 10 may also include a digital camera 13, and may further include any other imaging device, such as, for example, a smartphone or an endoscope.

The client terminal 20 is a terminal operated by a user and has an input device and a display device. The image processing device 100 may output a sharpened image to the client terminal 20, for example, in response to a request from the client terminal 20.

The client terminal 20 may be, for example, a portable type client terminal 21 such as a laptop computer, or a mobile type client terminal 22 such as a tablet or smartphone. It may also be a stationary type client terminal 23 such as a desktop computer.

Note that FIG. 1 shows an example in which the imaging device 10, the client terminal 20, and the image processing device 100 are separate devices in different locations, but the image processing system 1 only needs to have these functions, and the specific configuration is not limited to this example. The image processing system 1 may be, for example, a microscope system including a microscope device and a microscope control device, and an image acquired by the microscope device may be sharpened by the microscope control device, and the generated sharpened image may be displayed on a display device provided in the microscope control device.

Fig. 2 is a functional block diagram showing the functional configuration of an image processing device 100 according to one embodiment. Fig. 3 is an example of a flowchart of processing performed by the image processing device 100. Fig. 4 is a diagram for explaining a method for generating a sharpened image. Fig. 5 is a diagram for explaining the size of region R. Below, with reference to Figs. 2 to 5, the configuration common to the image processing devices according to each embodiment described later and the image sharpening method will be explained using the image processing device 100 as an example.

As shown in FIG. 2, the image processing device 100 includes an acquisition unit 110, a division unit 120, a classification unit 130, a sharpening processing unit 140, and an output unit 150.

When the process shown in FIG. 3 is started, in the image processing device 100, first, the acquisition unit 110 acquires an image of the specimen captured by the imaging device 10 (step S1). Here, the acquisition unit 110 reads, for example, image 41 of FIG. 4 stored in an auxiliary storage device (not shown) of the image processing device 100 into a main storage device (not shown). Image 41 is, for example, an image of a pathological specimen captured by the microscope device 11. Image 41 shows a large number of cells densely packed together.

When the image is acquired by the acquisition unit 110, the division unit 120 divides the image acquired by the acquisition unit 110 into a plurality of regions (step S2). Here, the division unit 120 divides the image 41 into a plurality of regions R of a certain size, for example. The certain size is not particularly limited, but as shown in the region information 42 in FIG. 4, the region division may be performed on the image 41 so that each region is a square region (region R) of n×n pixels in size. In addition, it is desirable that the certain size is a size that includes at least one cell, for example, as shown in FIG. 5. This point will be described in detail later.

When the image is divided into multiple regions, the classification unit 130 classifies the multiple regions constituting the image into multiple classes (step S3). Here, the classification unit 130 assigns one of multiple classes to each of the multiple regions R, for example, as shown in class information 43 in FIG. 4. There is no particular limit to the number of classes classified by the classification unit 130. However, as described below, the number of times the sharpening process is performed depends on the number of classes, and the more classes there are, the more times the process is performed. For this reason, it is desirable that the number of classes is not too large. Note that the class information 43 in FIG. 4 shows an example in which multiple regions R are classified into four classes.

When the regions are classified into classes, the sharpening processing unit 140 sharpens the multiple regions with an intensity determined for each classified class (step S4). Here, the sharpening processing unit 140 performs contour enhancement processing such as unsharp masking on the image 41 so that each region R of the image 41 is sharpened with an intensity corresponding to the class to which the region R is assigned, and generates a sharpened image 44. Note that the specific method of sharpening the image 41 with different intensities depending on the region in the image 41 is not particularly limited. For example, if multiple regions R are classified into four classes, the image 41 may be sharpened with four different intensities (e.g., strong, medium, weak, and very weak) corresponding to each class to generate four class images, and information on the region assigned to the class may be extracted from each of the four generated class images and combined to generate one sharpened image 44. Note that the intensity determined by the sharpening processing unit 140 also includes an intensity for sharpening by 1x. When sharpening at 1x, the sharpening processor 140 may omit the sharpening process.

Once the sharpened image is generated, the output unit 150 outputs the sharpened image (step S5). Here, the output unit 150 outputs the sharpened image 44 to, for example, the client terminal 20. The client terminal 20 that receives the sharpened image from the image processing device 100 displays the sharpened image 44 on, for example, a display device of the client terminal 20.

In the image processing device 100 configured as described above, the multiple regions constituting an image are classified into multiple classes, and the multiple regions are sharpened with an intensity according to the classified class. This makes it possible to sharpen each region with an appropriate intensity even when the image contains a mixture of regions with high and low sharpness. This makes it possible to avoid situations where a specific region is overly or insufficiently sharpened, and to appropriately sharpen the entire image.

In addition, in the image processing device 100, classification into classes is performed in units of regions, rather than in units of pixels, which are the smallest unit of an image. This feature greatly contributes to the image processing device 100 sharpening images appropriately. This point will be described in detail below.

Differences in sharpness in an image are caused by various factors. For example, in the case of a microscope image, aberration is generally well corrected near the optical axis and sharpness is high, whereas in the peripheral areas of the field of view, the image is easily blurred due to the effects of aberration and sharpness is low. Such differences in sharpness caused by the performance of the device usually occur in units larger than the pixel unit. Also, if the specimen is placed at an angle to the optical system, areas with high and low sharpness may occur due to differences in the degree of focus. Sharpness differences caused by the shooting environment, such as the way the specimen is placed, also occur in units larger than the pixel unit, just like those caused by the performance of the device. In light of the goal of observing the specimen well, it is desirable to correct these sharpness differences so that they are small during the sharpening process.

On the other hand, it is not necessarily desirable to correct differences in sharpness caused by the specimen itself, which is the object of observation, and in some cases it may be desirable to maintain the differences as they are. Taking the cell shown in Figure 5 as an example, differences in sharpness may occur within the cell depending on the cell structure. Specifically, there are areas of high and low sharpness within the cell, such as the tendency for the cell nucleus to be visualized with better contrast than the cell membrane. However, since such differences in sharpness caused by the cell structure are thought to represent the cell structure itself, it is not necessarily desirable to correct these differences by sharpness processing. Differences in sharpness caused by the structure occur on a pixel basis or in units close to that, depending on the observation magnification.

The image processing device 100 therefore uses the difference in units, i.e., spatial size, between the sharpness difference to be corrected and the sharpness difference to be not corrected, to effectively correct only the sharpness difference to be corrected. Specifically, it uses the relationship that the sharpness difference to be not corrected due to the structure of the specimen, etc. occurs in smaller units than the sharpness difference to be corrected due to the device, environment, etc. More specifically, the image processing device 100 divides the image into intermediate units smaller than the above-mentioned large units and larger than the above-mentioned small units, classifies each of the multiple regions obtained by dividing the image into classes, and performs sharpening processing on the regions classified into the classes with strength according to the class.

In this way, by performing the sharpening process using regions, rather than pixels, as the smallest unit, sharpening is uniformly performed at the same intensity within at least one region. Therefore, for objects smaller than a region, such as cells, it is possible to appropriately sharpen the image while maintaining differences in sharpness due to their structure. In addition, differences in sharpness due to device performance or the shooting environment can be corrected so that the differences in sharpness are reduced by performing the sharpening process with the intensity changed on a region-by-region basis.

Each embodiment of the present invention will be specifically described below.
[First embodiment]
Fig. 6 is a functional block diagram showing the functional configuration of the image processing apparatus 101 according to this embodiment. Fig. 7 is an example of a flowchart of the process performed by the image processing apparatus 101.

As shown in FIG. 6, the image processing device 101 differs from the image processing device 100 in that the image processing device 101 includes a detection unit 160 that detects the sharpness of multiple regions based on an image. The detection unit 160 further includes a calculation unit 161 that calculates the sharpness distribution of the image, and a determination unit 162 that determines the sharpness of multiple regions based on the sharpness distribution calculated by the calculation unit 161.

When the image processing device 101 configured as described above starts the process shown in FIG. 7, the acquisition unit 110 first acquires an image of the specimen captured by the imaging device 10 (step S11). This process is similar to the process of step S1 in FIG. 3.

Next, the division unit 120 divides the image acquired in step S11 into multiple regions of a predetermined size (step S12). This predetermined size is a size determined in advance, for example, n x n pixels.

Then, the calculation unit 161 calculates the sharpness distribution of the image acquired in step S11 (step S13). Here, the calculation unit 161 performs, for example, a smoothing differential process on the image. By extracting edges while smoothing, it is possible to calculate a sharpness distribution in which the effects of noise are suppressed. Specifically, a Sobel filter can be used, but the sharpness distribution may also be calculated using other edge detection methods, such as the Kenny edge detection algorithm.

If the image is a color image, the color image may be converted to a grayscale image, and the sharpness distribution may be calculated for the resulting grayscale image.

When the sharpness distribution is detected, the determination unit 162 determines the sharpness of the multiple regions based on the sharpness distribution (step S14). Here, for example, the determination unit 162 performs an averaging process on the sharpness within each of the multiple regions identified by the division unit 120. In other words, the determination unit 162 determines the average value of the sharpness at each position within the region as the sharpness of the region. Note that the sharpness of a region need only be a value that represents the sharpness distribution of the region, and is not necessarily limited to the average value. For example, the median or mode may be used.

Once the sharpness for each region has been determined, the classification unit 130 classifies the regions into multiple classes according to the sharpness of the regions determined in step S14 (step S15). Here, the classification unit 130 classifies the regions into multiple classes, for example, based on a predetermined threshold value. That is, the classification unit 130 classifies the regions into one of multiple classes by comparing the threshold value with the sharpness of the region. The threshold value may be an absolute value (e.g., a sharpness of 100) or a relative value (e.g., 1/2 the difference between the maximum sharpness and the minimum sharpness).

As an example of the output result of step S15, for example, filters may be generated that are the same number as the number of classes and each filter is made up of multiple pixels corresponding to multiple regions. In this case, the pixels of each filter are binary data that distinguish between the classified region of the corresponding class (0) and the rest (1).

Once the regions are classified into classes, the sharpening processor 140 sharpens the regions for each class with an intensity corresponding to the sharpness of the region classified into that class (step S16). Here, the sharpening processor 140 generates a sharpened image by sharpening the regions for each class with a higher intensity the lower the sharpness of the region classified into that class. More specifically, each region may be sharpened with an intensity that is inversely proportional to the sharpness.

In step S16, the entire image may be sharpened with a strength according to the class, thereby generating the same number of images as the number of classes. After that, a filter corresponding to each class generated in step S15 may be used on the image corresponding to each class to extract only information on the area classified into that class. Specifically, for example, an image corresponding to each class may be filtered with a filter corresponding to each class, and the filtered image may be subtracted from the image corresponding to the class to extract only information on the area classified into that class. A single sharpened image may be generated by combining the information extracted for each filter class.

Once the sharpened image is generated, the output unit 150 outputs the sharpened image (step S17). This process is similar to the process of step S5 in FIG. 3.

The image processing device 101 according to this embodiment can appropriately sharpen the entire image, similar to the image processing device 100. Furthermore, the image processing device 101 classifies multiple regions based on the sharpness distribution of the image, making it possible to perform classification that reflects the sharpness of the image to be sharpened. Furthermore, by sharpening each region with a strength according to the sharpness of the region, it becomes possible to effectively suppress the sharpness difference between regions.

Second Embodiment
Fig. 8 is a functional block diagram showing the functional configuration of the image processing device 102 according to this embodiment. As shown in Fig. 8, the image processing device 102 differs from the image processing device 101 according to the first embodiment in that the image processing device 102 includes a dividing unit 120, a classification unit 130, and a setting unit 170 that sets the operation parameters of the sharpening processing unit 140. The other configurations are the same as those of the image processing device 101.

The setting unit 170 sets the operating parameters according to input from the user, for example, setting each operating parameter to a value specified by the user. Specifically, the region size, which is an operating parameter used in the division process in the division unit 120, the number of classes and threshold, which are operating parameters used in the classification process in the classification unit 130, and a coefficient for determining the sharpening strength, which is an operating parameter used in the sharpening process in the sharpening processing unit 140, may be set according to user specifications. In addition, it may be set to omit the sharpening process itself for a specific class.

The image processing device 102 according to this embodiment, like the image processing device 101, is capable of appropriately sharpening the entire image and effectively suppressing the sharpness difference between regions. Furthermore, according to the image processing device 102, the setting unit 170 can adjust the operating parameters, so that the image can be sharpened according to the characteristics of the image (specimen) and the user's preferences.

Note that, in the image processing device 102, an example has been shown in which the setting unit 170 sets the operating parameters of the division unit 120, classification unit 130, and sharpening processing unit 140, but the setting unit 170 only needs to set the operating parameters of at least any one of these.

[Third embodiment]
Fig. 9 is a functional block diagram showing the functional configuration of the image processing device 103 according to this embodiment. As shown in Fig. 9, the image processing device 103 is similar to the image processing device 101 according to the first embodiment in that it includes a detection unit 160 that detects the sharpness of multiple regions based on an image, and is also similar to the image processing device 101 in that the sharpness of multiple regions detected by the detection unit 160 is used in the classification unit 130. However, the image processing device 103 differs from the image processing device 101 in that the sharpness of multiple regions detected by the detection unit 160 is not used in the sharpening processing unit 140.

In the image processing device 103, the sharpening processing unit 140 sharpens multiple regions, for example, with a predetermined strength, for each class classified according to the sharpness of the multiple regions. Note that although the sharpening processing unit 140 does not directly reflect the sharpness of the region, for example, inversely proportional to the sharpness of the region, it is desirable to sharpen the region with a lower predetermined strength as the sharpness of the region increases.

The image processing device 103 according to this embodiment, like the image processing device 101, is capable of appropriately sharpening the entire image, and is able to effectively suppress the sharpness difference between regions.

[Fourth embodiment]
Fig. 10 is a functional block diagram showing a functional configuration of an image processing device 104 according to this embodiment. As shown in Fig. 10, the image processing device 104 is similar to the image processing device 101 according to the first embodiment in that it includes a detection unit 160 that detects the sharpness of multiple regions based on an image, and is also similar to the image processing device 101 in that the sharpness of multiple regions detected by the detection unit 160 is used in the sharpening processing unit 140. However, the image processing device 104 differs from the image processing device 101 in that the sharpness of multiple regions detected by the detection unit 160 is not used in the classification unit 130.

In the image processing device 104, the classification unit 130 classifies a plurality of regions into a plurality of classes according to, for example, a predetermined distance from the center of the visual field. More specifically, the regions may be classified into a class for strong sharpening, a class for medium sharpening, and a class for weak sharpening, starting from the region closest to the center of the visual field. The sharpening processing unit 140 then determines the specific sharpening strength according to the sharpness of the region classified into a class and performs the sharpening.

The image processing device 104 according to this embodiment, like the image processing device 101, is capable of appropriately sharpening the entire image, and is able to effectively suppress the sharpness difference between regions.

[Fifth embodiment]
Fig. 11 is a functional block diagram showing the functional configuration of an image processing device 105 according to this embodiment. As shown in Fig. 11, the image processing device 105 differs from the image processing device 101 according to the first embodiment in that it includes a stitching unit 180 that stitches together a plurality of sharpened images to generate a stitched image. The other configurations are the same as those of the image processing device 101.

In this embodiment, the microscope device 11 uses an objective lens with a relatively high magnification to scan the specimen by sequentially moving the field of view relative to the specimen to obtain multiple images. The image processing device 105 sharpens each of the multiple images thus obtained, which correspond to a range wider than the field of view of the microscope device 11, and then stitches them together to generate a stitched image that is wider than the field of view of the microscope device 11 and has a higher resolution.

The image processing device 105 according to this embodiment, like the image processing device 101, can appropriately sharpen the entire image and effectively suppress the sharpness difference between regions. In addition, due to a general characteristic of a microscope device in which the image quality of the peripheral part of the field of view is degraded compared to the central part, if images acquired by the microscope device are directly stitched together, a periodic stripe pattern that depends on the stitching period may appear in the stitched image. According to the image processing device 105, by stitching together sharpened images that suppress the sharpness difference between regions to generate a stitched image, it is possible to prevent or suppress the occurrence of such artifacts, and a high-quality stitched image in which the stitched parts are not noticeable can be generated. In addition, improvement in the stitching accuracy itself can also be expected.

Figures 12 and 13 are flowcharts of the processing performed by an image processing system including the image processing device 105. When generating a composite image, the sharpening process may be performed synchronously with image acquisition, or may be performed asynchronously with image acquisition.

For example, as shown in FIG. 12, when the image processing system receives an instruction from a user to generate a composite image (step S21), the microscope device 11 acquires an image (step S22), and the image processing device 105 performs a sharpening process on the image acquired by the microscope device 11 (step S23) are repeated a required number of times. After that, when the repeated process ends (step S24 NO), the image processing device 105 may generate a composite image using the multiple sharpened images sharpened in step S23 (step S25).

Also, for example, as shown in FIG. 13, in the image processing system, the microscope device 11 may acquire images as appropriate (steps S31 and S32). After that, when an instruction to generate a composite image including a designation of an image to be used is received from the user (step S33), the imaging device 10 may sharpen the designated images in sequence (steps S34 and S35), and generate a composite image using the generated multiple sharpened images (step S36).

Although the relationship between the timing of image acquisition and the timing of sharpening processing has been described above, a check box for setting whether or not to perform sharpening processing on the user interface may be provided so that the user can select whether or not to perform sharpening processing before stitching the images when instructing the generation of a stitched image. This allows the user to select to omit sharpening processing in order to shorten the processing time when generating a stitched image if degradation in image quality is acceptable. In addition, a check box for setting whether or not to save the image before sharpening processing may be further provided. This makes it possible to omit saving the image before sharpening processing and save memory capacity.

Sixth embodiment
Fig. 14 is a functional block diagram showing the functional configuration of an image processing device 106 according to this embodiment. As shown in Fig. 14, the image processing device 106 differs from the image processing device 101 according to the first embodiment in that it includes a display control unit 151 instead of the output unit 150 and in that there is feedback from the display control unit 151 to the sharpening processing unit 140. The other configurations are similar to those of the image processing device 101.

The display control unit 151 displays the sharpened image on a display device such as the client terminal 20. The display control unit 151 further feeds back to the sharpening processing unit 140 the display magnification of the sharpened image displayed on the display device. The sharpening processing unit 140 changes the strength of sharpening the multiple regions according to the display magnification fed back from the display control unit 151. At this time, the sharpening processing unit 140 may determine the sharpening strength using, for example, a mathematical model of the contrast sensitivity characteristics. Note that the display magnification fed back may be the overall magnification, or may be the digital zoom magnification in the display control unit 151 for the sharpened image output from the sharpening processing unit 140.

The image processing device 106 according to this embodiment, like the image processing device 101, is capable of appropriately sharpening the entire image, and is capable of effectively suppressing the difference in sharpness between regions. In addition, by feeding back the display magnification to the sharpening processing unit 140, an image sharpened with an intensity optimized for the display magnification can be displayed on the display device.

For example, in the case of a stitched image, it is known that the stitching boundary is more noticeable at lower magnifications, and is not noticeable at higher magnifications. This is related to the fact that the contrast sensitivity of the human eye is generally maximized at a spatial frequency of about 10 cycles/deg. In this way, the need for sharpening processing may differ depending on the display magnification. Users who obtain analysis results from images, such as researchers who use microscope images, include users who do not like image processing, but according to the image processing device 105, it is also possible to provide users with a sharpened image that has not been processed when the display magnification is high.

[Seventh embodiment]
Fig. 15 is a functional block diagram showing the functional configuration of an image processing device 107 according to this embodiment. As shown in Fig. 15, the image processing device 107 differs from the image processing device 101 according to the first embodiment in that the image processing device 107 includes a detection unit 190 instead of the detection unit 160. The other configurations are similar to those of the image processing device 101.

The detection unit 190 detects the sharpness characteristics of the image capturing device 10 that captures the image to be sharpened. The sharpness characteristics of the image capturing device 10 may be, for example, an aberration characteristic such as field curvature possessed by the image capturing device 10. The sharpness characteristics may be calculated from an image of the calibration sample obtained by capturing an image of the calibration sample having a flat surface using the image capturing device 10.

In the image processing device 107, the classification unit 130 classifies multiple regions into multiple classes according to the sharpness characteristics of the image capturing device 10 detected by the detection unit 190, rather than the sharpness distribution of the image. Furthermore, the sharpening processing unit 140 sharpens the multiple regions for each classified class with a strength according to the portion of the sharpness characteristics that corresponds to the region classified into that class, rather than the sharpness of the region classified into that class.

The image processing device 107 according to this embodiment, like the image processing device 101, is capable of appropriately sharpening the entire image and effectively suppressing the sharpness difference between regions. This is particularly suitable for observing specimens whose flatness is guaranteed, such as those used in industrial systems, and by omitting the process of detecting the sharpness distribution from the image, the entire image can be appropriately sharpened in a shorter time.

In the image processing device 107, an example has been shown in which the sharpness characteristics are used by both the classification unit 130 and the sharpening processing unit 140, but the sharpness characteristics may be used by only one of the classification unit 130 and the sharpening processing unit 140.

Fig. 16 is a diagram illustrating an example of the hardware configuration of a computer 200 for realizing the image processing device described above. The hardware configuration shown in Fig. 16 includes, for example, a processor 201, a memory 202, a storage device 203, a reading device 204, a communication interface 206, and an input/output interface 207. The processor 201, the memory 202, the storage device 203, the reading device 204, the communication interface 206, and the input/output interface 207 are connected to each other, for example, via a bus 208.

The processor 201 may be, for example, a single processor, a multiprocessor, or a multicore processor. The processor 201 reads out and executes a program stored in the storage device 203, thereby operating as the acquisition unit 110, division unit 120, classification unit 130, sharpening processing unit 140, output unit 150, detection unit 160, setting unit 170, stitching unit 180, and detection unit 190 described above.

The memory 202 may be, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 203 may be, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

The reading device 204 accesses the storage medium 205 according to instructions from the processor 201, for example. The storage medium 205 is realized by, for example, a semiconductor device, a medium where information is input and output by magnetic action, or a medium where information is input and output by optical action. An example of a semiconductor device is a USB (Universal Serial Bus) memory. An example of a medium where information is input and output by magnetic action is a magnetic disk. An example of a medium where information is input and output by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

The communication interface 206 communicates with other devices, for example, according to instructions from the processor 201. The input/output interface 207 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, a mouse, or a touch panel that accepts instructions from a user. The output device is, for example, a display device such as a display, and an audio device such as a speaker. The acquisition unit 110 and the output unit 150 described above may include at least one of the communication interface 206 or the input/output interface 207.

The program executed by the processor 201 is provided to the computer 200 in, for example, the following form.
(1) It is pre-installed in the storage device 203.
(2) Provided by storage medium 205.
(3) Provided from a server such as a program server.

The hardware configuration of the computer 200 for realizing the image processing device described with reference to FIG. 16 is an example, and the embodiment is not limited to this. For example, part of the above-mentioned configuration may be deleted, or new configuration may be added. In another embodiment, for example, part or all of the functions of the above-mentioned electrical circuit may be implemented as hardware using an FPGA (Field Programmable Gate Array), a SoC (System-on-a-Chip), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or the like.

The above-mentioned embodiments are illustrative examples for the purpose of facilitating understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-mentioned embodiments and alternatives to the above-mentioned embodiments may be included. In other words, the components of each embodiment can be modified without departing from the spirit and scope of the embodiment. In addition, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. Some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction. In other words, the image processing device, image processing system, image sharpening method, and program of the present invention can be modified and changed in various ways without departing from the scope of the claims.

1 Image processing system 10 Photographing device 11 Microscope device 12 Observation device 13 Digital camera 20 to 23 Client terminal 30 Network 41 Image 42 Region information 43 Class information 44 Sharpened image 100 to 107 Image processing device 110 Acquisition unit 120 Division unit 130 Classification unit 140 Sharpening processing unit 150 Output unit 151 Display control unit 160, 190 Detection unit 161 Calculation unit 162 Determination unit 170 Setting unit 180 Joint unit 200 Computer 201 Processor 202 Memory 203 Storage device 204 Reading device 205 Storage medium 206 Communication interface 207 Input/output interface 208 Bus R Region

Claims (18)

a microscope device for scanning a specimen to obtain a plurality of images;
an image processing device including: a classification unit that classifies a plurality of regions constituting the image into a plurality of classes; a sharpening processing unit that sharpens the plurality of regions with an intensity determined for each classified class; a display control unit that displays a sharpened image that is an image sharpened by the sharpening processing unit on a display device; and a stitching unit that stitches together a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image that is wider than the field of view of the microscope device;
The image processing system is characterized in that the sharpening processing unit changes the strength of sharpening the multiple regions depending on the display magnification of the sharpened image displayed on the display device, with the strength being higher as the display magnification is lower. a microscope device for scanning a specimen to obtain a plurality of images;
an image processing device including: a classification unit that classifies a plurality of regions constituting the image into a plurality of classes; a sharpening processing unit that sharpens the plurality of regions with an intensity determined for each classified class; a display control unit that displays a sharpened image that is an image sharpened by the sharpening processing unit on a display device; and a stitching unit that stitches together a plurality of sharpened images, each of which is an image sharpened by the sharpening processing unit, to generate a stitched image that is wider than the field of view of the microscope device;
The image processing system according to claim 1, wherein the sharpening processing unit changes the strength of sharpening the plurality of regions to a strength corresponding to a display magnification of the sharpened image, the display magnification being determined based on a contrast sensitivity characteristic. 3. The image processing system according to claim 1,
The image processing device further includes a detection unit that detects sharpness of the plurality of regions based on the image,
The image processing system according to claim 1, wherein the classification unit classifies the plurality of regions into the plurality of classes according to sharpness of the plurality of regions detected by the detection unit. 4. The image processing system according to claim 3,
The image processing system is characterized in that the sharpening processing unit sharpens the multiple regions for each classified class with a strength corresponding to the sharpness detected by the detection unit for the region classified into that class. 3. The image processing system according to claim 1,
The image processing device further includes a detection unit that detects sharpness of the plurality of regions based on the image,
The image processing system is characterized in that the sharpening processing unit sharpens the multiple regions for each classified class with a strength corresponding to the sharpness detected by the detection unit for the region classified into that class. 6. The image processing system according to claim 3,
The image processing system is characterized in that the sharpening processing unit sharpens the plurality of regions for each classified class with a higher sharpness level as the sharpness level of the region classified into that class decreases. 7. The image processing system according to claim 3,
The detection unit is
A calculation unit that calculates a sharpness distribution of the image;
a determination unit that determines sharpness of the plurality of regions based on the sharpness distribution calculated by the calculation unit. 8. The image processing system according to claim 7,
The image processing system according to claim 1, wherein the calculation unit performs smoothing differentiation processing on the image. 9. The image processing system according to claim 7,
The image processing system according to claim 1, wherein the determination unit performs an averaging process on the sharpness within each of the plurality of regions. 3. The image processing system according to claim 1,
The image processing device further includes a detection unit that detects a sharpness characteristic of an image capturing device that captures the image, the image capturing device being the microscope device;
The image processing system according to claim 1, wherein the classification unit classifies the plurality of regions into the plurality of classes according to the sharpness characteristics detected by the detection unit. 11. The image processing system according to claim 10,
The image processing system is characterized in that the sharpening processing unit sharpens the multiple regions for each classified class with a strength corresponding to the portion of the sharpness characteristics detected by the detection unit that corresponds to the region classified into that class. 3. The image processing system according to claim 1,
The image processing device further includes a detection unit that detects a sharpness characteristic of an image capturing device that captures the image, the image capturing device being the microscope device;
The image processing system is characterized in that the sharpening processing unit sharpens the multiple regions for each classified class with a strength corresponding to the portion of the sharpness characteristics detected by the detection unit that corresponds to the region classified into that class. 13. The image processing system according to claim 1,
The image processing system according to the present invention, wherein the image processing device further comprises a division unit that divides the image into the plurality of regions of a predetermined size. 14. The image processing system according to claim 13,
The image processing device further includes a setting unit that sets an operation parameter of at least one of the classification unit, the sharpening processing unit, and the division unit,
The image processing system according to claim 1, wherein the setting unit sets operation parameters in response to an input from a user. For each of a plurality of images acquired by scanning a specimen with a microscope device,
Classifying a plurality of regions constituting the image into a plurality of classes;
sharpening the plurality of regions with a strength determined for each classified class;
Displaying the sharpened image on a display device;
stitching together a plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device;
An image sharpening method, comprising changing the strength of sharpening the plurality of regions depending on a display magnification of the sharpened image displayed on the display device, the strength increasing as the display magnification decreases. For each of a plurality of images acquired by scanning a specimen with a microscope device,
Classifying a plurality of regions constituting the image into a plurality of classes;
sharpening the plurality of regions with a strength determined for each classified class;
Displaying the sharpened image on a display device;
stitching together a plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device;
11. A method for sharpening an image, comprising changing an intensity for sharpening the plurality of regions to an intensity corresponding to a display magnification of the sharpened image, the display magnification being determined based on a contrast sensitivity characteristic. On the computer,
For each of a plurality of images acquired by scanning a specimen with a microscope device,
Classifying a plurality of regions constituting the image into a plurality of classes;
sharpening the plurality of regions with a strength determined for each classified class;
Displaying the sharpened image on a display device;
stitching together a plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device;
A program characterized by executing a process of changing the strength of sharpening the multiple regions to a higher strength as the display magnification of the sharpened image displayed on the display device is lower, depending on the display magnification. On the computer,
For each of a plurality of images acquired by scanning a specimen with a microscope device,
Classifying a plurality of regions constituting the image into a plurality of classes;
sharpening the plurality of regions with a strength determined for each classified class;
displaying the sharpened image on a display device;
stitching together a plurality of sharpened images, each of which is a sharpened image, to generate a stitched image that is larger than the field of view of the microscope device;
A program for executing a process of changing the intensity of sharpening the plurality of regions to an intensity corresponding to a display magnification of the sharpened image, the display magnification being determined based on a contrast sensitivity characteristic.

Images