JP7833722B2 - Learning device, three-dimensional measurement system, learning method, and program - Google Patents

Description

This invention relates to a height image generation device, a learning device, a three-dimensional measurement system, a method for generating height images, a learning method, and a program.

A three-dimensional measuring machine is known that uses a white light interferometer to generate an interference fringe image based on the height-direction topography of the sample surface by irradiating the sample surface and a reference surface with white light, and interfering the reflected light from the sample surface with the reference light reflected from the reference surface. The measurement results of the topography of the sample surface are output as a height image (see, for example, Patent Document 1).

Japanese Patent Publication No. 2018-48914

Chen Kai, "Measurement of extremely thin transparent electrode ito film thickness based on amplitude and phase information of white light interference," Proceedings of the Institute of Electrical Engineers of Japan, IM = The papers of technical meeting on instrumentation and measurement, IEE Japan, Institute of Electrical Engineers of Japan, vol. 2017, pp. 13-15.

L. I. Mingzhou, “DEVELOPMENT OF FRINGE ANALYSIS TECHNIQUES IN WHITE LIGHT INTERFEROMETRY FOR MICRO-COMPONENT MEASUREMENT,” Tech. Rep., 2008

Such three-dimensional measuring machines can output highly accurate height images (height maps) of the sample surface by using high-contrast interference fringe images where the interference fringes are clearly visible. In other words, if the sample surface has a steep slope or vertical wall, or if the sample surface is made of a material that does not reflect light well, the contrast of the interference fringe image becomes low, and the three-dimensional measuring machine may output a height image superimposed with noise that looks like spike noise.

Therefore, the present invention has been made in view of these points, and aims to enable a three-dimensional measuring machine using a white light interferometer to output a height image with reduced noise, even when the contrast of the interference fringe image is low due to the shape and material of the sample surface.

In a first embodiment of the present invention, a height image generation device is provided for a three-dimensional measuring machine using a white light interferometer that irradiates a sample surface and a reference surface with white light, interferes the reflected light from the sample surface with the reference light reflected from the reference surface, and generates an interference fringe image based on the height-direction irregularities of the sample surface. The device includes: a volume data acquisition unit that acquires a plurality of interference fringe images as volume data each time the optical path difference between the reference surface and the sample surface is varied by a predetermined value; a learning model that indicates the expected value of the height of the sample surface in two-dimensional coordinates and outputs height data corresponding to the volume data in response to the input of the volume data; and a display control unit that displays the height image of the sample surface on a display unit based on the height data.

The learning model may further include a centering processing unit that calculates a moving average of the pixel data values for each identical two-dimensional coordinate of the multiple interference fringe images contained in the volume data, and subtracts the calculated moving average from the pixel data values used to calculate the moving average.

The learning model includes a convolutional neural network (CNN) comprising a network layer that calculates multiple output interference fringe images corresponding to multiple interference fringe images contained in the input volume data as output volume data, and a conversion processing unit that converts the calculated output volume data into a probability distribution of the height of the sample surface in two-dimensional coordinates, and a height data calculation unit that calculates the height data based on the probability distribution of the height of the sample surface. The CNN may be trained to reduce noise generated in the height data calculated from the output volume data.

In a second aspect of the present invention, a learning device for training the learning model of the height image generation device of the first aspect is provided, comprising: a learning volume data conversion unit that converts reference height data, which indicates the height of a reference sample surface in two-dimensional coordinates, into learning volume data corresponding to the reference height data; a volume data supply unit that supplies the converted learning volume data to the height image generation device; a height data acquisition unit that acquires output height data output by the height image generation device corresponding to the learning volume data; a first loss function calculation unit that calculates a first loss function based on a comparison result between the acquired output height data and the reference height data; and a learning unit that trains the learning model based on the calculation result of the first loss function, wherein the learning volume data includes a plurality of learning interference fringe images in which the brightness of at least some of the interference fringes is reduced corresponding to the shape of the reference sample surface indicated by the reference height data.

The learning device further comprises a volume data conversion unit that converts the aforementioned reference height data into reference volume data corresponding to the reference height data, including multiple reference interference fringe images. The volume data conversion unit further converts the output height data acquired by the height data acquisition unit into corresponding converted volume data. The learning device further comprises a second loss function calculation unit that calculates a second loss function based on a comparison result between the converted converted volume data and the reference volume data. The learning unit may learn the learning model based on the calculation results of the first and second loss functions.

The first loss function includes at least one of the following: a first function for calculating the similarity of the normal vectors of the height image; a second function for calculating the Harris Corner detection response value based on the difference between the output height data and the reference height data; and a third function for calculating the L1 loss between the output height data and the reference height data. The second loss function may also include a fourth function for calculating the L1 loss between the converted volume data and the reference volume data.

In a third aspect of the present invention, a three-dimensional measuring system is provided, comprising: a three-dimensional measuring machine using a white light interferometer that irradiates the sample surface and the reference surface with white light, and generates an interference fringe image based on the height-direction irregularities of the sample surface by interfering the reflected light from the sample surface with the reference light reflected from the reference surface; the three-dimensional measuring machine outputting a plurality of the interference fringe images generated each time the optical path difference between the reference surface and the sample surface is changed by a predetermined value; and a height image generation device of the first aspect that acquires the output plurality of interference fringe images as volume data and displays the height image of the sample surface corresponding to the volume data on a display unit.

The height image generation device further includes a storage unit that stores multiple learning parameters obtained by learning the learning model using multiple different reference height data, and a reception unit that receives a request to display a second height image, different from the first height image, on the display unit using parameters different from the first height image, after the first height image of the sample surface corresponding to one volume data has been displayed on the display unit using the first learning parameters. The learning model outputs the height data corresponding to the one volume data using the second learning parameters, different from the first learning parameters, in response to the reception unit receiving a request to display the second height image on the display unit. The display control unit may then generate a second height image of the sample surface corresponding to the one volume data based on the height data and display it on the display unit.

In a fourth aspect of the present invention, a method for generating a height image is provided for a three-dimensional measuring machine using a white light interferometer, which is operated by a computer. This method involves irradiating a sample surface and a reference surface with white light, interfering the reflected light from the sample surface with the reference light reflected from the reference surface, to generate an interference fringe image based on the height-direction topography of the sample surface. The method includes the steps of: acquiring a plurality of interference fringe images as volume data each time the optical path difference between the reference surface and the sample surface is varied by a predetermined value; outputting height data indicating the expected value of the height of the sample surface in the two-dimensional coordinate system corresponding to the volume data, in response to the input of the volume data; and displaying the height image of the sample surface on a display unit based on the height data.

In a fifth aspect of the present invention, a learning method is provided for training the learning model of the height image generation apparatus of the first aspect, which is executed by a computer, comprising the steps of: converting reference height data, which indicates the height of a reference sample surface in two-dimensional coordinates, into learning volume data corresponding to the reference height data; supplying the converted learning volume data to the height image generation apparatus; acquiring output height data output by the height image generation apparatus corresponding to the learning volume data; calculating a first loss function based on the acquired output height data and the reference height data; and training the learning model based on the calculation result of the first loss function, wherein the learning volume data includes a plurality of learning interference fringe images in which the brightness of at least some interference fringes is reduced corresponding to the shape of the reference sample surface indicated by the reference height data.

In a sixth aspect of the present invention, a program is provided that, when executed by a computer, causes the computer to function as the height image generation device of the first aspect.

In a seventh aspect of the present invention, a program is provided that, when executed by a computer, causes the computer to function as the learning device of the second aspect.

According to the present invention, a three-dimensional measuring machine using a white light interferometer can output a height image with reduced noise, even if the contrast of the interference fringe image is low due to the shape and material of the sample surface.

This example shows the configuration of the three-dimensional measurement system S according to this embodiment. This shows an example of the configuration of the height image generation device 100 according to this embodiment. An example of volume data acquired by the volume data acquisition unit 110 according to this embodiment is shown. This diagram illustrates the centering process performed by the centering processing unit 130 according to this embodiment. An example of a height image output by the height image generation device 100 according to this embodiment is shown. This shows an example of the configuration of the learning device 200 according to this embodiment. This is an example of a reference height image based on reference height data according to this embodiment. This shows a first example of the operation flow of the three-dimensional measurement system S according to this embodiment. A second example of the operation flow of the three-dimensional measurement system S according to this embodiment is shown.

<Example configuration of the three-dimensional measurement system S>
Figure 1 shows an example of the configuration of the three-dimensional measurement system S according to this embodiment. The three-dimensional measurement system S generates multiple interference fringe images using a white light interferometer, and outputs a height image showing the surface topography of a sample using a trained model (hereinafter sometimes referred to as the "trained model") based on the generated multiple interference fringe images. The three-dimensional measurement system S comprises a three-dimensional measuring machine 10, a height image generation device 100, and a learning device 200.

The three-dimensional measuring machine 10 generates an interference image of the sample using a white light interferometer. The white light interferometer irradiates the sample surface and the reference surface with white light, and interferes the reflected light from the sample surface with the reference light reflected from the reference surface to generate an interference fringe image based on the height-direction topography of the sample surface. The three-dimensional measuring machine 10, for example, moves the reference surface by a predetermined distance to vary the optical path difference between the reference surface and the sample surface by a predetermined value. Alternatively, the three-dimensional measuring machine 10 may move the objective lens by a predetermined distance to vary the optical path difference between the reference surface and the sample surface by a predetermined value.

Conventional three-dimensional measurement methods involved capturing interference fringe images each time the optical path difference was varied, and then observing and analyzing the different brightness levels of interference fringes appearing in the multiple captured images to estimate the three-dimensional shape of the sample surface. Such three-dimensional measurement using a white light interferometer is a known technique, as described in, for example, Patent Document 1, so a detailed explanation is omitted here. The three-dimensional measuring machine 10 according to this embodiment outputs multiple interference fringe images generated each time the optical path difference between the reference surface and the sample surface is varied by a predetermined value (Figure 1(1)). The height image generation device 100 then outputs a height image showing the three-dimensional shape of the sample surface in the height direction as the measurement result of the three-dimensional measurement system S (Figure 1(2)).

The height image generation device 100 acquires multiple interference fringe images as volume data. Then, using a trained model, the height image generation device 100 generates height data corresponding to the volume data and displays a height image of the sample surface based on the height data on the display unit. The height image generation device 100 is, for example, a computer such as a server. The height image generation device 100 is connected via a network to the three-dimensional measuring machine 10, a database, etc., and acquires volume data. Alternatively, the height image generation device 100 may be configured integrally with the three-dimensional measuring machine 10.

The learning device 200 trains a learning model for the height image generation device 100. Here, "training a learning model" means updating the learning parameters of the learning model so that the training data input by the learning device 200 to the learning model closely matches the data output by the learning model. It is desirable that the learning device 200 trains the learning model before the measurement of the sample surface by the three-dimensional measuring machine 10 and the height image generation device 100. The learning device 200 may be connected to the height image generation device 100 via a network, or it may be integrated with the height image generation device 100. Alternatively, the learning device 200 may be integrated with the three-dimensional measuring machine 10.

The learning device 200 learns a learning model using reference height data and learning volume data converted from the reference height data. The reference height data is data representing a shape with steep slopes and/or vertical walls. The learning volume data is data corresponding to multiple learning interference fringe images calculated and generated using the optical interference equation based on the reference height data (Figure 1(3)). The learning device 200 supplies the learning volume data to the height image generation device 100 (Figure 1(4)) and acquires the output height data output by the height image generation device 100 (Figure 1(5)). Then, the learning device 200 calculates learning parameters so that the acquired output height data approaches the reference height data, and provides the calculated learning parameters to the height image generation device 100 to update it (Figure 1(6)).

For example, the height image generation device 100 stores the pre-training model, and the learning device 200 updates the parameters of the height image generation device 100's training model to make it a trained model. Alternatively, the learning device 200 may provide the height image generation device 100 with a trained model by transmitting information about the trained model to the height image generation device 100.

The learning device 200 learns the learning model of the height image generation device 100 using, for example, training volume data with interference fringe contrast so low that conventional techniques could not reproduce height images. As a result, the three-dimensional measurement system S according to this embodiment can output height images with reduced noise even when measuring a sample surface with steep slopes and/or vertical walls. Note that if the three-dimensional measurement system S is a system intended for learning a learning model, the three-dimensional measuring machine 10 may be omitted. Also, if the three-dimensional measurement system S is a system intended for measuring the sample surface using a trained model, the learning device 200 may be omitted. Details of the height image generation device 100 and the learning device 200 will now be described.

<Example of the configuration of the height image generation device 100>
Figure 2 shows an example of the configuration of the height image generation device 100 according to this embodiment. The height image generation device 100 is a computer such as a CPU (Central Processing Unit). The height image generation device 100 includes a volume data acquisition unit 110, a storage unit 120, a centering processing unit 130, a learning model 140, a display control unit 150, and a display unit 160.

The volume data acquisition unit 110 acquires multiple interference fringe images output by the three-dimensional measuring machine 10 as volume data. The volume data acquisition unit 110 may acquire the volume data directly from the three-dimensional measuring machine 10, or it may acquire it via a network or the like. Alternatively, the volume data acquisition unit 110 may acquire the volume data from a database or the like that stores volume data. While the learning device 200 is creating the trained model 140, the volume data acquisition unit 110 acquires the volume data supplied by the learning device 200.

Figure 3 shows an example of volume data acquired by the volume data acquisition unit 110 according to this embodiment. Figure 3 shows five interference fringe images as volume data, each having data values indicating the brightness of the interference fringes for each pixel in the two-dimensional xy coordinate system. The volume data consists of multiple interference fringe images captured while continuously changing the optical path difference between the sample surface and the reference surface by scanning the position of the objective lens in the z-axis direction.

In the measurement of multiple interference fringe images by the three-dimensional measuring machine 10, interference fringes with maximum contrast are captured in the region where the optical path difference between the sample surface and the reference surface is zero. Focusing on the brightness of each point in the multiple interference fringe images, the data values aligned in the z direction at each pixel with the same x and y coordinates constitute an interference waveform. Figure 3 shows an example of an interference waveform formed by the data of five pixels aligned in the z direction at coordinates (x ₁ , y ₁ ). In this embodiment, the interference waveform at coordinates (x, y) is denoted as F(x, y, z _i ). Here, i is the image number, and z _i is the data value aligned in the z direction at each pixel with the x and y coordinates.

The position in the z-direction where the peak value of such interference waveform occurs corresponds to the height of the sample surface. For example, in the interference waveform F( _x1 , _y1 , _zi ) in Figure 3, the data value _z1 is the peak when i=1, indicating that the height of the sample surface at coordinate ( _x1 , _y1 ) corresponds to the height of image number 1. Conventional three-dimensional measurements determined the height of the sample surface based on the peak value of the interference waveform in this way, so in order to perform high-precision, high-resolution measurements, it was necessary to further subdivide the height direction (making the optical path difference between the reference plane and the sample surface more finely varied) to generate more sharp interference fringe images. In contrast, the three-dimensional measurement system S, by using a trained model, can measure the shape of the sample with higher accuracy than conventional three-dimensional measurements, even when the contrast of the interference fringe image is low or when noise is superimposed on the interference fringe image.

The storage unit 120 stores the volume data acquired by the volume data acquisition unit 110. The storage unit 120 may also store intermediate data, calculation results, thresholds, reference values, and parameters generated (or used) by the height image generation device 100 during its operation. Furthermore, the storage unit 120 may supply the stored data to the requesting party in response to requests from various parts of the height image generation device 100 or the learning device 200.

The storage unit 120 is a storage medium including ROM (Read Only Memory) and RAM (Random Access Memory), etc. The storage unit 120 may also include a large-capacity storage device such as an HDD (Hard Disk Drive) and/or an SSD (Solid State Drive). For example, if the computer functions as a height image generation device 100, it may store information such as the OS (Operating System) and programs that enable the computer to function. Furthermore, the storage unit 120 may store various information, including a database referenced during program execution.

The volume data acquisition unit 110, the centering processing unit 130, and the display control unit 150 are implemented, for example, by the CPU executing a program stored in the memory unit 120. The computer generates the height image using the learning model 140 by executing the program stored in the memory unit 120. In the example shown in Figure 2, the height image generation device 100 has the learning model 140, but the learning model 140 may be provided in an external device to the height image generation device 100. Furthermore, the learning model 140 may have a processor for operating the trained model.

The centering processing unit 130 performs centering on multiple interference fringe images contained in the volume data. The centering process involves calculating a moving average of the pixel data values for each identical two-dimensional coordinate of the multiple interference fringe images contained in the volume data, and subtracting the calculated moving average from the pixel data values used to calculate the moving average. For example, the interference waveform F _C (x, y, z _i ) of the volume data after centering is shown by the following equation.

Figure 4 is a diagram illustrating the centering process performed by the centering processing unit 130 according to this embodiment. Figure 4A shows an example of the interference waveform F(x, y, z _i ) of volume data. Figure 4B shows an example of the interference waveform F _C (x, y, z _i ) after centering processing. In Figures 4A and 4B, the horizontal axis represents the position i in the z direction, and the vertical axis represents the pixel data value z _i indicating brightness.

The interference waveform F(x,y, _zi ) in Figure 4A shows an example of a waveform where the intensity of the reflected light detected is low due to the shape of the surface, such as a steep slope or vertical wall, making the interference fringes difficult to see. When a centering process is applied to the interference waveform F(x, _y ,zi) in which the interference fringes are buried in background noise, the moving average value becomes 0, and the signal components of the interference fringes stand out, as shown in the interference waveform _FC (x,y, _zi ) in Figure 4B. The centering processing unit 130 then supplies the absolute value of the interference waveform _FC (x,y, _zi ), from which background noise has been reduced, to the learning model 140.

The learning model 140 is a program that outputs height data corresponding to volume data after the volume data processed by the centering processing unit 130 has been input. The height data represents the expected value of the height of the sample surface in two-dimensional coordinates. The learning model 140 includes a convolutional neural network (CNN) 141 and a height data calculation unit 142.

CNN141 outputs a probability distribution of the height of the sample surface in the two-dimensional coordinate system corresponding to the volume data, in response to the input volume data. CNN141 includes a network layer 143 and a conversion processing unit 144.

The network layer 143 calculates multiple output interference fringe images corresponding to multiple interference fringe images contained in the input volume data, using these as output volume data. The network layer 143 has multiple layers, and each layer performs processes such as activation function processing and convolution. One example of an activation function is a normalized linear function called a Rectified Linear Unit (ReLU).

Such a network layer 143 is a known learning model used in machine learning for image recognition, video recognition, etc., and a detailed explanation is omitted here. Note that training the learning model 140 is equivalent to optimizing the learning parameters of the network layer 143 to more appropriate values.

The conversion processing unit 144 converts the output volume data calculated by the network layer 143 into a probability distribution of the height of the sample surface in two-dimensional coordinates. The conversion processing unit 144 converts the output volume data into a probability distribution P _i of the height in the z direction in the x and y coordinates, for example, using the softmax function shown in the following equation. The probability distribution P _i indicates the probability that the interference fringes can take a peak value at position i in the z direction.

The height data calculation unit 142 calculates height data that represents the expected value of the height of the sample surface in two-dimensional coordinates, based on the probability distribution of the height of the sample surface. The height data calculation unit 142 calculates the dot product of the possible height values i of the sample surface and the probability distribution P _i , as shown in the following equation, and uses this as the expected value _He of the height in the z direction in the x and y coordinates. The height data calculation unit 142 outputs height data _He (x, y) with the expected value of the height calculated for each coordinate.

The display control unit 150 displays a height image of the sample surface on the display unit 160 based on the height data. For example, the display control unit 150 generates a height image showing the shape of the sample surface viewed from a predetermined angle. The display unit 160 is a display or the like. The display control unit 150 may store the height image in the storage unit 120, an external database, or the like.

The network layer 143 has learned its learning parameters so that noise generated in the height data calculated from the output volume data output by the network layer 143 is reduced. Therefore, the height image generation device 100 according to this embodiment can display a height image with reduced noise on the display unit 160.

Figure 5 shows an example of a height image output by the height image generation device 100 according to this embodiment. Figure 5A shows a height image based on height data generated by the conversion processing unit 144 and the height data calculation unit 142, without passing through the centering processing unit 130 and the network layer 143, from the volume data acquired by the volume data acquisition unit 110. In other words, Figure 5A shows an example of a comparison image output by conventional three-dimensional measurement. The comparison image shows that because the sample surface has a steep slope, a lot of spike-like noise is superimposed.

Figure 5B shows a height image based on height data generated via the centering processing unit 130 and the learning model 140 (CNN 141 and height data calculation unit 142) using the volume data used to generate the height image in Figure 5A. Figure 5B shows that, according to this embodiment, the height image generation device 100 can output the shape of a sample surface with reduced noise, even when using volume data measured from a sample surface with a steep slope.

As mentioned above, conventional three-dimensional measurement detected the height of the sample surface by detecting the peak value of the interference waveform from the interference fringe image. Therefore, higher resolution and accuracy were achieved by using more interference fringe images. It was also known that learning models were used to sharpen each interference fringe image. However, even if each interference fringe image was trained to sharpen independently, it did not optimize for appropriately reducing the noise generated after converting multiple interference fringe images into height data. Therefore, noise reduction in conventional three-dimensional measurement was difficult.

Furthermore, even if each interference fringe image can be made clearer, the number of interference fringe images required for high-precision measurement does not change. Therefore, conventional three-dimensional measurements often resulted in significant superposition of spike-like noise when the sample surface had steep slopes or vertical walls. Additionally, achieving high precision required measuring many more interference fringe images, leading to lengthy measurements.

In contrast, the height image generation device 100 according to this embodiment uses a learning model 140 that has been trained to reduce noise superimposed on the height data, outputting height data corresponding to the volume data as input data, rather than independently sharpening each of the multiple interference fringe images. In other words, the inventors have found that by combining multiple interference fringe images as volume data and having the learning model 140 process them, the noise in the height data, which is the objective of three-dimensional measurement, is directly reduced, rather than sharpening each of the multiple interference fringe images independently.

As a result, the trained model 140 does not operate to sharpen the interference fringes of a single interference fringe image, but rather to sharpen and smoothly connect multiple interference waveforms corresponding to multiple interference fringe images in order to reduce noise in the height data. The trained model 140 then calculates the peak values of the smoothly connected and sharpened interference waveforms, thereby improving the height resolution of the sample surface.

Therefore, the height image generation device 100 can improve the measurement resolution and accuracy of the three-dimensional measurement system S while reducing noise in the height data corresponding to the input volume data. For example, it has been found that the height image generation device 100 can achieve a similar level of measurement resolution and accuracy using volume data of interference fringe images with approximately 200 to 500 pixels, compared to conventional three-dimensional measurement using interference fringe images with a z-direction resolution of approximately 0.06 μm. The learning device 200 for training the learning model 140 of such a height image generation device 100 will be described next.

<Example of the configuration of the learning device 200>
Figure 6 shows an example of the configuration of the learning device 200 according to this embodiment. The learning device 200 is, for example, a computer. The learning device 200 may be the same computer as the height image generation device 100, or it may be a different computer from the height image generation device 100. The learning device 200 includes a reference height data acquisition unit 210, a storage unit 220, a learning volume data conversion unit 230, a volume data supply unit 240, a height data acquisition unit 250, a first loss function calculation unit 260, a volume data conversion unit 270, a second loss function calculation unit 280, and a learning unit 290.

The reference height data acquisition unit 210 acquires reference height data. The reference height data represents the height z = f(x, y) of the reference sample surface in two-dimensional coordinates. The reference height data is, for example, data created by the user of the height image generation device 100 as the surface irregularities of the sample surface to be measured by the three-dimensional measuring machine 10. The reference height data acquisition unit 210 may acquire the reference height data from a database or the like, or it may acquire the data by accepting data input from the user.

Figure 7 shows an example of a reference height image based on reference height data according to this embodiment. Figure 7 shows an example where a user creates reference height data as simulation data, and generates a height image of the shape of the created reference height data viewed from a predetermined angle. The reference height data is created so that the height image has shapes with various slopes and a hemispherical shape. In other words, the reference height data is data that has a shape that the three-dimensional measurement system S should correspond to as three-dimensional measurement.

The reference height data acquisition unit 210 may acquire the reference height data created by the user in this manner, or alternatively, it may acquire the user's formulas, algorithms, etc., for generating the reference height data and then acquire that data.

The storage unit 220 stores the reference height data acquired by the reference height data acquisition unit 210. The storage unit 220 may also store intermediate data, calculation results, thresholds, reference values, and parameters generated (or used) by the learning device 200 during its operation. Furthermore, the storage unit 220 may supply the stored data to the requesting party in response to requests from various parts of the learning device 200 or the height image generation device 100.

The storage unit 220 is a storage medium including ROM and RAM. The storage unit 220 may also include a large-capacity storage device such as an HDD and/or SSD. For example, if the computer functions as a learning device 200, it may store information such as the OS and programs that enable the computer to function. Furthermore, the storage unit 220 may store various types of information, including a database referenced during program execution.

For example, the computer functions as a reference height data acquisition unit 210, a storage unit 220, a learning volume data conversion unit 230, a volume data supply unit 240, a height data acquisition unit 250, a first loss function calculation unit 260, a volume data conversion unit 270, a second loss function calculation unit 280, and a learning unit 290 by executing a program stored in the storage unit 220.

Before describing the learning volume data conversion unit 230, the operation of the volume data conversion unit 270 will be explained. The volume data conversion unit 270 converts the reference height data acquired by the reference height data acquisition unit 210 into reference volume data that includes multiple reference interference fringe images and corresponds to the reference height data. The volume data conversion unit 270 calculates the reference volume data using, for example, the optical interference equation shown in the following equation. The optical interference equation is a known technique, as described in Non-Patent Documents 1, Non-Patent Documents 2, etc., so a detailed explanation will be omitted here. The reference volume data represents ideal volume data corresponding to the reference height data.

In response, the training volume data conversion unit 230 calculates training volume data such that the brightness of the interference fringe image decreases according to the magnitude of each slope in the reference height image. For example, the training volume data conversion unit 230 has a coefficient that multiplies the brightness of the interference fringe image by the square of the magnitude of the z component of the normalized normal vector of the height image. The training volume data conversion unit 230 may also add a term that takes background noise into account to the interference equation. The training volume data conversion unit 230 converts the reference height data to training volume data corresponding to the reference height data using, for example, the optical interference equation shown in the following equation.

In equation 5, N _z is the magnitude of the z component of the normal vector of the shape of the height image. By multiplying the second term on the right-hand side by the coefficient of the square of _{N z} , we can represent the phenomenon where the brightness of the interference fringes decreases as the slope of the incline increases, making the interference fringes less visible. Furthermore, the third term on the right-hand side of equation 5 represents the change in background intensity, and the fourth term on the right-hand side represents noise. Thus, the volume data for training will correspond to the shape of the reference sample surface indicated by the reference height data and will include multiple training interference fringe images in which the brightness of at least some of the interference fringes has been reduced compared to multiple reference interference fringe images.

The volume data supply unit 240 supplies the training volume data converted by the training volume data conversion unit 230 to the height image generation device 100. As a result, the training model 140 used by the height image generation device 100 outputs height data corresponding to the training volume data. Here, the height data output by the training model 140 corresponding to the training volume data is called the output height data.

The height data acquisition unit 250 acquires the output height data output by the learning model 140 of the height image generation device 100 in accordance with the training volume data. Ideally, the output height data acquired by the height data acquisition unit 250 matches the reference height data. However, since the training volume data has unclear interference fringes based on the various slopes and other shapes shown in the reference height data, spike noise and other interferences are superimposed on the output height data. Therefore, the learning device 200 learns the learning model 140 so that the output height data approaches the reference height data based on the comparison result between the output height data and the reference height data.

The first loss function calculation unit 260 calculates a first loss function based on the comparison result between the acquired output height data and the reference height data. The first loss function is a first function that calculates the similarity of the normal vectors of the height images, for example, as shown in the following equation.

Furthermore, the first loss function may be a second function that calculates a Harris corner detection response value based on the difference between the output height data and the reference height data, as shown in the following equation. In the following equation, k is a parameter that serves as a threshold for determining edges and corners, and for example, k is a value of about 0.05.
Here,

Furthermore, the first loss function may be a third function that calculates the L1 loss between the output height data and the reference height data. The first loss function calculation unit 260 calculates a first loss function that includes at least one of the first, second, and third functions. The first loss function calculation unit 260 may calculate all three functions. Note that the first, second, and third functions are examples of the first loss function, and the first loss function calculation unit 260 may calculate other functions as the first loss function.

The learning unit 290 trains the learning model 140 based on the calculation result of the first loss function. When using the first function, the learning unit 290 adjusts the learning parameters of the network layer 143 of the learning model 140 so that the value of the first function increases. For example, the learning unit 290 normalizes the value of the first function to 1 and adjusts the learning parameters of the network layer 143 so that the value obtained by subtracting the normalized value from 1 decreases.

The learning unit 290 adjusts the learning parameters of the network layer 143 so that the value of the second function is reduced when the second function is used. Similarly, the learning unit 290 adjusts the learning parameters of the network layer 143 so that the value of the third function is reduced when the third function is used. This allows the learning device 200 to learn the learning model 140 to reduce the noise superimposed on the height data output by the height image generation device 100. The learning unit 290 supplies the adjusted learning parameters to the height image generation device 100, updating the learning parameters of the learning model 140.

As a result, the learning device 200 can learn the learning model 140. Note that when the learning device 200 learns the learning model 140 using the first function, the second function, and the third function, the volume data conversion unit 270 may be omitted.

The learning device 200 according to this embodiment described an example in which the learning model 140 is trained using reference height data supplied to the height image generation device 100 and output height data output by the height image generation device 100. However, the learning device is not limited to this example. The learning device 200 may also train the learning model 140 using reference volume data converted from the reference height data and volume data converted from the output height data output by the height image generation device 100.

In this case, the volume data conversion unit 270 converts the reference height data to reference volume data using equation (Equation 4) as described above. The volume data conversion unit 270 also converts the output height data acquired by the height data acquisition unit 250 to the corresponding volume data. Here, the volume data converted by the volume data conversion unit 270 from the output height data is called the converted volume data. The volume data conversion unit 270 calculates the converted volume data using the optical interference equation shown in equation (Equation 4).

The second loss function calculation unit 280 then calculates a second loss function based on the comparison result between the converted volume data and the reference volume data. The second loss function includes a fourth function that calculates the L1 loss between the converted volume data and the reference volume data. Note that the fourth function is an example of the second loss function, and the second loss function calculation unit 280 may calculate other functions as the second loss function.

When the second loss function calculation unit 280 calculates the second loss function, the learning unit 290 learns the learning model 140 based on the calculation result of the second loss function. If the fourth function is used, the learning unit 290 adjusts the learning parameters of the network layer 143 so that the value of the fourth function becomes smaller. This allows the learning device 200 to learn the learning model 140 in a way that reduces the noise superimposed on the height data output by the height image generation device 100.

Furthermore, the learning unit 290 may learn the learning model 140 based on the calculation results of both the first and second loss functions. This allows the learning unit 290 to learn the learning model 140 such that, for example, the output height data approaches the reference height data, and the converted volume data approaches the reference volume data. The operation flow of the three-dimensional measurement system S equipped with the above learning device 200 will now be described.

<Example 1 of the operation flow of the three-dimensional measurement system S>
Figure 8 shows a first example of the operation flow of the three-dimensional measurement system S according to this embodiment. By executing the operation flow of the first example, the learning device 200 learns the learning model 140 of the height image generation device 100. In the operation flow of the first example, it is assumed that the user of the three-dimensional measurement system S has already created reference height data.

First, the reference height data acquisition unit 210 of the learning device 200 acquires reference height data (S51). Next, the learning volume data conversion unit 230 converts the reference height data acquired by the reference height data acquisition unit 210 into learning volume data (S52). Next, the volume data supply unit 240 supplies the learning volume data converted by the learning volume data conversion unit 230 to the height image generation device 100 (S53).

As a result, the volume data acquisition unit 110 of the height image generation device 100 acquires the training volume data supplied from the volume data supply unit 240. Next, the centering processing unit 130 performs a centering process on the training volume data acquired by the volume data acquisition unit 110. The learning device 200 may also perform the centering process on the training volume data, or alternatively, the training volume data conversion unit 230 may generate training volume data with a moving average of 0, similar to the centered data. In this case, the height image generation device 100 may omit the centering process.

Next, the learning model 140 calculates output height data corresponding to the training volume data after centering (S54). The height image generation device 100 supplies the output height data output by the learning model 140 to the learning device 200 (S55). As a result, the height data acquisition unit 250 of the learning device 200 acquires the output height data output by the height image generation device 100 corresponding to the reference volume data.

Next, the first loss function calculation unit 260 calculates a first loss function based on the comparison result between the output height data acquired by the height data acquisition unit 250 and the reference height data (S56). Next, the volume data conversion unit 270 converts the reference height data acquired by the reference height data acquisition unit 210 into corresponding reference volume data. The volume data conversion unit 270 also converts the output height data acquired by the height data acquisition unit 250 into corresponding converted volume data (S57). Finally, the second loss function calculation unit 280 calculates a second loss function based on the comparison result between the converted volume data converted by the volume data conversion unit 270 and the reference volume data (S58).

Next, the learning unit 290 adjusts and updates the learning parameters of the learning model 140 based on the calculation results of the first and second loss functions (S59). The learning device 200 repeats the operations from S53 to S59 until the learning of the learning unit 290 is completed (S60: No). For example, the learning unit 290 terminates learning by repeating the update of the learning parameters a predetermined number of times (S60: Yes). Alternatively, the learning unit 290 may terminate learning when one or more of the calculation results of the first, second, third, and fourth functions fall within a predetermined range.

Through the above operation flow, the learning device 200 can learn the learning model 140 of the height image generation device 100 based on the reference height data. If the user creates multiple reference height data sets with different values, the learning device 200 may repeat the learning process using these multiple reference height data sets. Furthermore, the learning device 200 may learn the learning model 140 for each set of reference height data sets to prepare multiple combinations of learning parameters.

Alternatively, or in addition to the above, the learning device 200 may generate multiple different training volume data from a single reference height data. For example, the training volume data conversion unit 230 generates multiple different training volume data from a single reference height data by changing the coefficients, parameters, etc., of the optical interference equation (equation 5). The learning device 200 may then repeat the training using the generated training volume data. Furthermore, the learning device 200 may train a training model 140 for each training volume data to prepare multiple combinations of training parameters.

<Example 2 of the operation flow of the three-dimensional measurement system S>
Figure 9 shows a second example of the operation flow of the three-dimensional measurement system S according to this embodiment. By executing the operation flow of the second example, the three-dimensional measurement system S has the height image generation device 100 output height images corresponding to the multiple interference fringe images measured by the three-dimensional measuring machine 10. In the operation flow of the second example, it is assumed that the learning model 140 of the height image generation device 100 has already been trained.

First, the three-dimensional measuring machine 10 measures the sample surface (S71) and outputs multiple interference fringe images (S72). Based on this, the volume data acquisition unit 110 of the height image generation device 100 acquires multiple interference fringe images from the three-dimensional measuring machine 10 as volume data. Next, the centering processing unit 130 performs a centering process on the volume data acquired by the volume data acquisition unit 110 (S73).

Next, the learning model 140 outputs height data corresponding to the volume data after the centering processing unit 130 has performed the centering process (S74). Then, the display control unit 150 generates a height image of the sample surface based on the height data and displays it on the display unit 160 (S75). As a result, the height image generation device 100 can display a height image with reduced noise on the display unit 160.

Furthermore, the height image generation device 100 may display the height image on the display unit 160 by switching the learning parameters. In this case, the storage unit 120 stores multiple learning parameters obtained by learning the learning model using multiple different reference height data. The height image generation device 100 also includes a receiving unit that accepts a request to display a second height image, different from the first height image, on the display unit 160 using parameters different from the first learning parameters, after the first height image of the sample surface corresponding to one volume data has been displayed on the display unit 160 using the first learning parameters.

Here, in S74, the learning model 140 uses the first learning parameter, and in S75, the display control unit 150 displays the first height image on the display unit 160. Then, in response to the reception unit receiving confirmation from the user of the three-dimensional measurement system S that the second height image should be displayed on the display unit 160 (S76: Yes), the height image generation device 100 returns to S74, and the learning model 140 switches the first learning parameter to a second learning parameter that is different from the first learning parameter.

Next, the learning model 140 uses the second learning parameter to output height data corresponding to the volume data (first volume data) after centering. This allows the display control unit 150 to generate a second height image of the sample surface corresponding to the first volume data based on the height data and display it on the display unit 160 (S75).

The height image generation device 100 may repeatedly switch the learning parameters in response to user input. This allows the device to select more appropriate learning parameters for multiple interference fringe images measured by the three-dimensional measuring machine 10, thereby displaying a height image with reduced noise on the display unit 160. The height image generation device 100 terminates processing in response to user input (S77: No). Through this operation flow, the height image generation device 100 can display a height image with reduced noise corresponding to multiple interference fringe images measured by the three-dimensional measuring machine 10 on the display unit 160.

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of its gist. For example, all or part of the device can be configured by functionally or physically distributing and integrating them in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combinations are combined with the effects of the original embodiments.

10 Three-dimensional measuring machine 100 Height image generation device 110 Volume data acquisition unit 120 Storage unit 130 Centering processing unit 140 Learning model 141 CNN
142 Height data calculation unit 143 Network layer 144 Conversion processing unit 150 Display control unit 160 Display unit 200 Learning device 210 Reference height data acquisition unit 220 Storage unit 230 Learning volume data conversion unit 240 Volume data supply unit 250 Height data acquisition unit 260 First loss function calculation unit 270 Volume data conversion unit 280 Second loss function calculation unit 290 Learning unit

Claims (7)

A learning device for training a learning model of a height image generation device,
The height image generation device is
A three-dimensional measuring machine using a white light interferometer that irradiates a sample surface and a reference surface with white light, and interferes the reflected light from the sample surface with the reference light reflected from the reference surface to generate an interference fringe image based on the height-direction irregularities of the sample surface, comprising a volume data acquisition unit that acquires a plurality of the interference fringe images generated each time the optical path difference between the reference surface and the sample surface is changed by a predetermined value, as volume data,
A learning model that, in response to the input volume data, indicates the expected value of the height of the sample surface in two-dimensional coordinates and outputs height data corresponding to the volume data,
A display control unit that displays the height image of the sample surface on the display unit based on the height data,
Equipped with,
The learning device is
A learning volume data conversion unit converts reference height data, which indicates the height of a reference sample surface in two-dimensional coordinates, into learning volume data corresponding to the reference height data,
A volume data supply unit that supplies the converted volume data for learning to the height image generation device,
A height data acquisition unit that acquires output height data output by the height image generation device in accordance with the learning volume data,
A first loss function calculation unit calculates a first loss function based on the comparison result between the acquired output height data and the reference height data,
The system comprises a learning unit that learns the learning model based on the calculation result of the first loss function,
The training volume data includes a plurality of training interference fringe images in which the brightness of at least some of the interference fringes is reduced in accordance with the shape of the reference sample surface indicated by the reference height data,
Learning device. The volume data conversion unit further comprises a unit that converts the aforementioned reference height data into reference volume data that includes multiple reference interference fringe images and corresponds to the aforementioned reference height data.
The volume data conversion unit further converts the output height data acquired by the height data acquisition unit into corresponding converted volume data.
The learning device further includes a second loss function calculation unit that calculates a second loss function based on the comparison result between the converted volume data and the reference volume data.
The learning unit learns the learning model based on the calculation results of the first loss function and the second loss function.
The learning device according to claim 1 . The first loss function includes at least one of the following: a first function for calculating the similarity of the normal vectors of the height image; a second function for calculating a Harris Corner detection response value based on the difference between the output height data and the reference height data; and a third function for calculating the L1 loss between the output height data and the reference height data.
The second loss function includes a fourth function for calculating the L1 loss between the converted volume data and the reference volume data.
The learning device according to claim 2 . A three-dimensional measuring machine using a white light interferometer that irradiates the sample surface and the reference surface with white light, interferes the reflected light from the sample surface with the reference light reflected from the reference surface, and generates an interference fringe image based on the height-direction irregularities of the sample surface, wherein the three-dimensional measuring machine outputs a plurality of interference fringe images generated each time the optical path difference between the reference surface and the sample surface is changed by a predetermined value,
A height image generation device that acquires multiple interference fringe images output as volume data and displays a height image of the sample surface corresponding to the volume data on a display unit ,
A learning device according to any one of claims 1 to 3 for training the learning model of the height image generation device,
A three-dimensional measurement system equipped with the following features. The height image generation device is
A storage unit that stores multiple learning parameters obtained by training the learning model using multiple different reference height data,
The system further includes a receiving unit that, after displaying a first height image of the sample surface corresponding to a volume data using a first learning parameter on the display unit, receives a request to display a second height image different from the first height image on the display unit using a parameter different from the first learning parameter,
In response to the reception unit receiving a request to display the second height image on the display unit, the learning model outputs the height data corresponding to the first volume data using a second learning parameter different from the first learning parameter.
The display control unit generates a second height image of the sample surface corresponding to the first volume data based on the height data and displays it on the display unit.
The three-dimensional measurement system according to claim 4 . A computer executes
A three-dimensional measuring machine using a white light interferometer that irradiates a sample surface and a reference surface with white light, and interferes the reflected light from the sample surface with the reference light reflected from the reference surface to generate an interference fringe image based on the height-direction irregularities of the sample surface, comprising a volume data acquisition unit that acquires a plurality of the interference fringe images generated each time the optical path difference between the reference surface and the sample surface is changed by a predetermined value, as volume data,
A learning model that, in response to the input volume data, indicates the expected value of the height of the sample surface in two-dimensional coordinates and outputs height data corresponding to the volume data,
A display control unit that displays the height image of the sample surface on the display unit based on the height data,
A learning method for training the learning model of a height image generation device comprising the above,
A step of converting reference height data, which indicates the height of a reference sample surface in two-dimensional coordinates, into training volume data corresponding to the reference height data,
The steps include supplying the converted volume data for learning to the height image generation device,
The steps include: acquiring output height data output by the height image generation device in correspondence with the learning volume data;
A step of calculating a first loss function based on the acquired output height data and the reference height data,
The process includes the step of training the learning model based on the calculation result of the first loss function,
The training volume data includes a plurality of training interference fringe images in which the brightness of at least some of the interference fringes is reduced in accordance with the shape of the reference sample surface indicated by the reference height data,
Learning methods. A program that, when executed by a computer, causes the computer to function as the learning device described in any one of claims 1 to 3 .