JP7347750B2 - Verification device, learning device, method, and program - Google Patents

Description

The present invention relates to a verification device, a learning device, a method, and a program, and particularly relates to a verification device, a learning device, a method, and a program for verifying time-series data.

In recent years, demand for biometric authentication technology has been increasing in fields such as security, e-government, medical care, education, banking, and insurance to prevent third parties other than the person from being authenticated. As a supplement or substitute for conventional biometric authentication methods that rely on voiceprints and fingerprints, it is less susceptible to uncertain factors such as physical condition, does not place a large psychological burden on the person being authenticated, and is relatively simple for the person being authenticated. There are high expectations for personal authentication technology that uses handwritten signatures, which is a method that can perform personal authentication at low cost. Widely used signature authentication technology uses a signature collection device that inputs signatures using an electronic pen on the display of a tablet device, for example, and uses information such as the position information (coordinates) of the tip of the electronic pen and the brush stroke collected from the device. Multivariate time series data such as pressure information is handled as signature information. At the time of authentication, the input signature information (signature to be authenticated) is compared with one or more pieces of signature information (template signature) registered in advance in the database, and an authentication result is output. A particularly important issue in this technology is a method of collating signature information, that is, a method of measuring the distance between the signature to be authenticated and the template signature.

Non-Patent Document 1 describes a typical signature information matching method based on global features. In this method, the signature information includes temporal characteristics such as the duration of the handwriting and pen up/down times, velocity/acceleration characteristics calculated from the first and second time derivatives of the coordinates of the handwriting, and the direction of the handwriting. It is described by one multidimensional feature vector that includes directional features such as a histogram and geometric features extracted from the entire handwriting and each stroke. The distance between the signature to be authenticated and the template signature is defined by the Mahalanobis distance between the respective feature vectors. However, as shown in Figure 9, for many global features, signature time information disappears when features are extracted manually or by NN, so when measuring the distance between feature vectors, signature time information is lost. It is difficult to impose consistency (temporal consistency) as a constraint. As a result, when a signature of another person (other person's signature) or a signature maliciously imitated by another person (clone signature) is input as a signature to be authenticated, the probability that it will be mistakenly determined to match the principal's signature registered as a template signature is determined. (other person acceptance rate) is high.

For example, consider two pieces of time series data shown in FIG. It is assumed that these time-series data are signature information of different people, that is, they should not be determined to be a match. However, as shown in FIG. 10, these time series data have many locally similar parts. If temporal consistency is not imposed as a constraint such that the times of the fragments must be the same in the correspondence between locally similar fragments (the broken lines in FIG. 10), the signature information may be mistakenly identified as a match. High probability of judgment.

On the other hand, the same non-patent document 1 also describes a typical signature information matching method based on local features. In this method, at each time of the signature information, local fragments surrounding the signature information include the values of multiple discrete time functions calculated from handwriting, pen pressure, etc., and the values of their first and second time derivatives. is described by one multidimensional feature vector containing . Therefore, the entire signature information is described by a sequence of feature vectors made up of a plurality of feature vectors. The distance between the signature to be authenticated and the template signature is measured by applying Dynamic Time Warping (DTW) or a hidden Markov model that can deal with variations in time and speed to the sequence of these feature vectors. Since such a sequence of local feature vectors includes time information of the signature, by using a distance measurement method that imposes a temporal consistency constraint such as DTW, it is possible to prevent other person's signatures or clone signatures from being input. When this happens, it is possible to keep the acceptance rate lower. However, since such local features are manually designed, they are not necessarily optimal for the signature authentication purpose at hand. Performance is inadequate, especially when more complex feature representations are required.

Non-Patent Document 2 is a typical signature authentication method in which signature features are learned directly from data rather than being designed manually. In this method, signature information is described by one multidimensional feature vector extracted by a Recurrent Neural Network (RNN). The distance between the signature to be authenticated and the template signature is defined by the inner product between the respective feature vectors. The parameters of the RNN are determined by learning using a large amount of training data. In this method, when signature information is input to the RNN, one sequence including a plurality of multidimensional feature vectors is calculated in the last hidden layer of the RNN. Each feature vector of the sequence corresponds to each time of the input signature. Then, 1) the feature vector at the last time of the sequence to 2) the average feature vector of the entire sequence is calculated as the feature vector of the input signature. No matter which of the above methods is used, the time information of the signature is lost during this process, so as with the global features, temporal consistency is imposed as a constraint when measuring the distance between feature vectors. It is difficult to do so. As a result, when a signature of another person or a clone signature is input as a signature to be authenticated, the acceptance rate of the other person is high.

Marcos Martinez-Diaz, Julian Fierrez, Ram P. Krish, and Javier Galbally. Mobile signature verification: Feature robustness and performance comparison. IET Biometrics, Vol. 3, No. 4, pp. 267-277, 2014 Wenjie Pei. Models for supervised learning in sequence data. PhD thesis, Delft University of Technology, Netherlands, 2018.

As mentioned above, in the prior art of the signature information matching method based on global features or neural networks, the time information of the signature disappears with feature extraction, so it is difficult to maintain temporal consistency when measuring the distance between feature vectors. Difficult to impose as a constraint. As a result, when a signature of another person or a clone signature is input as a signature to be authenticated, the acceptance rate of the other person is high.

On the other hand, in the prior art methods of matching signature information based on local features, the features of the signature information are manually designed, and therefore are not necessarily optimal for the purpose of signature authentication at hand. Performance is inadequate, especially when more complex feature representations are required.

In the prior art, there is still no signature authentication method that can solve the above problems at the same time.

The present invention has been made to solve the above problems, and includes a matching device, a learning device, and a method that can accurately determine whether two pieces of time-series data are inputs related to the same object. and programs.

In order to achieve the above object, a verification device according to a first aspect includes an input section that receives first signature information and second signature information that are time-series data of input positions by input means, and For each of the signature information and the second signature information, learning is performed in advance to associate local feature vectors for pairs of signature information, each of which is assigned in advance whether the signature is related to the same person, at all corresponding times. a local feature calculation unit that extracts the local feature vector at each time using the extraction model that extracts the local feature vector extracted for the first signature information; The distance between the local feature vector at each time and the local feature vector at each time extracted for the second signature information is calculated, and the distance between the first signature information and the second signature information is calculated. The information processing apparatus includes a verification unit that determines whether the signature information is a signature related to the same person.

A collation device according to a second aspect includes an input unit that receives first time series data and second time series data according to input by an input means, and a second time series data input unit that receives first time series data and second time series data. For each piece of data, the local feature is learned in advance so as to associate a local feature vector for a pair of time series data, which is given in advance whether the input is related to the same object, at every corresponding time. a local feature calculation unit that extracts a local feature vector at each time using an extraction model that extracts a vector; a local feature calculation unit that extracts a local feature vector at each time extracted for the first time series data; Based on the distance from the local feature vector at each time extracted for the second time series data, it is determined whether the first time series data and the second time series data are inputs related to the same object. It is configured to include a verification unit that makes a determination.

The learning device according to the third aspect calculates a local feature vector based on a pair of time-series data corresponding to an input by an input means, to which a training label indicating whether the input is related to the same object is given in advance. A learning device for learning an extraction model to be extracted, the learning device comprising: a local feature calculation unit that extracts a local feature vector at each time using the extraction model for each of the pair of time series data; For a local feature vector at each time extracted from one of the time series data of a pair of series data, and a local feature vector at each time extracted from the other time series data, the local feature vector at each time is Calculate the distance of the feature vector, and optimize the value of the loss function expressed using the distance of the local feature vector at each time and the training label given to the pair of time series data, and a learning section that learns the extraction model.

In the verification method according to the fourth aspect, the input unit receives first signature information and second signature information that are time-series data of input positions by the input means, and the local feature calculation unit receives For each of the signature information and the second signature information, learning is performed in advance to associate local feature vectors for pairs of signature information, each of which is assigned in advance whether the signature is related to the same person, at all corresponding times. The matching unit extracts the local feature vector at each time using the extracted model for extracting the local feature vector, and the matching unit extracts the local feature vector at each time extracted for the first signature information. The distance between the vector and the local feature vector at each time extracted for the second signature information is calculated, and the distance between the local feature vector at each time is calculated. It is determined whether the signatures are related to the same person.

The program according to the fifth aspect is configured to receive, in a computer, first signature information and second signature information that are time-series data of input positions by an input means, and to receive first signature information and second signature information that are time-series data of input positions by an input means, For each of the above, the local feature vectors are learned in advance so as to associate the local feature vectors for each pair of signature information to which it is previously assigned whether or not they are signatures related to the same person at all corresponding times. The local feature vector is extracted at each time using the extraction model to be extracted, and the local feature vector at each time extracted for the first signature information and the local feature vector extracted for the second signature information are extracted at each time. The distance between the local feature vector at each time and the local feature vector at each time is calculated, and it is determined whether the first signature information and the second signature information are signatures related to the same person. It is a program that allows you to do something.

According to the matching device, learning device, method, and program according to one aspect of the present invention, it is possible to accurately determine whether two pieces of time-series data are inputs related to the same object. It will be done.

FIG. 6 is a diagram for explaining a method of evaluating differences between local fragments at the same time in the order of time information of time series data. FIG. 6 is a diagram for explaining a method of evaluating differences between local fragments at the same time in the order of time information of time series data. FIG. 2 is a diagram illustrating an overview of a process for determining a distance between two time series data. FIG. 1 is a block diagram showing the configuration of a verification device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a specific configuration of a temporal matching section, a local feature extraction section, and a local distance calculation section of the matching device according to the embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of a learning device according to an embodiment of the present invention. FIG. 2 is a schematic block diagram of an example of a computer that functions as a verification device and a learning device. 3 is a flowchart showing a learning processing routine in a learning device according to an embodiment of the present invention. 3 is a flowchart showing a verification processing routine in a verification device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an overview of a process for determining the distance between two time series data in a conventional technique. FIG. 7 is a diagram showing the correspondence between locally similar fragments in two pieces of time series data.

Embodiments of the present invention will be described in detail below with reference to the drawings.

<Overview of embodiments of the present invention>
In an embodiment of the present invention, a signature matching method that can simultaneously solve the above problems is proposed. In other words, local features of signature information are directly learned from data, and temporal consistency is achieved when measuring distances between signature information. We propose a matching method that can be imposed as a constraint.

In order to impose temporal consistency as a constraint when measuring the distance between signature information, it is necessary to use signature information of different people that have many locally similar parts, as shown in Fig. 10. For time-series data, as shown in Figures 1A and 1B, the correspondence of local fragments is identified according to the order of time information of the time-series data, and differences between local fragments at the same time are evaluated. There is a need to. This reduces the probability that the signature information will be mistakenly determined to be a match. Note that the dashed line and the dashed-dotted line indicate the correspondence between local fragments at the same time, with the dashed line indicating that they are similar, and the dashed-dotted line indicating that they are not similar.

To do this, it is necessary to extract a local feature, that is, a local multidimensional feature vector, from a local fragment that includes the signature information or its surroundings at each time of the signature information. In embodiments of the present invention, in order to learn this local feature directly from data, a one-dimensional convolutional neural network using convolution along the time axis or a bidirectional Long Short-Term Memory (LSTM) is used. The local features are extracted using an RNN such as a GRU or a bidirectional Gated Recurrent Unit (GRU). The parameters of such a neural network are determined by learning using a large amount of training data.

In the embodiment of the present invention, a sequence (horizontal direction) of multidimensional feature vectors (vertical direction) as shown in the upper right corner of FIG. 2 is extracted from the signature information as a local feature sequence. Each feature vector of the sequence corresponds to each time of the input signature. Then, for each time in the two local feature sequences, the distance (eg, Euclidean distance) between two feature vectors at the same time is calculated as the local distance. As a result, a sequence of local distances between signature information is calculated.

In Figure 2, the local distance at each time is one real value, but depending on the method of calculating the local distance, it may become a distance vector with one or more real values. . In that case, the local distance sequence becomes a multidimensional sequence of distance vectors.

In the step of performing signature authentication (authentication step), all the distance values included in the local distance sequence are integrated to calculate one distance value, which is used as the distance between signature information. As a result, as shown in FIG. 10, FIGS. 1A and 1B show signature information of different people in which many correspondences of local fragments that are not temporally consistent appear. As such, local differences between signature information (see broken lines in FIGS. 1A and 1B) can be identified without exception. Furthermore, by integrating all the local distances, it becomes possible to more accurately calculate the distance value of the entire signature information, emphasizing the local differences between the signature information. Note that, in this embodiment, one distance is calculated between signatures, but the distance for each corresponding time may be used as is. That is, the distance at each corresponding time may be used as time-series distance data, and the time-series distance data itself may be used as the distance. In short, it is sufficient to use the time-series distance data to finally output whether the signature of the verification source and the signature of the verification destination match. For example, the time-series distance data may be binarized based on a predetermined threshold value for each time. In other words, it is only necessary to define the distance between signatures based on the distance at which time series information is retained.

In the step of learning the parameters of the neural network (training step), the parameters are learned by minimizing the value of a loss function calculated from the local distance sequence. This loss function must have the property that by minimizing the function value, the local distance between the signature information of the same person becomes smaller, and the local distance between the signature information of different people becomes larger. To this end, in the present invention, a loss function is developed in which, in the case of signature information of the same person, the larger the local distance, the larger the loss function value, and in other cases, the smaller the local distance, the larger the loss function value. Define and use it for learning. This allows the trained neural network to accurately measure not only the differences in the entire signature information, but also the differences in local fragments at the same time when following the time order of the time series data. .

Furthermore, even in signature information of the same person, nonlinear temporal fluctuations such as local changes and changes in speed often occur. In both the authentication stage and the training stage, it is necessary to correctly specify the correspondence of local fragments in accordance with the order of time information of time-series data, so it is necessary to suppress the influence of the nonlinear temporal fluctuations. To this end, in an embodiment of the present invention, a temporal alignment unit is added before the neural network to compensate for the nonlinear temporal variation and align the two pieces of signature information along the time axis.

<Configuration of verification device according to embodiment of the present invention>
Next, the configuration of a verification device according to an embodiment of the present invention will be described. As shown in FIG. 3, the verification device 100 according to the embodiment of the present invention is a computer including a CPU, a RAM, and a ROM that stores programs and various data for executing a verification processing routine to be described later. It can be configured. Functionally, the verification device 100 includes an input section 10, an arithmetic section 20, and an output section 50, as shown in FIG.

The input unit 10 receives first signature information, which is time-series data of input positions by input means such as a pen tablet.

The input unit 10 also receives input of second signature information prepared in advance, which is time-series data of the input position by the input means and in which the person who signed the signature is known. For example, input of one or more pieces of second signature information is accepted as a template signature.

The calculation unit 20 includes a temporal matching unit 22, a local feature extraction unit 24, and a matching unit 26.

The matching unit 26 includes a local distance calculation unit 30, a local distance integration unit 32, and a determination unit 34.

<Temporal matching part>
The temporal matching unit 22 matches the first signature information, which is time-series data, and the second signature information, which is time-series data, along the time axis, and generates first matched signature information. and second matched signature information.

Specifically, the temporal alignment unit 22 compensates for nonlinear temporal fluctuations between the two pieces of signature information and aligns the two pieces of signature information along the time axis.

For example, a local feature sequence used to match two pieces of signature information (expressed as Signature in the example of FIG. 4) along the time axis is extracted. Specifically, using the method of Non-Patent Document 1, at each time of each signature information, one 7-dimensional feature vector is extracted from a local fragment surrounding the time, using handwriting coordinates. . As a result, for each signature information, one local feature sequence is calculated that includes feature vectors extracted from all times. Furthermore, as another embodiment, each piece of signature information can be used as is as one local feature sequence.

Next, in each dimension of the local feature sequence, all elements in that dimension are normalized so that the mean of all elements in that dimension is 0 and the variance is 1 (expressed as Channel-Wise Standardization in the example of Figure 4). ).

Next, dynamic time warping (DTW) is applied to the two normalized local feature sequences calculated from the two pieces of signature information, and the time correspondence between the two pieces of signature information is determined. One warping path (Warping Path) representing . Further, instead of DTW, it is possible to use any matching method that can compensate for nonlinear temporal fluctuations between two pieces of signature information, such as a method using Edit Distance.

The warping pass is used to align the two normalized local feature sequences along the time axis. The two aligned normalized local feature sequences are output as two aligned signature information. Furthermore, as another embodiment, it is also possible to match two pieces of signature information along the time axis using the warping pass. In that case, the two pieces of matched signature information will be output as two pieces of matched signature information.

Note that in matching the signature information by the temporal matching unit 22, the time correspondence between the two pieces of signature information may only be calculated without changing the two pieces of signature information.

<Local feature extraction unit>
The local feature extraction unit 24 inputs the first matched signature information and the second matched signature information, and extracts information about the pair of signature information that has been assigned in advance whether the signature information is related to the same person or not. Using an extraction model for extracting local feature vectors that has been trained in advance to associate local feature vectors at each time, a local feature vector is extracted at each time, and two local feature sequences are output. do.

Specifically, for each of the first matched signature information and the second matched signature information, at each time, the local feature extraction unit 24 extracts a local feature that includes the relevant time or the vicinity of the relevant time. One feature vector is extracted from the target fragment, and one local feature sequence containing the feature vectors extracted from all times is output.

The local feature sequence referred to here is different from the local feature sequence used by the temporal matching unit 22 to match two pieces of signature information along the time axis.

For example, the following processing is performed on each of the first matched signature information and the second matched signature information using a convolutional neural network.

First, the size of each of the first matched signature information and the second matched signature information is changed to a predetermined constant length (1024 in this embodiment) (expressed as Resize in the example of FIG. 4). . This corresponds to the process of changing the size of the matched signature information to a predetermined constant length in order to use batch learning or mini-batch learning to learn the parameters of the convolutional neural network in the learning device 200, which will be described later. It is something that is done.

Next, the normalization performed by the temporal matching unit 22 is performed again on the matched signature information whose size has been changed (expressed as Channel-Wise Standardization in the example of FIG. 4). Here, the normalized and resized matched signature information is considered to be a 1×1024×7 tensor. 1024 is the temporal length, and 7 is the number of dimensions. A convolutional neural network that receives this tensor as input includes one 1×7 convolution layer, one Max Pooling layer, and two 1×3 convolution layers. The number of dimensions of the outputs of all convolutional layers (expressed as Conv in the example of FIG. 4) is set to 64. For the 1×7 convolutional layer and Max Pooling layer, downsampling is performed so that the temporal length of each output is half of the temporal length of each input by setting Stride to 2. (expressed as "/2" in the example of FIG. 4). Further, a Batch Normalization layer (Reference Document 1) is used immediately after each convolution layer, and then a ReLU layer is used as an activation function. Following the last ReLU layer, at each time of its output, all elements at that location are normalized so that the L2 norm of all elements at that location is 1 (in the example of Figure 4, Location-Wise L2 Norm and Expression). The normalization completed at all times is output as a local feature sequence. Since the downsampling was performed twice in total, the temporal length of the local feature sequence is 256. The number of dimensions is 64.

In addition, as another example, it is sufficient that the neural network from the 1×7 convolutional layer to the final ReLU layer can be interpreted as a sequence of local feature vectors in which the set of values of the hidden unit that is the final output is interpreted as a sequence of local feature vectors. It is possible to replace it with any neural network that has a temporal length of . A typical example of a neural network that satisfies this condition is a convolutional neural network, but there are also recurrent neural networks (RNNs), bidirectional RNNs, and the like. For example, using Long Short-Term Memory (LSTM), the first layer of LSTM starts from the first time of the (normalized and resized) matched signature information and moves forward in time. However, the second layer LSTM can be a bidirectional LSTM that starts from the last time of the (normalized and resized) matched signature information and moves backward along the time axis. be. Furthermore, it is also possible to use a bidirectional GRU in which all the LSTMs are replaced with gated recurrent units (GRU).

[Reference 1] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pp. 448-456, 2015.

<Local distance calculation section>
The local distance calculation unit 30 calculates the distance between the local feature vector at each time extracted for the first matched signature information and the local feature vector at each time extracted for the second matched signature information. Calculate local distance at each time.

Specifically, the input two pieces of signature information are

and the distance between the output signature information is

shall be.

The two local feature sequences that are the outputs of the local feature extractor 24 and are calculated from . Let X and Y be W×K real matrices, that is,

shall be. Here, W is the temporal length and K is the number of dimensions.

For example, in the example of FIG. 4, W = 256 and K = 64. In X and Y, the feature vectors at the i-th time are respectively

shall be. Furthermore, in X and Y, the elements at the i-th time and j-th dimension are respectively x _ij and y _ij .

The local distance calculation unit 30 inputs two local feature sequences that are output from the local feature extraction unit 24, and outputs one local distance sequence. The local distance calculation unit 30 calculates the distance between two feature vectors at the same time at each time of the two local feature sequences, and calculates one feature vector that includes the distances calculated from all times. Outputs a local distance sequence. In this embodiment, the local distance sequence is calculated by calculating the Euclidean distance between the two feature vectors. Here, the above two feature vectors are

and the distance between two feature vectors is

shall be.

is calculated using equation (1).

(1)

The local distance integration unit 32 receives one local distance sequence as input and outputs the distance between one piece of signature information.

Specifically, the local distance integration unit 32 integrates all the distances included in the local distance sequence and calculates the distance between pieces of signature information. In this embodiment, the average of all distances included in the local distance sequence is calculated and used as the distance between the signature information. Here, the distance between the signature information is

Calculated using equation (2).

(2)

<Judgment section>
The determination unit 34 determines whether the first signature information and the second signature information are signature information regarding the same person based on the distance between the signature information.

Specifically, for each of the second signature information input as one or more template signatures, the distance between the first signature information input as the signature to be authenticated and the second signature information is determined as a local distance. The integrating unit 32 calculates the average of the distances calculated for all the second signature information, and calculates one score. The score is compared with a predetermined threshold, and if the score is smaller than the threshold, it is determined that the signature to be authenticated is signature information of the same person as the template signature; otherwise, it is determined to be signature information of a different person. Here, enter the signature to be authenticated.

and a set of one or more template signatures is

shall be. The average distance can be calculated using equation (3) or equation (4), for example.

(3)

(Four)

<Configuration of learning device according to embodiment of the present invention>
Next, the configuration of a learning device according to an embodiment of the present invention will be described. As shown in FIG. 5, the learning device 200 according to the embodiment of the present invention is a computer that includes a CPU, a RAM, and a ROM that stores programs and various data for executing a learning processing routine that will be described later. It can be configured. Functionally, the learning device 200 includes an input section 110 and a calculation section 120, as shown in FIG.

The input unit 110 receives an input of a pair of signature information, which is time-series data of an input position by an input means such as a pen tablet, to which a training label indicating whether the signature information is related to the same person is given in advance.

The calculation unit 120 includes a temporal matching unit 122, a local feature extraction unit 124, and a learning unit 126.

The learning section 126 includes a local distance calculation section 130, a loss function value calculation section 132, and an updating section 134.

As shown in FIG. 5, in the learning device 200, a pair of signature information that is time-series data and a training label indicating whether or not they are signature information of the same person are combined into a pair of training labels. The extraction model is updated based on one loss function value received as input as training data.

<Temporal matching part>
Similar to the temporal matching unit 22, the temporal matching unit 122 processes the first signature information of the signature information pair and the second signature information of the signature information pair on the time axis. The first matched signature information and the second matched signature information are output.

<Local feature extraction unit>
The local feature extraction unit 124 receives the first matched signature information and the second matched signature information as input, and uses an extraction model for extracting a local feature vector similarly to the local feature extraction unit 24. Then, at each time, a local feature vector is extracted and two local feature sequences are output.

<Local distance calculation section>
Similar to the local distance calculation unit 30, the local distance calculation unit 130 calculates the local feature vectors at each time extracted for the first matched signature information and the local feature vectors extracted for the second matched signature information. The local distance between the local feature vector of the time is calculated for each time, and one local distance sequence is output.

<Loss function value calculation section>
The loss function value calculation unit 132 calculates the distance between the local feature vector at each time extracted from the first matched signature information and the local feature vector at each time extracted from the second matched signature information, and calculates the value of the loss function expressed using the training label given to the pair of time series data.

Specifically, one local distance sequence and one training label are input, and in the case of signature information of the same person, the larger the local distance is, the larger the loss function value is. A single loss function value is calculated such that the smaller the loss function value, the larger the loss function value.

Two embodiments regarding the loss function value calculation section 132 will be described below.

<Example 1>
In this embodiment, first, at each time of the local distance sequence, in the case of signature information of the same person, the larger the local distance, the larger the value; otherwise, the smaller the local distance, the larger the value. Calculate one local loss function value such that Next, the average of the local loss function values calculated from all the times is calculated and used as the loss function value that is the output of the loss function value calculation unit 132. In this embodiment, the average of the local loss function values is used, but it may be the sum of the local loss function values. In short, it is sufficient if one value can be obtained using all the obtained local loss function values. Further, the loss function value is expected to (1) retain characteristics related to time series, and (2) be a value that is more in accordance with the learning data than in a heuristically determined method. Here, let L be the loss function value output from the loss function value calculation unit 132, and let z be the training label. Let z∈{0,1}, and let z = 1 if the signature information belongs to the same person, and z = 0 otherwise. Calculate L using equation (5). τ in equation (5) is a hyperparameter, and τ∈[0,√2].

(Five)

By equation (5), in the case of signature information of the same person, the local distance

The larger the loss function value L, the larger the loss function value L, otherwise the local distance

The smaller the loss function value L becomes.

<Example 2>
In the loss function value calculation unit 132 of this embodiment, the distance between signature information calculated by equation (2)

In the case of signature information of the same person,

The larger the value, the larger the value, and if

One loss function value is calculated such that the smaller the value, the larger the value, and is output from the loss function value calculation unit 132. Here, the loss function value is set to L, and calculation is performed using equation (6).

(6)

In the case of signature information of the same person, the local distance is

The larger is the distance between the signature information

becomes larger. Accordingly, the loss function value L increases according to equation (6). Otherwise, by equation (2), the local distance

The smaller is the distance between the signature information

becomes smaller. Accordingly, the loss function value L increases according to equation (6).

<Update section>
The updating unit 134 updates the parameters of the extraction model so as to optimize the value of the loss function.

Regarding the update section 134, two embodiments corresponding to two embodiments of the loss function value calculation section 132 will be described below.

<Example 1>
In this embodiment, the updating unit 134 updates the parameters of the extracted model using the differential method according to the following equation (7).

(7)

Equation (7) shows the partial differential coefficient with respect to each element x _ij of the local feature sequence X at the loss function value L . In formula (7)

is each local distance

is the partial differential coefficient with respect to x _ij of

It is calculated by , and has a different value depending on the time i of x _ij . Therefore, when learning the parameters of the neural network constituting the extraction model using the differential method, each element x _ij of the local feature sequence is adaptively updated according to the time i.

Also,

Similarly, in equation (7),

also has different values depending on time i of x _ij . Therefore, unlike the prior art such as the above-mentioned Non-Patent Document 2, in which each element of the local feature sequence is updated uniformly at all times during learning, in the embodiment of the present invention, during learning Since each element of the local feature sequence is adaptively updated according to its time, it becomes possible to more effectively learn local feature representations with higher discriminative power.

<Example 2>
In this embodiment, the updating unit 134 updates the parameters of the extracted model using the differential method according to equation (8) below.

(8)

Equation (8) shows the partial differential coefficient with respect to each element x _ij of the local feature sequence X at the loss function value L . As in Example 1, in formula (8)

has different values depending on time i of x _ij . When learning the parameters of the neural network constituting the extraction model using the differential method, each element x _ij of the local feature sequence is adaptively updated according to its time i. Therefore, it becomes possible to more effectively learn local feature expressions with higher discriminative power.

On the other hand, unlike Example 1, in formula (8)

has a uniform value at all times, regardless of time i of x _ij . As a result, when only a relatively small amount of training data can be used, the dependence of updates on the time of each element of the local feature sequence during learning can be appropriately suppressed, and the negative effects of overfitting can be reduced. become.

In an embodiment of the present invention, signature authentication is based on local distance between signature information.

It also depends on the distance between signature information, but ultimately the distance between signature information

It also depends on. Example 1 uses a loss function that depends on local distance, while Example 2 uses a loss function that depends on both local distance and distance between signature information. Therefore, it is considered that Example 2 is an example that achieves a better balance between effective learning of local feature expressions and suppression of overlearning.

Each of the verification device 100 and the learning device 200 is realized by a computer 84 shown in FIG. 6, as an example. The computer 84 includes a CPU 86, a memory 88, a storage section 92 that stores the program 82, a display section 94 that includes a monitor, and an input section 96 that includes a keyboard and a mouse. The CPU 86, memory 88, storage section 92, display section 94, and input section 96 are connected to each other via a bus 98.

The storage unit 92 is realized by an HDD, SSD, flash memory, or the like. The storage unit 92 stores a program 82 for causing the computer 84 to function as the verification device 100 or the learning device 200. The CPU 86 reads the program 82 from the storage unit 92, expands it to the memory 88, and executes the program 82. Note that the program 82 may be provided by being stored in a computer-readable medium.

<Action of the learning device according to the embodiment of the present invention>
Next, the operation of the learning device 200 according to the embodiment of the present invention will be explained. When the input unit 110 receives a plurality of pairs of signature information to which training labels have been assigned in advance, the learning device 200 executes the learning processing routine shown in FIG. 7 .

First, in step S100, the first signature information and the second signature information included in the pair of signature information are matched along the time axis, and the first matched signature information and the second matched signature information are matched. Output the completed signature information.

In step S102, the local feature extraction unit 124 inputs the first matched signature information and the second matched signature information, and uses an extraction model for extracting a local feature vector to perform the following operations at each time: Extract local feature vectors and output two local feature sequences.

In step S104, the local distance calculation unit 130 calculates the local feature vector at each time extracted for the first matched signature information and the local feature vector at each time extracted for the second matched signature information. The local distance between the two is calculated at each time, and one local distance sequence is output.

Then, in step S106, the loss function value calculation unit 132 calculates the local feature vector at each time extracted from the first matched signature information and the local feature vector at each time extracted from the second matched signature information. The value of the loss function expressed using the distance to the feature vector and the training label given to the pair of time series data is calculated.

In step S108, the updating unit 134 updates the parameters of the extracted model so as to optimize the value of the loss function.

In step S110, it is determined whether the processes of steps S100 to S108 have been performed for all pairs. If there is a pair for which the processes of steps S100 to S108 have not been performed, the process returns to step S100 and the processes of steps S100 to S108 are repeated for the pair. On the other hand, if the processes of steps S100 to S108 described above have been executed for all pairs, the learning process routine ends.

<Operation of the verification device according to the embodiment of the present invention>
Next, the operation of verification device 100 according to the embodiment of the present invention will be explained.

First, the input unit 10 receives first signature information, which is time-series data of input positions by input means such as a pen tablet. Then, when the input unit 10 receives input of one or more pieces of second signature information as a template signature, the verification device 100 executes the verification processing routine shown in FIG. 8 .

First, in step S120, the temporal alignment unit 22 aligns the first signature information and the second signature information along the time axis for each one or more pieces of second signature information. and outputs first matched signature information and second matched signature information.

In step S122, the local feature extraction unit 24 inputs the first matched signature information and the second matched signature information for each one or more pieces of second signature information, and the local feature extraction unit 24 receives the first matched signature information and the second matched signature information for each piece of second signature information, and receives Using the extracted extraction model, a local feature vector is extracted at each time, and two local feature sequences are output.

In step S124, the local distance calculation unit 30 calculates, for each one or more pieces of second signature information, the local feature vector at each time extracted for the first matched signature information and the second matched signature. A local distance between the extracted information and the local feature vector at each time is calculated for each time.

Then, in step S126, the local distance integration unit 32 integrates all the distances included in the local distance sequence for each one or more pieces of second signature information, and calculates the distance between one piece of signature information. .

In step S128, the determination unit 34 determines whether the first signature information and the second signature information are signature information regarding the same person based on the distance between the signature information calculated for each one or more pieces of second signature information. It is determined whether or not it is, and the verification processing routine is ended.

<Experiment results>
The results of an experiment conducted to confirm the effects of the method described in the above embodiment will be described. We conducted an experiment to determine whether the signature information test data was genuine signature information or signature information imitated by a forger. Four datasets were used as signature information datasets: MCYT (Reference 2), BiosecurID SONOF (Reference 3), and SUSIG visual and blind sub-corpuses (Reference 4).

[Reference 2] J. Ortega-Garcia, J. Fierrez-Aguilar, D. Simon, et al., “MCYT baseline corpus: A bimodal biometric database,” IEE Proceedings- Vision, Image and Signal Processing, vol. 150, no. 6, pp.395-401, 2003.
[Reference 3] J. Galbally, MD Cabrera, MA Ferrer-Ballester, et al., “Online signature recognition through the combination of real dynamic data and synthetically generated static data,” Pattern Recognition, vol. 48, no. 9, pp. 2921-2934, 2015.
[Reference 4] A. Kholmatov and BA Yanikoglu, “SUSIG: An on-line signature database, associated protocols and benchmark results,” Pattern Anal. Appl., vol. 12, no. 3, pp. 227-236, 2009 .

In addition, the four baselines are one in which DTW is applied to the function described in Reference 5, one in which Siamese network (SN) is used, one in which DTW is applied to Siamese network (SN), and one in which DTW is not applied. A Siamese network (SN) to which the above embodiment 1 was applied was adopted.

[Reference 5] M. Martinez-Diaz, J. Fierrez, R. P. Krish, and J. Galbally, “Mobile signature verification: Feature robustness and performance comparison,” IET Biometrics, vol. 3, no. 4, pp. 267- 277, 2014.

Furthermore, Table 1 shows the results of comparison using EER (%) as an evaluation index.

In Table 1, PW indicates that temporal matching is performed, and LE indicates that the loss functions of Examples 1 and 2 are used.

Also, MCYT (90/80/70%) indicates that the first 90/80/70% of MCYT is used as training data, and the rest is used as test data. FULL combines all four datasets, extracts and combines the first 90% of each dataset as training data, and uses the rest as test data.

Further, PSN w/ sequence-wise local embedding loss indicates the method of Example 2 in the above embodiment, and PSN w/ location-wise local embedding loss indicates the method of Example 1 in the above embodiment.

From Table 1 above, it was found that the method of this embodiment was superior to all baselines.

As explained above, the verification device according to the embodiment of the present invention associates local feature vectors for each pair of signature information to which it is previously determined whether the signatures are related to the same person or not at each time. Using the learned extraction model, the local feature vector is extracted at each time, the distance between the local feature vectors at each time is calculated, and the first signature information and the second signature information are , it is determined whether the signatures are related to the same person. Thereby, it is possible to accurately determine whether the two pieces of signature information are signature information about the same person.

Further, the learning device according to the embodiment of the present invention extracts local feature vectors at each time using an extraction model for pairs of signature information to which training labels are assigned in advance, and The extraction model is learned to calculate the distance between local feature vectors and optimize the distance between local feature vectors at each time and the value of the loss function expressed using the training labels. As a result, it is possible to learn an extraction model that extracts a local feature vector for accurately determining whether two pieces of signature information are related to the same person.

Further, according to an embodiment of the present invention, there is a function to impose temporal consistency as a constraint when measuring the distance between signature information (authentication stage), and a function to directly learn local characteristics of signature information from data. (training stage) and can be realized at the same time. As a result, when another person's signature or clone signature is input as the signature information to be authenticated, it is possible to lower the acceptance rate of others, and at the same time, it is possible to express signature characteristics that are optimal for the purpose of signature authentication at hand, resulting in higher signatures. It becomes possible to achieve authentication accuracy.

In addition, by using the temporal matching unit, nonlinear temporal fluctuations such as fluctuations in time and speed that may occur between signature information of the same person can be absorbed, which is important for learning local features directly from data. It becomes possible to accurately specify the correspondence between local fragments. As a result, when a principal's signature is input as a signature to be authenticated, it is possible to lower the probability that the principal's signature is determined to be inconsistent with the template signature (principal rejection rate).

<Modified example>
Note that the present invention is not limited to the embodiments described above, and various modifications and applications can be made without departing from the gist of the present invention.

For example, the local distance calculation section and the local distance integration section can be configured as in the following embodiment.

In this embodiment, the local distance calculation section and the local distance integration section do not use a manually designed distance measurement method and integration method as in the first and second embodiments, but instead use the distance measurement method and integration method that are designed manually as in the first and second embodiments. Methods and integration methods are also determined using learning. Thereby, when a sufficient amount of training data is available, it becomes possible to more effectively learn the signature information feature vector and more accurately match two pieces of signature information.

Specifically, the local distance calculation section first calculates the difference between the two local feature sequences output from the local feature extraction section, that is, XY. Regard the calculated XY as a 1×W×K tensor. W is the time length and K is the number of dimensions; for example, in the embodiment shown in FIG. 4, W = 256 and K = 64. Next, using a neural network, inputting the tensor XY, a local distance sequence having a sufficient temporal length and a dimension of 1 or more is calculated and output. For example, use two 1x1 convolutional layers, set the dimensionality of both outputs to 64, and set the stride to 1, and use a Batch Normalization layer and a ReLU layer immediately after each convolutional layer. It is possible to use convolutional neural networks. In this case, the output local distance sequence has a temporal length of 256 and a dimensionality of 64, that is, a tensor of 1×256×64. Furthermore, instead of the convolutional neural network, it is also possible to use a bidirectional LSTM or a bidirectional GRU.

In addition, as another embodiment, in the two local feature sequences, instead of their differences, both are combined along the dimensional direction, and the combined sequences are input to the neural network, and the same neural network The local distance sequence may be calculated using . If the two local feature sequences are each considered as a 1×W×K tensor, the combined sequence becomes a 1×W×2K tensor. As the neural network, a convolutional neural network, a bidirectional LSTM, or a bidirectional GRU can be used, as in the embodiments described above.

The local distance integration unit uses a neural network, receives the local distance sequence as input, and outputs the distance between pieces of signature information. An example of the neural network is a neural network that includes one fully connected layer and one Sigmoid layer with an output dimension set to one. Here, the distance between the calculated signature information is

shall be. When using the neural network,

becomes.

In the training stage, the parameters of the neural network of the local distance calculation unit and the local distance integration unit are determined by learning using a large number of training data, similar to and simultaneously with the parameters of the extraction model of the local feature extraction unit. . The loss function value calculation section first receives the local distance sequence as input and calculates the distance between pieces of signature information using the same neural network as the local distance integration section. Next, the distance between the signature information

and the training label z, one loss function value L is calculated, for example, according to equation (9).

(9)

In the case of signature information of the same person, the larger the local distance, the greater the distance between the signature information.

becomes larger, and according to equation (9), the loss function value L becomes larger. Otherwise, the smaller the local distance, the more the distance between the signature information

becomes smaller, and the loss function value L becomes larger according to equation (9).

Further, although the present invention has been described as an example in which the present invention is applied to a verification device that verifies signature information, the present invention is not limited thereto, and may be applied to a verification device that verifies other types of time-series data. For example, the present invention may be applied to a verification device that uses a sensor as the input means and verifies fingerprint information that is time-series data in response to input by the sensor. This fingerprint information is time-series data that shows which part of the finger touches the sensor first and with what pressure, and the fingerprint information is compared to determine whether the fingerprint information is from the same person. .

Further, depending on the target time-series data, there may be no temporal deviation, and in such a case, the temporal matching section may be omitted.

10, 110 Input section 20, 120 Calculation section 22, 122 Temporal matching section 24, 124 Local feature extraction section 26 Matching section 30, 130 Local distance calculation section 32 Local distance integration section 34 Judgment section 50 Output section 82 Program 84 Computer 100 Collation device 126 Learning unit 132 Loss function value calculation unit 134 Update unit 200 Learning device

Claims (4)

an input unit that accepts first signature information and second signature information that are time-series data of input positions by the input means, and that are different from each other;
For a local feature sequence including a feature vector extracted from each time for the first signature information and a local feature sequence including a feature vector extracted from each time for the second signature information, and calculates a temporal correspondence that matches the local feature sequence of the first signature information and the local feature sequence of the second signature information along the time axis using the time correspondence . A matching part;
For each of the matched local feature sequence of the first signature information and the local feature sequence of the second signature information, a pair of signature information is assigned in advance whether or not the signature is related to the same person. The convolutional neural network is a convolutional neural network that associates the local feature vectors of 2 to 3 for each corresponding time, and that extracts the local feature vectors that have been learned in advance so that the local feature vectors of signatures related to the same person correspond. a local feature calculation unit that extracts the local feature vector at each time using the output of the intermediate layer of the extraction model;
The distance between the local feature vectors at each time is calculated for the local feature vector at each time extracted for the first signature information and the local feature vector at each time extracted for the second signature information. If the score according to the distance is smaller than the threshold, it is determined that the first signature information and the second signature information are signatures related to the same person, and the score according to the distance is greater than or equal to the threshold. a verification unit that determines that the first signature information and the second signature information are not signatures related to the same person;
including;
The local feature calculation unit calculates the local feature sequence of the first signature information based on the matched local feature sequence of the first signature information and the local feature sequence of the second signature information. A first tensor of 1×T×D (T is the temporal length and D is the number of dimensions of the feature vector) representing a feature vector at each time obtained from the feature sequence , and the second signature information. generate a 1×T×D second tensor representing the feature vector at each time obtained from the local feature sequence of A verification device that extracts a local feature vector at a time, inputs the second tensor into the extraction model, and extracts a local feature vector at each time for the second signature information. First signature information and second signature information, each of which is time-series data of input positions by an input means, to which a training label indicating whether or not the signatures are related to the same person are given in advance, and each of which is different from the first signature information. A learning device that learns an extraction model that is a convolutional neural network that extracts a local feature vector based on a pair of signature information and the second signature information, comprising:
for a local feature sequence including a feature vector extracted from each time for the first signature information of the pair and a local feature sequence including a feature vector extracted from each time for the second signature information of the pair. , calculate a time correspondence , and use the time correspondence to match the local feature sequence of the first signature information and the local feature sequence of the second signature information along the time axis. a temporal matching section;
a local feature calculation unit that extracts a local feature vector at each time using the output of the intermediate layer of the extraction model for each local feature sequence of the matched pair of signature information;
The distance between the local feature vectors at each time between the local feature vector at each time extracted from the signature information of one of the pair and the local feature vector at each time extracted from the signature information of the other pair. When the training label indicates that the signatures are related to the same person, the larger the distance, the larger the loss function value, and when the training label indicates that the signatures are not related to the same person, the distance The value of the loss function expressed using the distance of the local feature vector at each time and the training label given to the pair is optimized so that the smaller the value of the loss function is, the larger the loss function value is. a learning unit that learns the extraction model;
including;
The local feature calculation unit calculates a local feature sequence of the first signature information based on the matched local feature sequence of the first signature information and the local feature sequence of the second signature information. A first tensor of 1×T×D (T is the temporal length and D is the number of dimensions of the feature vector) representing a feature vector at each time obtained from the feature sequence , and the second signature information. A 1×T×D second tensor representing the feature vector at each time obtained from the local feature sequence of is generated, and the first tensor is input to the extraction model, and each A learning device that extracts a local feature vector at a time, inputs the second tensor to the extraction model, and extracts a local feature vector at each time for the second signature information. an input unit receives first signature information and second signature information that are time-series data of input positions by the input means, the first signature information and the second signature information being different, respectively;
The temporal matching unit generates a local feature sequence including a feature vector extracted from each time for the first signature information, and a local feature sequence including a feature vector extracted from each time for the second signature information. A time correspondence is calculated for , and the local feature sequence of the first signature information and the local feature sequence of the second signature information are plotted on the time axis using the time correspondence. aligned along
The local feature calculation unit assigns in advance whether or not the signatures are related to the same person, to each of the local feature sequence of the matched first signature information and the local feature sequence of the second signature information. The local feature vectors for each pair of signature information are associated with each corresponding time, and the local feature vectors are learned in advance so that the local feature vectors of signatures related to the same person correspond to each other. Extracting the local feature vector at each time using the output of the intermediate layer of the extraction model, which is a convolutional neural network,
The verification unit determines the local feature at each time with respect to the local feature vector at each time extracted for the first signature information and the local feature vector at each time extracted for the second signature information. The distance between the vectors is calculated, and if the score according to the distance is smaller than the threshold, it is determined that the first signature information and the second signature information are signatures related to the same person, and the signature information according to the distance is determined. determining that the first signature information and the second signature information are not signatures related to the same person when the score is equal to or higher than a threshold;
The local feature calculation unit calculates the local feature sequence of the first signature information based on the matched local feature sequence of the first signature information and the local feature sequence of the second signature information. A first tensor of 1×T×D (T is the temporal length and D is the number of dimensions of the feature vector) representing a feature vector at each time obtained from the feature sequence , and the second signature information. generate a 1×T×D second tensor representing the feature vector at each time obtained from the local feature sequence of A matching method comprising: extracting a local feature vector at each time, inputting the second tensor into the extraction model, and extracting a local feature vector at each time for the second signature information. to the computer,
receiving first signature information and second signature information that are time-series data of input positions by the input means, the first signature information and the second signature information being different, respectively;
For a local feature sequence including a feature vector extracted from each time for the first signature information and a local feature sequence including a feature vector extracted from each time for the second signature information, and aligning the local feature sequence of the first signature information and the local feature sequence of the second signature information along the time axis using the time correspondence ;
For each of the matched local feature sequence of the first signature information and the local feature sequence of the second signature information, a pair of signature information is assigned in advance whether or not the signature is related to the same person. The convolutional neural network is a convolutional neural network that associates the local feature vectors of 2 to 3 for each corresponding time, and that extracts the local feature vectors that have been learned in advance so that the local feature vectors of signatures related to the same person correspond. Extracting the local feature vector at each time using the output of the intermediate layer of the extraction model,
The distance between the local feature vectors at each time is calculated for the local feature vector at each time extracted for the first signature information and the local feature vector at each time extracted for the second signature information. If the score according to the distance is smaller than the threshold, it is determined that the first signature information and the second signature information are signatures related to the same person, and the score according to the distance is greater than or equal to the threshold. , the program determines that the first signature information and the second signature information are not signatures related to the same person,
Extracting the local feature vector may include extracting the local feature vector of the first signature information based on the matched local feature sequence of the first signature information and the local feature sequence of the second signature information. a first tensor of 1×T×D (T is the temporal length and D is the dimensionality of the feature vector) representing the feature vector at each time obtained from the local feature sequence of generate a 1×T×D second tensor representing the feature vector at each time obtained from the local feature sequence of the signature information, input the first tensor to the extraction model, and generate the first signature A program that extracts a local feature vector at each time for information, inputs the second tensor to the extraction model, and extracts a local feature vector at each time for the second signature information.