JP7755215B2 - Learning device, conversion device, learning method, conversion method, and program - Google Patents

Description

The present invention relates to a learning device, a conversion device, a learning method, a conversion method, and a program.

Technology is known that uses machine learning to generate a mathematical model that converts the representation of data to be converted into a predetermined representation. The predetermined representation is, for example, a representation required by a downstream task. Note that converting the representation of data to be converted into a predetermined representation refers to an encoding process that converts the data to be converted into data expressed in a predetermined representation.

Kaiming He, et al., "Masked autoencoders are scalable vision learners," arXiv preprint arXiv:2111.06377, 2021. A. Baevski, et al., “data2vec: A general framework for self-supervised learning in speech, vision and language,” in arXiv preprint arXiv:2202.03555, 2022.

Masked Autoencoders (MAE) and data2vec have been proposed as such technologies. Both of these mask part of the input information and use the results for learning. However, both technologies sometimes have poor conversion accuracy.

In consideration of the above circumstances, the present invention aims to provide a technology that improves the accuracy of converting the representation of data to be converted into a specified representation.

One aspect of the present invention is a learning device that includes a control unit that acquires, through learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation. The control unit executes the following: a first representation conversion process that converts the representation of the target data into a target representation, using first data, which is data obtained by removing mask data that is part of the zeroth data, which is data expressed by tensors; a mask data representation prediction process that predicts a target representation of the mask data based on the result of the first representation conversion process; a second representation conversion process that converts the representation of the second data into a target representation based on second data that is part or all of the mask data; and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process; and the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is met.

One aspect of the present invention comprises a conversion target acquisition unit that acquires data to be converted into a target representation, which is a predetermined representation; and a control unit that acquires, through learning, a target representation conversion process, which is a process of converting the representation of the data to be converted into a target representation, which is a predetermined representation. The control unit performs a first representation conversion process, which converts the representation of the target data into a target representation, using first data, which is data obtained by removing mask data, which is part of the zeroth data, from the zeroth data, which is data expressed by a tensor, as the processing target; and a mask data representation prediction process that predicts the target representation of the mask data based on the result of the first representation conversion process. a learning device that executes: a second representation conversion process that converts a representation of the second data into a target representation based on second data that is part or all of the mask data; and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process; and a representation conversion unit that converts the representation of the data acquired by the conversion target acquisition unit using the target representation conversion process obtained by the learning device, wherein the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is satisfied.

One aspect of the present invention is a learning method that includes a control step for obtaining, through learning, a target representation conversion process, which is a process for converting a representation of data to be converted into a target representation, which is a predetermined representation. The control step executes the following steps: a first representation conversion process that converts the representation of the target data into a target representation, using first data, which is data obtained by removing mask data, which is part of the zeroth data, from the zeroth data, which is data expressed by tensors; a mask data representation prediction process that predicts a target representation of the mask data based on the result of the first representation conversion process; a second representation conversion process that converts the representation of the second data into a target representation based on second data, which is part or all of the mask data; and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process; and the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is met.

One aspect of the present invention includes a conversion target acquisition step for acquiring data to be converted into a target representation, which is a predetermined representation; and a control step for acquiring, by learning, a target representation conversion process, which is a process for converting the representation of the conversion target data into a target representation, which is a predetermined representation. The control step includes a first representation conversion process for converting the representation of the target data into a target representation, using first data, which is data obtained by removing mask data, which is part of the zeroth data, from the zeroth data, which is data expressed by a tensor; and a mask data representation prediction process for predicting the target representation of the mask data based on the result of the first representation conversion process. a learning method that executes: a second representation conversion process that converts a representation of the second data into a target representation based on second data that is a part or all of the mask data; and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process, and in which the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is satisfied.

One aspect of the present invention is a program for causing a computer to function as either the above-mentioned learning device or the above-mentioned conversion device.

This invention makes it possible to improve the accuracy of converting the representation of the data to be converted into a specified representation.

FIG. 1 is a diagram showing an example of the configuration of an expression conversion system according to an embodiment. FIG. 3 is an explanatory diagram illustrating a patch according to the embodiment. FIG. 3 is an explanatory diagram illustrating an outline of a processing flow executed by a learning unit in the embodiment. 10 is a flowchart showing an example of the flow of a learning process executed by a learning unit in the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning apparatus according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of a control unit in the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a conversion device according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of a control unit in the embodiment. 10 is a flowchart showing an example of a flow of processing executed by a conversion device according to an embodiment. FIG. 10 is a diagram showing an example of a result of an experiment according to the embodiment.

(Embodiment)
1 is a diagram illustrating an example of the configuration of an expression conversion system 100 according to an embodiment. The expression conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 acquires, through learning, a process (hereinafter referred to as a "target expression conversion process") for converting an expression of data to be converted into a predetermined expression (hereinafter referred to as a "target expression").

Note that converting the representation of data into a target representation refers to an encoding process that converts the data to be encoded into data expressed in the target representation. Therefore, converting the representation of data to be converted into a target representation refers to an encoding process that converts the data to be converted into data expressed in the target representation.

The target representation is, for example, a representation embedding. The target representation is, for example, a representation of 768 floating-point values. The target representation may also be, for example, a representation of 1024 floating-point values. The target representation may also be, for example, a representation of 2048 floating-point values. The representation conversion process is a type of learning model.

The data to be converted may be any data obtained based on image data, acoustic signal data, natural language data, or general time-series data, as long as it is expressed as a tensor. Therefore, the data to be converted may be, for example, image data, a spectrogram of acoustic signal data, or a series of word samples in which each word is expressed as an M-dimensional vector (M is a natural number greater than or equal to 1) (hereinafter referred to as a "natural language series"). The data to be converted may also be, for example, general time-series data in which symbols and numbers other than letters are expressed as vectors.

A spectrogram of the acoustic signal data is obtained based on the acoustic signal data. A natural language sequence is obtained based on the natural language data.

The learning device 1 includes a learning unit 10. The learning unit 10 executes a first representation conversion process, a mask data representation prediction process, a second representation conversion process, and an update process. Note that the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process may have been updated in advance, for example, by transfer learning.

The first expression conversion process is a process of converting an expression to be processed into a target expression. When the learning unit 10 executes the first expression conversion process, the processing target of the first expression conversion process is the first data.

The first data is data in which the mask data, which is part of the zeroth data, has been removed from the zeroth data. The zeroth data is data represented by a tensor.

The 0th data is, for example, a tensor that represents the original data after division into patches, and also has data position information that indicates the position of each patch, and is a tensor with tensors as elements. The 0th data is, for example, a matrix whose elements are the vectors of each patch resulting from patch division of the tensor to be divided. Patch division is the process of dividing the object to be divided into parts called patches.

Original data is data that corresponds to the object to be converted and is expressed as a tensor. Therefore, if the object to be converted is image data, the original data is image data data.

When the data to be converted is data obtained based on acoustic signal data, the original data is data obtained based on the acoustic signal data and expressed as a tensor. Such data is, for example, a spectrogram of the acoustic signal data.

When the data to be converted is data obtained based on natural language data, the original data is data obtained based on the natural language data and expressed as a tensor. Such data is, for example, a natural language sequence. When the data to be converted is data obtained based on general time series data, the original data is data obtained based on general time series data and expressed as a tensor. Such data is, for example, a time series of numerical values such as stock prices or temperature.

In this way, the original data is either image data, data obtained based on acoustic signal data and expressed in tensors, data obtained based on natural language data and expressed in tensors, or data obtained based on general time series data and expressed in tensors.

One element of the tensor in the 0th data represents information about one patch. One patch contains one or more elements of the tensor representing the original data. One element is data represented, for example, by a 768-dimensional vector. The position of a patch is the position within the 0th data of the element corresponding to each patch.

The division of patches and the assignment of data position information may be performed by the learning device 1. In other words, the process of obtaining the 0th data based on the original data (hereinafter referred to as the "0th data generation process") may be performed by the learning device 1, or by a device other than the learning device 1. The 0th data may be, for example, the original data itself. In such a case, there is no need to perform the 0th data generation process.

Figure 2 is an explanatory diagram explaining patches in an embodiment. Specifically, Figure 2 is an explanatory diagram explaining patches using an example in which the conversion target is acoustic signal data. More specifically, in the example of Figure 2, it is an explanatory diagram explaining patches using an example in which the acoustic signal data is a spectrogram composed of a frequency axis and a time axis.

To explain patches, Figure 2 shows the spectrogram divided into multiple rectangular sections of the same size on both the frequency axis and the time axis. A patch is data that represents each of these sections as a vector. For example, one patch is the data that represents one section of area D1 in Figure 2.

A patch is, for example, a vector that indicates the pixel values of each pixel included in the corresponding partition. The number of elements of a vector representing a patch is the same regardless of the vector. The number of elements of a vector representing a patch may be the same regardless of the vector, and may be proportional to the number of pixels included in the partition, for example. The proportionality coefficient may or may not be 1. If the proportionality coefficient is not 1, the value of each element may be a value obtained by interpolation, for example. For example, if one patch contains 256 pixels and the coefficient is 3, the patch is a 768-dimensional vector.

The data position information is, for example, a vector with the same number of dimensions as the vector representing the patch. Each element of the 0th data is, for example, a vector represented by the vector sum of the vector representing the patch and a vector indicating the data position information.

In the example of Figure 2, the position of each patch is indicated by its position in the frequency axis direction and the time axis direction. Therefore, in the example of Figure 2, the 0th data is a matrix with vectors as elements. In the example of Figure 2, if the patch is a vector with a number of elements proportional to the number of pixels, the 0th data is a matrix with first-order tensors as elements. It goes without saying that a first-order tensor is a vector, and a second-order tensor is a matrix. Incidentally, a 0th-order tensor is a scalar.

In addition, if the proportionality coefficient is greater than 1, it is possible to suppress an increase in information loss when data is encoded in various processes such as the first representation conversion process.

In this way, the 0th data is obtained based on the original data. The value of one element of the tensor representing the 0th data is a vector that represents each section when the tensor representing the original data is divided according to a predetermined rule. The dimension of the vector representing the section is the number of elements contained in the section, which is, for example, larger than the number of elements of the tensor representing the original data.

Returning to the explanation of Figure 1, the mask data representation prediction process is a process for predicting a target representation of mask data based on the result of the first representation conversion process. The second representation conversion process is a process for converting the representation of second data, which is part or all of the mask data, into a target representation based on the second data. The update process is a process for updating the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process (hereinafter referred to as the "conversion error").

The conversion error may be the MSE (mean square error) between the result of the mask data representation prediction process and the result of the second representation conversion process, or L1, which is the mean absolute value of the difference.

In this way, the learning unit 10 learns the first representation conversion process and the mask data representation prediction process so as to reduce the difference between the target representation of mask data obtained based on data excluding the mask data and the target representation of mask data obtained based on some or all of the mask data.

The conversion device 2 converts the representation of the data to be processed using the first representation conversion process at the time of update until a predetermined condition regarding the termination of the update (hereinafter referred to as the "update termination condition") is met.

Here, an example of the flow of processing executed by the learning unit 10 will be described with reference to the drawings.
3 is an explanatory diagram outlining the flow of processing executed by the learning unit 10 in the embodiment. In FIG. 3, data x is an example of 0th data. In the example of FIG. 3, the learning unit 10 generates first data and second data based on the 0th data x. In the example of FIG. 3, data D101 is an example of the first data, and data D102 is an example of the second data.

Data D101 is data in which a portion of the 0th data has been masked. Masking refers to a process that prevents data from being processed by other specified processes. In other words, masking refers to a process that restricts information access. The other specified processes are, for example, masked data representation prediction processes. An example of the results of masking processing will be explained by showing an example of how masked data is handled in the next stage. For example, if the input data is a series of "010101010101" and is converted by masking into data of the series "****0101****", the "*" data will not be processed in the next stage. This is the result of masking. In this example, the "*" will not be processed in the next stage. In other words, in this example, the "*" is an example of data with restricted information access.

In Figure 3, solid patches represent masked patches. Non-solid patches in Figure 3 represent unmasked patches. Patch P1 is an example of a solid patch. Patch P2 is an example of a non-solid patch. A collection of masked patches is an example of mask data.

In the example of Figure 3, mask data in the first data is data that is not mask data in the second data. Therefore, data that includes data that is not mask data in the first data and data that is not mask data in the second data includes data 0.

However, as described in the explanation of the second representation conversion process, it is not necessary that all of the mask data in the first data be non-mask data in the second data. Non-mask data in the second data may be part of the mask data in the first data.

The decision as to which data from the 0th data will be used as the mask data in the first data may be made in any manner, for example randomly. Hereinafter, the process of deciding which data from the 0th data will be used as the mask data in the first data will be referred to as the first mask data decision process.

As described above, the second data is, for example, data determined as mask data in the first data. However, as described above, not all of the data determined as mask data in the first data necessarily needs to be second data. An example of how to determine the second data when only some, but not all, of the data determined as mask data in the first data is second data will be explained below. In such a case, the determination of which of the data determined as mask data in the first data will be used as mask data may be made in any manner, for example, randomly.

Hereinafter, the process of determining which of the data determined to be mask data in the first data will be used as mask data will be referred to as the second mask data determination process. As can be seen from the explanation so far, if the second data is all of the data determined to be mask data in the first data, the second mask data determination process does not necessarily need to be performed. This is because the data determined to be mask data in the first data is the second data.

In the example of Figure 3, the first mask data determination process and the second mask data determination process are performed by the learning unit 10. However, the first mask data determination process and the second mask data determination process do not necessarily have to be performed by the learning unit 10. For example, a device other than the learning device 1 may perform the process up to the generation of the first data and the second data, and the generated first data and second data may be executed by the learning unit 10. In such a case, the learning unit 10 does not perform the first mask data determination process and the second mask data determination process. Note that if the second data is all of the mask data in the first data, the second mask data determination process does not need to be performed.

In the example of Fig. 3, the learning unit 10 performs a first conversion process on the first data. The function f _θ (·) in Fig. 3 represents the first conversion process. z _θ in Fig. 3 represents the result of the addition process. The addition process is a process of adding mask tokens to the result of the first conversion process on the first data, the number of which is the same as the number of elements belonging to the mask data. The mask token is information indicating whether or not an element in the 0th data belongs to the mask data.

An element belonging to mask data is an element of the tensor representing the 0th data that belongs to the mask data in the 1st data. Therefore, an element belonging to mask data is, for example, a patch that is determined to be mask data in the first mask data determination process.

In other words, the addition process is a process in which a vector obtained by adding together information indicating a position within the 0th data of patch P1 and information indicating that it is a mask token is added to the result of the first conversion process. In this way, the result of the addition process is an example of a result based on the result of the first conversion process. When the addition process is performed, as in the example of Figure 3, the addition process is performed, for example, by the learning unit 10.

In the example of Fig. 3, the learning unit 10 performs a mask data representation prediction process on the result of the addition process. In this way, the addition process is performed after the first representation conversion process and before the mask data representation prediction process. The function q _θ (·) in Fig. 3 represents the mask data representation prediction process. y _θ in Fig. 3 represents the result of the mask data representation prediction process.

In the example of Fig. 3, the learning unit 10 performs a second conversion process on the second data. The function f _ξ (·) in Fig. 3 represents the second conversion process. The function f _θ representing the first conversion process and the function f _ξ representing the second conversion process are both parameterized functions, and the parameter values are updated by the update process.

The symbol "function f _θ" means function f with a parameter value of θ, and the symbol "function f _ξ" means function f with a parameter value of ξ. Therefore, the symbols "function f _θ " and "function f _ξ " indicate that the functions of the first conversion process and the second conversion process are the same except for the difference in parameters. z' _ξ in Figure 3 represents the result of the second conversion process.

The result of the mask data representation prediction process is a target representation of the mask data. The result of the second conversion process is a target representation of all or part of the mask data. Therefore, by updating the contents of the first representation conversion process, mask data representation prediction process, and second representation conversion process based on the conversion error to reduce the conversion error, the accuracy of converting the representation of the data to be converted into a specified representation is improved.

"Maximize agreement" in Figure 3 indicates that the contents of the first representation conversion process, mask data representation prediction process, and second representation conversion process are updated to reduce the conversion error. In other words, "Maximize agreement" in Figure 3 indicates that the learning unit 10 executes the update process.

"Stop gradient" in FIG. 3 indicates that backpropagation is not performed when updating the content of the second representation conversion process. Specifically, updating the content of the process involves updating the values of parameters included in the function executed in the process. Therefore, in the example of the second conversion process expressed by the function f _ξ , updating the value of the parameter ξ is equivalent to updating the content of the second conversion process. In the example of FIG. 3, the content of the second representation conversion process is updated not by backpropagation but by another update process. The other update process is, for example, a predetermined exponential moving average process based on the content of the first representation conversion process.

An example of exponential moving average processing will be described. Specifically, the content of the first conversion processing and the content of the second conversion processing change by changing the parameter values of the parameterized functions that represent each processing. The parameterized function that represents the first conversion processing is, for example, the function _fθ , and the parameterized function that represents the second conversion processing is, for example, the function _fξ . In this case, the value of the parameter ξ after the tth update obtained by the exponential moving average processing is, for example, ξ[t] = βξ[t-1] + (1-β)θ[t-1]. β is a predetermined constant. β is, for example, 0.99.

In fact, ξ[t] = βξ[t-1] + (1-β)θ[t-1] is the sum of the values of θ at each update, weighted according to a predetermined formula. Therefore, the exponential moving average process is the average value of the θ values at each update. The average value of the θ values at each update converges as the updates progress and accuracy increases. Therefore, the exponential moving average process enables updates to ξ with higher conversion accuracy. Then, if the value of ξ converges to a value with high conversion accuracy, the value of θ will also converge to a value with high conversion accuracy as a result of the update process, which attempts to reduce conversion error.

Figure 4 is a flowchart showing an example of the flow of the learning process performed by the learning unit 10 in an embodiment. The learning unit 10 acquires first data and second data (step S101). Next, the learning unit 10 performs a first conversion process (step S102). Next, the learning unit 10 performs a mask data expression prediction process (step S103). Next, the learning unit 10 performs a second conversion process (step S104).

Next, the learning unit 10 executes the update process (step S105). Next, the learning unit 10 determines whether the update termination condition is satisfied (step S106). If the update termination condition is satisfied (step S106: YES), the process ends. On the other hand, if the update termination condition is not satisfied (step S106: NO), the process returns to step S101.

The first representation conversion process performed when the update termination condition is met is the target representation conversion process. During learning by the learning unit 10, the processing target of the first representation conversion process was the first data. However, the mask data is not predetermined and is not the same each time learning is performed. Therefore, the first representation conversion process (i.e., the target representation conversion process) performed when the update termination condition is met can convert the representation of a processing target that does not contain mask data into a target representation with high accuracy. Note that the processes of steps S102 to S103 and step S104 may be performed in parallel.

Figure 5 is a diagram showing an example of the hardware configuration of the learning device 1 of an embodiment. The learning device 1 has a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and memory 92 connected by a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, input unit 12, communication unit 13, memory unit 14, and output unit 15.

More specifically, the processor 91 reads the program stored in the storage unit 14 and stores the read program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

The control unit 11 controls the operation of the various functional units included in the learning device 1. The control unit 11 includes a learning unit 10. Therefore, the control unit 11 executes, for example, a first representation conversion process, a mask data representation prediction process, a second representation conversion process, and an update process. The control unit 11 may further execute a first mask data determination process, a second mask data determination process, or an additional process.

The control unit 11 controls, for example, the operation of the output unit 15. The control unit 11 records in the memory unit 14 various information generated by the execution of various processes, such as the first representation conversion process, the mask data representation prediction process, the second representation conversion process, and the update process.

The input unit 12 is configured to include input devices such as a mouse, keyboard, and touch panel. The input unit 12 may be configured as an interface that connects these input devices to the learning device 1. The input unit 12 accepts input of various information to the learning device 1.

The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, the device that sent the zeroth data. When the external device is the device that sent the zeroth data, the communication unit 13 acquires the zeroth data by communicating with the device that sent the zeroth data.

The external device is, for example, the device that sent the original data. When the external device is the device that sent the original data, the communication unit 13 acquires the original data by communicating with the device that sent the original data. The external device is, for example, the device that sent the acoustic signal data.

The external device is, for example, a device that transmits acoustic signal data. When the external device is a device that transmits acoustic signal data, the communication unit 13 acquires the acoustic signal data by communicating with the device that transmits the acoustic signal data.

The external device is, for example, a device that transmits natural language data. When the external device is a device that transmits natural language data, the communication unit 13 acquires the natural language data by communicating with the device that transmits the natural language data.

The external device is a device that transmits general time series data, for example, vector representations of symbols and numbers other than characters. When the external device is a device that transmits general time series data, the communication unit 13 acquires the general time series data by communicating with the device that transmits the general time series data.

In addition, various information input to the communication unit 13 may be input to the input unit 12 instead of the communication unit 13.

The external device is, for example, the conversion device 2. The communication unit 13 communicates with the conversion device 2 and transmits to the conversion device 2 information indicating the content of the first expression conversion process (i.e., the target expression conversion process) at the time the update termination condition is satisfied.

The memory unit 14 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or semiconductor storage device. The memory unit 14 stores various information related to the learning device 1. The memory unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The memory unit 14 stores various information generated by the operation of, for example, the control unit 11.

The output unit 15 outputs various types of information. The output unit 15 is configured to include a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs information input to the input unit 12, for example. The output unit 15 may also display the results of processing by the control unit 11, for example.

Figure 6 is a diagram showing an example of the configuration of the control unit 11 in an embodiment. The control unit 11 includes a learning unit 10, a data acquisition unit 110, a memory control unit 120, a communication control unit 130, and an output control unit 140.

The data acquisition unit 110 acquires data to be sent to the learning unit 10. The data to be sent to the learning unit 10 may be the 0th data or a set of the 1st data and the 2nd data. When the data acquisition unit 110 sends a set of the 1st data and the 2nd data to the learning unit 10 and acquires the 0th data, the data acquisition unit 110 executes a first mask data determination process and a second mask data determination process.

When the input unit 12 or the communication unit 13 acquires acoustic signal data, the data acquisition unit 110 acquires, as raw data, data expressed in tensors that is obtained based on the acoustic signal data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired raw data.

When the input unit 12 or the communication unit 13 acquires natural language data, the data acquisition unit 110 acquires, as original data, data expressed as tensors that are obtained based on the natural language data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired original data.

When the input unit 12 or the communication unit 13 acquires general time series data in which symbols and numbers other than letters are expressed as vectors, the data acquisition unit 110 acquires, as original data, data expressed as tensors that is obtained based on the general time series data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired original data.

When the input unit 12 or the communication unit 13 acquires the zeroth data, the data acquisition unit 110 acquires the zeroth data acquired by the input unit 12 or the communication unit 13. When the input unit 12 or the communication unit 13 acquires a pair of first data and second data, the data acquisition unit 110 acquires the pair of first data and second data acquired by the input unit 12 or the communication unit 13.

The memory control unit 120 records various information in the memory unit 14. The communication control unit 130 controls the operation of the communication unit 13. The output control unit 140 controls the operation of the output unit 15.

Figure 7 is a diagram showing an example of the hardware configuration of the conversion device 2 in an embodiment. The conversion device 2 has a control unit 21 including a processor 93 such as a CPU and memory 94 connected by a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, input unit 22, communication unit 23, memory unit 24, and output unit 25.

More specifically, the processor 93 reads the program stored in the storage unit 24 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the conversion device 2 functions as a device comprising a control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

The control unit 21 controls the operation of the various functional units of the conversion device 2. The control unit 21 acquires, for example, information obtained by the learning device 1 indicating the content of the first expression conversion process (i.e., the target expression conversion process) at the time when the update termination condition was satisfied, and records this information in the memory unit 24.

The control unit 21 executes the target expression conversion process. The control unit 21 executes the target expression conversion process, for example, by reading and executing information indicating the contents of the target expression conversion process recorded in the memory unit 24. The control unit 21 controls, for example, the operation of the output unit 25. The control unit 21 records, for example, various information generated by the execution of the target expression conversion process in the memory unit 24.

The input unit 22 is configured to include input devices such as a mouse, keyboard, and touch panel. The input unit 22 may be configured as an interface that connects these input devices to the conversion device 2. The input unit 22 accepts input of various information for the conversion device 2.

The communication unit 23 includes a communication interface for connecting the conversion device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless connections. The external device is, for example, a device that transmits data to be converted into a target expression. The communication unit 23 acquires data to be converted into a target expression by communicating with such an external device. The external device is, for example, the learning device 1. The communication unit 23 acquires information indicating the content of the target expression conversion process by communicating with the learning device 1.

In addition, various information input to the communication unit 23 may be input to the input unit 22 instead of the communication unit 23.

The memory unit 24 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or semiconductor storage device. The memory unit 24 stores various information related to the conversion device 2. The memory unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The memory unit 24 stores various information generated by, for example, the operation of the control unit 21. The memory unit 24 stores, for example, the contents of the target expression conversion process.

The output unit 25 outputs various types of information. The output unit 25 is, for example, a communication interface connected to a device that executes a downstream task so that it can communicate with the device. The output unit 25 may be configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the conversion device 2. The output unit 25 outputs, for example, information input to the input unit 22. The output unit 25 may also output, for example, the results of executing the target expression conversion process.

Figure 8 is a diagram showing an example of the configuration of the control unit 21 in an embodiment. The control unit 21 includes a conversion target acquisition unit 210, an expression conversion unit 220, a memory control unit 230, a communication control unit 240, and an output control unit 250. The conversion target acquisition unit 210 acquires data that is input to the communication unit 23 and is the target of expression conversion to a target expression.

The expression conversion unit 220 uses a target expression conversion process to convert the expression of the data acquired by the conversion target acquisition unit 210. The expression conversion unit 220 may execute various processes executed by the data acquisition unit 110, such as the 0th data generation process, depending on the data acquired by the conversion target acquisition unit 210, so as to be able to execute the target expression conversion process.

The memory control unit 230 records various information in the memory unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control unit 250 controls the operation of the output unit 25.

Figure 9 is a flowchart showing an example of the flow of processing executed by the conversion device 2 in an embodiment. The conversion target acquisition unit 210 acquires data that is input to the communication unit 23 and is the target of conversion of its expression to a target expression (step S201). Next, the expression conversion unit 220 converts the expression of the data acquired in step S201 using a target expression conversion process (step S202). Next, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the result of step S202 (step S203). Note that the output destination of the output unit 25 may be, for example, a device that executes a downstream task.

<Experimental Results>
The results of an experiment in which downstream tasks were performed on data whose expression was converted using the target expression conversion process will be described with reference to Fig. 10. The downstream tasks performed in the experiment were environmental sound classification, voice command word identification, speaker identification, spoken language classification, emotion classification contained in voice, music genre classification, musical instrument classification, and musical pitch classification.

Figure 10 shows an example of the results of an experiment in an embodiment. "ESC50" and "US8K" both indicate the task of classifying environmental sounds. "SPCV2" indicates the task of identifying voice command words. "VC1" indicates the task of speaker identification. "VF" indicates the task of speech language classification. "CRM-D" indicates the task of classifying emotions contained in voice. "GTZAN" indicates the task of music genre classification. "NSynth" indicates the task of musical instrument classification of musical sounds. "Surge" indicates the task of musical pitch classification of musical sounds.

"MAE" refers to MAE (Masked Autoencoders), an existing technology. "MABL" refers to target representation conversion processing. The results in Figure 9 show that the accuracy of downstream tasks using the results of representation conversion by target representation conversion processing is higher than the accuracy of downstream tasks using the results of representation conversion by MAE, for all downstream tasks.

For example, for "ESC50", the accuracy of the downstream task "ESC50" using the results of expression conversion by MAE is 87.35%, while the accuracy of the downstream task "ESC50" using the results of expression conversion by the target expression conversion process is 89.03%.

For example, for "VC1", the accuracy of the downstream task "VC1" using the results of expression conversion by MAE is 54.64%, while the accuracy of the downstream task "VC1" using the results of expression conversion by target expression conversion processing is 58.96%.

The learning device 1 configured in this manner learns the first representation conversion process so as to reduce the difference between the representation of mask data obtained based on data excluding the mask data and the representation of mask data obtained based on some or all of the mask data.

This differs from MAE, which reconstructs the patch image of the masked portion from the representation of the unmasked patch output by the model and calculates the loss using the difference between the input signal and the reconstructed signal. With MAE, reconstruction (i.e., decoding) is performed after the representation is converted, which can result in errors in the information. Therefore, unlike MAE, the learning device 1, which does not perform reconstruction during learning, can improve the accuracy of converting the representation of the data to be converted into a specified representation (i.e., target representation).

Furthermore, learning by learning device 1 differs from data2vec. With data2vec, all patches are input to obtain the target representation of the moving average model, and only the masked portion of that is used as the training signal. As a result, the target representation of the masked portion contains information about the unmasked portion. Note that the masked portion refers to the masked data, and the unmasked portion refers to the data in the 0th data other than the masked data. As a result, with data2vec, learning relies on information about the unmasked portion, and learning that improves the accuracy of representation conversion may not be performed.

On the other hand, in learning by the learning device 1, the second representation conversion process obtains a target representation of some or all of the masked data without using any data that is not masked data among the 0th data. The result of the second representation conversion process is then compared with the result of the masked data representation prediction process. That is, in learning by the learning device 1, the result of the second representation conversion process is used as the teacher signal. As mentioned above, the result of the second representation conversion process does not include information on the non-masked portion. Therefore, unlike data2vec, the learning device 1 can improve the accuracy of converting the representation of the data to be converted into a specified representation.

(Modification)
In the first data, the proportion of mask data may be 50% of the data included in the zeroth data. When the proportion of mask data is 50%, the accuracy of converting the representation of the data to be converted into a predetermined representation is improved compared to when the proportion is not 50% for the following reason (hereinafter referred to as the "first reason").

<First reason>
The target representations output by the first representation conversion process and the second representation conversion process each have a small conversion error when the entire data, including the respective mask data, is modeled. Modeling is easy when there is little mask data, and difficult when there is a lot of mask data. Target representations processed with different levels of difficulty are heterogeneous, with different amounts of information available for modeling. By setting the proportion of mask data to 50%, the first data and second data become homogeneous target representations processed using the same amount of information, and the conversion error is optimal for learning better modeling.

In the first data, the proportion of data that is not mask data may be 50% or less of the data contained in the 0th data, and the proportion of data that is part of the mask data and is used in the second representation conversion process may also be 50% or less of the data contained in the 0th data. In such cases, modeling becomes more difficult, and the accuracy of converting the representation of the data to be converted into a specified representation is improved compared to other cases.

The learning device 1 and the conversion device 2 may each be implemented using multiple information processing devices connected to each other via a network. In this case, the functional units of the learning device 1 and the conversion device 2 may be distributed and implemented across multiple information processing devices.

Note that the learning device 1 and the conversion device 2 do not necessarily need to be implemented as separate devices. The learning device 1 and the conversion device 2 may, for example, be implemented as a single device that combines the functions of both.

In addition, all or part of the functions of the expression conversion system 100, learning device 1, and conversion device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via telecommunications lines.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and also includes designs that do not deviate from the gist of the present invention.

100... Expression conversion system, 1... Learning device, 2... Conversion device, 10... Learning unit, 11... Control unit, 12... Input unit, 13... Communication unit, 14... Memory unit, 15... Output unit, 110... Data acquisition unit, 120... Memory control unit, 130... Communication control unit, 140... Output control unit, 21... Control unit, 22... Input unit, 23... Communication unit, 24... Memory unit, 25... Output unit, 210... Conversion target acquisition unit, 220... Expression conversion unit, 230... Memory control unit, 240... Communication control unit, 250... Output control unit, 91... Processor, 92... Memory, 93... Processor, 94... Memory

Claims (8)

a control unit that acquires, through learning, a target expression conversion process, which is a process of converting a representation of data to be converted into a target expression that is a predetermined representation;
Equipped with
The control unit
a first representation conversion process for converting a representation of the target data to a target representation, the first representation conversion process being data obtained by removing mask data that is a part of the zeroth data from the zeroth data that is data expressed by a tensor; a mask data representation prediction process for predicting a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process for converting a representation of the second data to a target representation based on second data that is a part or all of the mask data; and an update process for updating contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process;
the first representation conversion process at the time when a predetermined condition regarding the end of the update is satisfied is the target representation conversion process;
Learning device. the control unit executes an addition process after execution of the first representation conversion process and before the mask data representation prediction process, in which mask tokens indicating whether or not elements in the zero data belong to mask data are added to a result of the first representation conversion process, the number of mask tokens being the same as the number of elements belonging to the mask data;
The processing target of the mask data representation prediction processing is the result of the additional processing.
The learning device according to claim 1 . a ratio of mask data in the first data to data included in the 0 data is 50%;
The learning device according to claim 1 . In the first data, a ratio of data that is not mask data to a ratio of data included in the 0 data is 50% or less, and a ratio of data that is part of the mask data and is used in the second representation conversion process is also 50% or less to a ratio of data included in the 0 data.
The learning device according to claim 1 . a conversion target acquisition unit that acquires data that is a target of expression conversion into a target expression that is a predetermined expression;
a control unit that acquires, by learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation, wherein the control unit executes: a first representation conversion process, which takes first data as a processing target, which is data obtained by excluding mask data that is a part of zeroth data, which is data expressed by a tensor, from the zeroth data, the first data, and converts the representation of the processing target into a target representation; a mask data representation prediction process, which predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process, which converts a representation of the second data into a target representation based on second data that is a part or all of the mask data; and an update process, which updates contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; and a representation conversion unit that converts the representation of data acquired by the conversion target acquisition unit using the target representation conversion process acquired by a learning device, wherein the first representation conversion process at the time when a predetermined condition for completion of the update is satisfied is the target representation conversion process;
A conversion device comprising: a control step of acquiring, by learning, a target expression conversion process, which is a process of converting a representation of data to be converted into a target expression that is a predetermined representation;
and
The control step
a first representation conversion process for converting a representation of the target data to a target representation, the first representation conversion process being data obtained by removing mask data that is a part of the zeroth data from the zeroth data that is data expressed by a tensor; a mask data representation prediction process for predicting a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process for converting a representation of the second data to a target representation based on second data that is a part or all of the mask data; and an update process for updating contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process;
the first representation conversion process at the time when a predetermined condition regarding the end of the update is satisfied is the target representation conversion process;
How to learn. a conversion target acquisition step of acquiring data to be converted into a target expression, which is a predetermined expression;
a control step of acquiring, by learning, a target representation conversion process that converts a representation of data to be converted into a target representation that is a predetermined representation, wherein the control step executes: a first representation conversion process that converts a representation of a target representation into a target representation, using first data as a processing target, the first data being data obtained by excluding mask data that is a part of the zeroth data, the zeroth data being data expressed by a tensor; a mask data representation prediction process that predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process that converts a representation of the second data into a target representation based on second data that is a part or all of the mask data; and an update process that updates contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process, and wherein the first representation conversion process at the time a predetermined condition for ending the update is satisfied is the target representation conversion process;
A conversion method having the following structure: A program for causing a computer to function as either the learning device described in any one of claims 1 to 4 or the conversion device described in claim 5.