JP7552000B2 - Method and apparatus for training a multimodal representation model and for cross-modal search - Google Patents

Description

The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and smart search technology, and more specifically to a method and device for training a multimodal representation model, a method and device for cross-modal search, an electronic device, a computer-readable storage medium, and a computer program product.

A data modal is the form in which data exists, e.g., text, image, video, audio, etc. The same object or event can be represented by data of different modalities. In other words, data of different modalities may have the same or similar meaning.

The methods described in this section are not necessarily methods previously contemplated or adopted. Unless otherwise noted, any methods described in this section should not be considered prior art by their inclusion in this section alone. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art.

The present disclosure provides a method and apparatus for training a multimodal representation model, a method and apparatus for cross-modal search, an electronic device, a computer-readable storage medium, and a computer program product.

According to one aspect of the present disclosure, there is provided a method for training a multimodal representation model, the multimodal representation model including a first representation module corresponding to a first data modal and a second representation module corresponding to a second data modal, the method including: acquiring a sample tuple, the sample tuple including a first sample and a second sample of the first data modal, and a third sample and a fourth sample of the second data modal, the meanings of the samples included in the sample tuple being matched; inputting the first sample and the second sample into the first representation module, respectively, to acquire a first representation of the first sample and a second representation of the second sample; inputting the third sample and the fourth sample into the second representation module, respectively, to acquire a third representation of the third sample and a fourth representation of the fourth sample; and, based on the first representation and the second representation, acquiring the first representation. determining a first single-modal loss for the data modal of the first data modal based on the third representation and the fourth representation; determining a second single-modal loss for the second data modal based on the third representation and the fourth representation; determining a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal based on the first target representation and the second target representation, the first target representation being one of the first representation and the second representation, and the second target representation being one of the third representation and the fourth representation; determining an overall loss of the multimodal representation model based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss; and adjusting parameters of the multimodal representation model based on the overall loss.

According to one aspect of the present disclosure, a cross-modal search method is provided, comprising: obtaining a search target of a first data modality; inputting the search target into a multimodal representation model to obtain a first representation corresponding to the search target; obtaining a second representation of each of a plurality of candidate targets of a second data modality, the second representations being obtained by inputting the corresponding candidate targets into the multimodal representation model; determining a target target matching the search target from the plurality of candidate targets based on a similarity between the first representation and each second representation, the multimodal representation model being obtained by training using a training method based on the multimodal representation model.

According to one aspect of the present disclosure, there is provided a training apparatus for a multimodal representation model, the multimodal representation model including a first representation unit corresponding to a first data modal and a second representation unit corresponding to a second data modal, the apparatus including a first acquisition unit configured to acquire a sample tuple, the sample tuple including a first sample and a second sample of the first data modal and a third sample and a fourth sample of the second data modal, the meanings of the samples included in the sample tuple matching each other; a first input unit configured to input the first sample and the second sample into the first representation unit, respectively, to acquire a first representation of the first sample and a second representation of the second sample; a second input unit configured to input the third sample and the fourth sample into the second representation unit, respectively, to acquire a third representation of the third sample and a fourth representation of the fourth sample; and a first loss unit configured to acquire a first representation of the first data modal based on the first representation and the second representation. a second loss unit configured to determine a second single-modal loss for the second data modal based on the third representation and the fourth representation; a third loss unit configured to determine a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal based on the first target representation and the second target representation, the first target representation being one of the first representation and the second representation, and the second target representation being one of the third representation and the fourth representation; a fourth loss unit configured to determine an overall loss of the multimodal representation model based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss; and an adjustment unit configured to adjust parameters of the multimodal representation model based on the overall loss.

According to one aspect of the present disclosure, a cross-modal search device is provided, the device including: a first acquisition unit configured to acquire a search target of a first data modality; a representation unit configured to input the search target into a multimodal representation model to acquire a first representation corresponding to the search target; a second acquisition unit configured to acquire a second representation of each of a plurality of candidate targets of a second data modality, the second representation being obtained by inputting the corresponding candidate target into the multimodal representation model; and a determination unit configured to determine a target target matching the search target from the plurality of candidate targets based on a similarity between the first representation and each second representation, the multimodal representation model being obtained by training with a training device based on the multimodal representation model.

According to one aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform any one of the methods described above.

According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to execute any one of the above methods.

According to one aspect of the present disclosure, there is provided a computer program product including a computer program that, when executed by a processor, realizes any one of the above methods.

According to one or more embodiments of the present disclosure, a multimodal representation model can be guided to simultaneously learn intra- and inter-modal semantic associations, improving the generalizability and robustness of the multimodal representation model, thereby enabling the multimodal representation model to generate a unified semantic representation of different modal data.

It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be readily understood from the following specification.

The drawings illustratively illustrate the examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements.
1 is a flowchart illustrating a method for training a multi-modal representation model according to some embodiments of the present disclosure. FIG. 1 is an architectural block diagram illustrating a multimodal representation model according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram illustrating a training process of a multi-modal representation model according to some embodiments of the present disclosure. 1 is a flow chart illustrating a cross-modal search method according to some embodiments of the present disclosure. FIG. 1 is a schematic diagram illustrating cross-modal search effects on images and texts of a multi-modal representation model according to an embodiment of the present disclosure. 1 is a flowchart illustrating a cross-modal reasoning method according to some embodiments of the present disclosure. FIG. 1 is a block diagram illustrating an apparatus for training a multi-modal expression model according to some embodiments of the present disclosure. FIG. 1 is a block diagram illustrating a cross-modal search device according to some embodiments of the present disclosure. FIG. 1 is a block diagram illustrating a cross-modal reasoning device according to some embodiments of the present disclosure. FIG. 1 is a block diagram illustrating an example electronic device in which some embodiments of the present disclosure can be implemented.

Below, exemplary embodiments of the present disclosure are described in conjunction with the drawings, and various details in the embodiments of the present disclosure contained therein are to be considered as merely illustrative, in order to facilitate understanding. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

In this disclosure, unless otherwise specified, the use of terms such as "first," "second," and the like to describe various elements is not intended to limit the location, timing, or importance of those elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.

The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless the number of elements is specifically limited. It should be noted that the term "and/or" as used in the present disclosure covers any and all possible combinations of the listed items.

In some scenarios, a user may have a need for cross-modal data processing. For example, a user may have a need to perform a cross-modal search, which is a search scenario in which the modality of the input data is different from the modality of the retrieved data. For example, in a cross-modal search scenario involving images and text, a user inputs the search text "dog" and desires to retrieve images containing dogs on the screen.

In related technologies, a multimodal semantic representation model (abbreviated as "multimodal representation model" in the following content) may be used to convert data of different modalities into semantic representations in the same semantic space (abbreviated as "representation" in the following content), and then perform cross-modal data processing tasks such as cross-modal search and cross-modal inference (e.g., visual inference, visual entertainment) based on the representations. However, the generalizability and robustness of the multimodal representation model in related technologies is low, making it difficult to generate a unified representation of multimodal data.

In response to the above problems, an embodiment of the present disclosure provides a method for training a multimodal representation model, improving the generalizability and robustness of the multimodal representation model, so that the multimodal representation model can generate a unified representation of multimodal data.

The following describes in detail the embodiments of the present disclosure with reference to the drawings.

FIG. 1 is a flowchart illustrating a method 100 for training a multimodal representation model according to an embodiment of the present disclosure. The method 100 may be executed on a server or on a client device. That is, each step of the method 100 may be executed by a server or a client device.

In an embodiment of the present disclosure, the multimodal representation model includes a first representation module corresponding to a first data modal and a second representation module corresponding to a second data modal.

As shown in FIG. 1, method 100 includes steps S110 to S180.

In step S110, a sample tuple is obtained, the sample tuple including a first sample and a second sample of a first data modal and a third sample and a fourth sample of a second data modal, where the meanings of the samples included in the sample tuple match.

In step S120, the first sample and the second sample are input to a first representation module to obtain a first representation of the first sample and a second representation of the second sample.

In step S130, the third sample and the fourth sample are input to a second representation module to obtain a third representation of the third sample and a fourth representation of the fourth sample.

In step S140, a first single-modal loss for the first data modal is determined based on the first representation and the second representation.

In step S150, a second single-modal loss for the second data modal is determined based on the third and fourth representations.

In step S160, a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal are determined based on the first target representation and the second target representation, where the first target representation is one of the first representation and the second representation, and the second target representation is one of the third representation and the fourth representation.

In step S170, an overall loss of the multimodal representation model is determined based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss.

In step S180, the parameters of the multimodal representation model are adjusted based on the overall loss.

According to an embodiment of the present disclosure, a sample tuple is used to train a multimodal representation model, where the sample tuple includes two samples of different data modalities, and the meanings of each sample in the sample tuple are matched. Based on the sample tuple, in the training process, not only the single-modal loss of each data modal is calculated, but also the cross-modal loss between different modalities is calculated, so that the multimodal representation model can be guided to simultaneously learn the intra-modal and inter-modal semantic associations, and the generalization and robustness of the model can be improved, so that the model can generate a unified representation of multimodal data.

In general, a multimodal representation model can be trained based on a large number of sample tuples in various application scenarios, and thus the trained multimodal representation model is applicable to multiple application scenarios but is a preliminary training model that does not correspond to a specific downstream task.

A data modal is a physical form of data existence, such as text, image, video, audio, etc. The multimodal representation model of the embodiment of the present disclosure relates to multiple (two and more than two) data modalities, and its training samples (i.e., sample tuples) include data of multiple data modalities. In the embodiment of the present disclosure, by processing the sample tuples including multimodal data, semantic associations between multiple data modalities can be learned and a pre-trained multimodal representation model can be generated. Thus, the training method of the multimodal representation model of the embodiment of the present disclosure is a data processing method used to process multimodal data.

For example, if the multimodal representation model relates to an image modal and a text modal, the sample tuples include image data and text data, and the training method of the multimodal representation model is not only an image processing method but also a natural language processing method. If the multimodal representation model relates to an image modal and an audio modal, the sample tuples include image data and audio data, and the training method of the multimodal representation model is not only an image processing method but also an audio processing method.

In an embodiment of the present disclosure, the multimodal representation model includes a first representation module corresponding to a first data modal and a second representation module corresponding to a second data modal.

It should be noted that in this specification, a multimodal representation model including only two data modal representation modules is taken as an example to describe the training method of the multimodal representation model of the embodiment of the present disclosure. It should be understood by those skilled in the art that the training method of the multimodal representation model of the embodiment of the present disclosure also applies to situations where the multimodal representation model includes three or more data modal representation modules.

In an embodiment of the present disclosure, the first data modal and the second data modal are any two different data modals.

According to some embodiments, the first data modal and the second data modal may be any two of a text modal, an image modal, a video modal, and an audio modal. For example, the first data modal may be a text modal (or an image modal) and the second data modal may be an image modal (or a text modal).

In another embodiment, the first data modal or the second data modal may be a data modal in an industrial scene, such as an electrical signal, an infrared signal, sensor data, etc.

The first representation module is used to generate a representation of the data by encoding (or encoding and decoding) the data of a first data modal. The second representation module is used to generate a representation of the data by encoding (or encoding and decoding) the data of a second data modal. In embodiments of the present disclosure, the representation of the data may be in any form, such as a vector, a matrix, a binary code, etc.

According to some embodiments, the first representation module and the second representation module may be two modules independent of each other and do not influence each other when generating the representation of the data of the corresponding data modal. For example, in an embodiment as shown in FIG. 2, the multimodal representation model 200 includes a first representation module 210 and a second representation module 220, and the first representation module 210 and the second representation module 220 are independent of each other.

According to another embodiment, the first representation module and the second representation module may be two modules having an interaction relationship, and are influenced by each other when generating a representation of data of the corresponding data modal. For example, the first representation module and the second representation module may realize the interaction via a cross-attention mechanism.

The first and second representation modules may be realized in any structure, and the structures of the first and second representation modules may be different. For example, the first and second representation modules may be realized as neural networks with different structures, such as convolutional neural networks (CNNs), transformers, encoders, and encoder-decoders.

According to some embodiments, to improve the training efficiency of the multimodal representation model, the first representation module and the second representation module may use pre-trained models of the corresponding data modal. For example, the first data modal may be text, and thus the first representation module may use a pre-trained text representation model, such as ERNIE (Textual Mind Model), BERT (Bidirectional Encoder Representations from Transformers), etc. The second data modal may be image, and thus the second representation module may use a pre-trained visual representation model, such as ViT (Vision Transformer), DeiT (Data-efficient image Transformers), etc.

In an embodiment of the present disclosure, a multimodal representation model is trained using sample tuples. Each sample tuple includes a first sample and a second sample of a first data modal, and a third sample and a fourth sample of a second data modal. The first sample, the second sample, the third sample, and the fourth sample are semantically matched.

According to some embodiments, the sample tuples may be generated based on a raw data pair with matching meaning. That is, the training method 100 of the embodiment of the present disclosure may further include the steps of generating the sample tuples by obtaining a raw data pair, the raw data pair including first raw data of a first data modal and second raw data of a second data modal, the first raw data and the second raw data having matching meaning. For example, the first raw data may be an image, and the second raw data may be text describing the image. Then, based on the first raw data, a first sample and a second sample may be generated. Based on the second raw data, a third sample and a fourth sample may be generated.

According to the above embodiment, sample tuples can be automatically generated based on raw data pairs with matching cross-modal semantics, without the need for additional data annotation processing, thereby improving the efficiency of obtaining training samples (i.e., sample tuples).

It should be understood that in the process of generating sample tuples based on the above embodiment, the processing on the raw data does not change (or only slightly changes) the meaning of the raw data, i.e., the meaning of the generated first and second samples is approximately the same as the first raw data, and the meaning of the generated third and fourth samples is approximately the same as the second raw data. Since the meanings of the first and second raw data match, the meanings of the four generated samples, i.e., the first, second, third and fourth samples, also match each other.

According to some embodiments, the step of "generating a first sample and a second sample based on the first raw data" includes performing a first data extension process on the first raw data to generate a first sample, and performing a second data extension process on the first raw data to generate a second sample different from the first sample.

Data Augmentation processing involves processing raw data (e.g., the first raw data) to generate new data (e.g., the first sample and the second sample) that have a similar meaning to the raw data, under the premise that the meaning of the raw data (e.g., the first raw data) is guaranteed not to change.

The specific processing methods of the first data augmentation process and the second data augmentation process are determined based on the modality of the first raw data, i.e., the first data modality. For example, if the first data modality is an image, the first data augmentation process and the second data augmentation process may be processes such as adding random noise, rotating, cropping, masking, and discoloring. If the first data modality is text, the first data augmentation process and the second data augmentation process may be processes such as replacing synonyms, random insertion, random exchange, random deletion, and back-translation (translating an original text into text in another language using a machine translation model, and then back-translating it back into the original language using a machine translation model). If the first data modality is audio, the first data augmentation process and the second data augmentation process may be processes such as adding random noise, changing the speed of sound, and changing the volume.

The first data augmentation process may be a similar data augmentation method to the second data augmentation process, or may be a different data augmentation method, but the first data augmentation process and the second data augmentation process should cause the generated first sample to differ from the second sample, thereby causing the subsequently generated first representation to differ from the second representation, thereby improving the learning effect of the multimodal representation model.

According to another embodiment, both the first sample and the second sample are the same as the first raw data, and the first representation module is configured to make the generated first representation different from the second representation by adding random noise. The first representation module can for example be a neural network model that adds random noise.

For example, the first data modal may be a text modal and the first representation module may be a text encoder including a dropout processing layer whose drop probability is less than a pre-defined threshold (e.g., 20%), thereby ensuring that the first representation module does not change the meaning of its input data.

The execution step of "generating a third sample and a fourth sample based on the second raw data" is similar to the execution step of "generating a first sample and a second sample based on the first raw data" described above.

Specifically, according to some embodiments, "generating a third sample and a fourth sample based on the second raw data" may include performing a third data augmentation process on the second raw data to generate the third sample, and performing a fourth data augmentation process on the second raw data to generate the fourth sample. The third sample is different from the fourth sample, such that the subsequently generated third representation and the fourth representation are different, thereby improving the learning effect of the multimodal representation model.

According to another embodiment, the third and fourth samples are the same as the second raw data and the second representation module is configured to make the third representation different from the fourth representation by adding random noise. The second data modal may for example be a text modal and the second representation module may be a text encoder including a dropout processing layer, the drop probability of which is less than a pre-defined threshold (for example 20%), thereby ensuring that the first representation module does not change the meaning of its input data.

For other details of "generating a third sample and a fourth sample based on the second raw data", you may refer to the explanation of "generating a first sample and a second sample based on the first raw data" above, and detailed explanation will be omitted here.

The first and second samples are input to a first representation module, and the third and fourth samples are input to a second representation module, to obtain four corresponding representations: a first representation, a second representation, a third representation, and a fourth representation.

A first unimodal loss for the first data modal may be calculated based on the first representation and the second representation. A second unimodal loss for the second data modal may be calculated based on the third representation and the fourth representation. A first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal may be calculated based on the first target representation and the second target representation. Here, the first target representation is one of the first representation and the second representation, and the second target representation is one of the third representation and the fourth representation.

According to some embodiments, the first single-modal loss can be determined by the following steps: Determine a first single-modal loss based on a first similarity between the first representation and the second representation and a second similarity between the first representation and the second representation of another sample tuple, where the first single-modal loss is negatively correlated with the first similarity and positively correlated with the second similarity. The first similarity and the second similarity may be, for example, a dot product of the two representations.

According to the above embodiment, the first single-modal loss can make the representations (i.e., the first representation and the second representation) of two data (i.e., the first sample and the second sample) that match in meaning similar, and guide the multimodal representation model to learn semantic associations in the first data modal by making the representations (i.e., the first representation and the second representation) of two data (i.e., the first sample and the second sample of another sample tuple) that do not match in meaning similar (i.e., the first representation and the second representation of another sample tuple).

Specifically, the first single-modal loss can be calculated by the following formula (1).

In equation (1), _L1 denotes the first single-modal loss, N denotes the number of sample tuples participating in training per batch (i.e., batchsize ⁾ , _h1i ^and _h2i denote the first and second representations of the i-th sample tuple, respectively, _h2j denotes the second representation of the j-th ^sample tuple, and τ is a trainable parameter.

According to some embodiments, the second single-modal loss can be determined by the following steps: Determine a second single-modal loss based on a sixth similarity between the third representation and the fourth representation and a seventh similarity between the third representation and the fourth representation of another sample tuple, where the second single-modal loss is negatively correlated with the sixth similarity and positively correlated with the seventh similarity. The sixth similarity and the seventh similarity may be, for example, a dot product of the two representations.

According to the above embodiment, the second single-modal loss can make the representations (i.e., the third representation and the fourth representation) of two data (i.e., the third sample and the fourth sample) that match in meaning similar, and guide the multimodal representation model to learn semantic associations in the second data modal by making the representations (i.e., the third representation and the fourth representation) of two data (i.e., the third sample and the fourth sample of another sample tuple) that do not match in meaning similar (i.e., the third representation and the fourth representation of another sample tuple).

Specifically, the second single-modal loss can be calculated by the following formula (2).

^In equation (2), _L2 denotes the second single-modal loss, N denotes the number of sample tuples participating in training per batch (i.e., batchsize ⁾ , _h3i ^and _h4i denote the third and fourth representations of the i-th sample tuple, respectively, _h4j denotes the fourth representation of the j-th sample tuple, and τ is a trainable parameter.

According to some embodiments, the first cross-modal loss can be determined by the following steps: Determine a first cross-modal loss based on a third similarity between the first target representation and the second target representation and a fourth similarity between the first target representation and the second target representation of another sample tuple, where the first cross-modal loss is negatively correlated with the third similarity and positively correlated with the fourth similarity. The third similarity and the fourth similarity may be, for example, a dot product of the two representations.

According to the above embodiment, the first cross-modal loss can make two semantically matching data representations (i.e., the first target representation and the second target representation) similar, and guide the multi-modal representation model to learn cross-modal semantic associations from the first data modal to the second data modal by making two semantically inconsistent data representations (i.e., the first target representation and the second target representation of another sample tuple) dissimilar.

Specifically, taking the first target representation as an example, the first target representation is the second representation, and the second target representation is the third representation, the first cross-modal loss can be calculated by the following equation (3):

^In equation (3), _L3 denotes the first cross-modal loss, N denotes the number of sample tuples participating in training per batch (i.e., batchsize), _h2i ^and _h3i denote the second representation (i.e., the first target representation) and the third representation (i.e., the second target representation) of the i-th sample tuple, respectively ^, _h3j denotes the third representation (i.e., the second target representation) of the j-th sample tuple, and τ is a trainable parameter.

According to some embodiments, the second cross-modal loss can be determined by the following steps: Determine a second cross-modal loss based on a third similarity between the first target representation and the second target representation and a fifth similarity between the second target representation and the first target representation of another sample tuple, where the second cross-modal loss is negatively correlated with the third similarity and positively correlated with the fifth similarity. The third similarity and the fifth similarity may be, for example, a dot product of the two representations.

According to the above embodiment, the second cross-modal loss can make two semantically matching data representations (i.e., the first target representation and the second target representation) similar, and guide the multi-modal representation model to learn cross-modal semantic associations from the second data modal to the first data modal by making two semantically inconsistent data representations (i.e., the second target representation and the first target representation of another sample tuple) dissimilar.

Specifically, taking the first target representation as an example, the second representation and the second target representation as a third representation, the second cross-modal loss can be calculated by the following equation (4):

^In equation (4), _L4 denotes the second cross-modal loss, N denotes the number of sample tuples participating in training per batch (i.e., batchsize), _h2i ^and _h3i denote the second representation (i.e., the first ^target representation) and the third representation (i.e., the second target representation) of the i-th sample tuple, respectively, _h2j denotes the second representation (i.e., the first target representation) of the j-th sample tuple, and τ is a trainable parameter.

Based on the first single-modal loss _L1 , the second single-modal loss _L2 , the first cross-modal loss _L3 , and the second cross-modal loss _L4 , an overall loss L of the multi-modal representation model can be determined.

According to some embodiments, the overall loss L of the multimodal representation model is a weighted sum of a first single-modal loss _L1 , a second single-modal loss _L2 , a first cross-modal loss _L3 , and a second cross-modal loss _L4 , i.e.,

In formula (5), w _i is the weight of the i-th loss L _i . By flexibly setting the value of w _i , the learning direction of the multimodal representation model can be flexibly adjusted, and the compatibility of the multimodal representation model with practical application scenarios can be improved.

After determining the overall loss of the multimodal representation model, the parameters of the multimodal representation model can be adjusted based on the overall loss.

It should be understood that steps S110 to S180 may be repeated multiple times until a predetermined termination condition is met (e.g., the total loss is less than a predetermined value, the number of iterations reaches a predetermined maximum number of iterations, etc.), thereby terminating the model training process and obtaining a pre-trained multimodal representation model.

As mentioned above, the first data modal and the second data modal may be any two of a text modal, an image modal, a video modal, and an audio modal.

For example, the first data modal may be an image modal, and the second data modal may be a text modal. Thus, the multimodal representation model is an image or text multimodal representation model, which includes an image representation module (i.e., a first representation module) corresponding to the image modal and a text representation module (i.e., a second representation module) corresponding to the text modal, and the training method for the image or text multimodal representation model includes the following steps S110' to S180'.

In step S110', an image or text sample tuple is obtained, and the image or text sample tuple includes a first image sample (i.e., the first sample) and a second image sample (i.e., the second sample) in the image modal, and a first text sample (i.e., the third sample) and a second text sample (i.e., the fourth sample) in the text modal. The meanings of the samples included in the image or text sample tuple match.

In step S120', the first image sample and the second image sample are input into an image representation module, respectively, to obtain a first image representation (i.e., a first representation) of the first image sample and a second image representation (i.e., a second representation) of the second image sample.

In step S130', the first text sample and the second text sample are input into a text representation module to obtain a first text representation (i.e., a third representation) of the first text sample and a second text representation (i.e., a fourth representation) of the second text sample.

In step S140', a first single-modal loss for the image modal is determined based on the first image representation and the second image representation.

In step S150', a second single-modal loss for the text modal is determined based on the first text representation and the second text representation.

In step S160', a first cross-modal loss from image modal to text modal and a second cross-modal loss from text modal to image modal are determined based on a target image representation (i.e., a first target representation) and a target text representation (i.e., a second target representation), where the target image representation is one of the first image representation and the second image representation, and the target text representation is one of the first text representation and the second text representation.

In step S170', an overall loss of the multimodal representation model is determined based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss.

In step S180', the parameters of the multimodal representation model are adjusted based on the overall loss.

According to the above embodiment, an image and text sample tuple is used to train an image and text multimodal representation model, where the image and text sample tuple includes two image samples in an image modal and two text samples in a text modal, and the meanings of the samples in the image and text sample tuple match. In the model training process, not only the single-modal loss of the image modal and the text modal are calculated, but also the cross-modal loss between the image and text modal is calculated, so that the image and text multimodal representation model can be guided to simultaneously learn the intra-modal and inter-modal semantic associations, and the generalization and robustness of the model can be improved, so that the model can generate a unified representation of the image and text multimodal data. Based on the unified representation, the accuracy of downstream image and text cross-modal tasks (e.g., image and text cross-modal search, image and text semantic matching, etc.) can be improved.

It should be understood that steps S110' to S180' correspond to steps S110 to S180 described above. The embodiment of steps S110' to S180' may refer to the description of steps S110 to S180 above, and a detailed description will be omitted here.

Figure 3 is a schematic diagram illustrating the training process of a multimodal image and text representation model according to some embodiments of the present disclosure. As shown in Figure 3, first, an image 331 and a text 332 are obtained, and the meanings of the image 331 and the text 332 are matched.

A data extension process is performed twice on image 331 without changing its meaning to obtain images 341 and 342. A data extension process is performed twice on text 332 without changing its meaning to obtain text 343 and text 344.

Images 341 and 342 are input to image representation module 310, and image representations 351 and 352 are obtained. Text 343 and text 344 are input to text representation module 320, and text representations 353 and 354 are obtained.

Based on the image representation 351 and the image representation 352, a single-modal loss 361 for the image modal can be calculated. Based on the text representation 353 and the text representation 354, a single-modal loss 362 for the text modal can be calculated. Based on the image representation 352 and the text representation 353, a cross-modal loss 363 from the image modal to the text modal and a cross-modal loss 364 from the text modal to the image modal can be calculated.

The overall loss 370 is obtained by performing a weighted sum of the single-modal loss 361, the single-modal loss 362, the cross-modal loss 363, and the cross-modal loss 364. Based on the overall loss 370, the parameters of the image representation module 310 and the text representation module 320 are adjusted using an algorithm such as backpropagation.

According to some embodiments, a pre-trained multimodal expression model is obtained based on steps S110 to S180, and the multimodal expression model is fine-tuned based on matching data pairs in a pre-set application scenario (e.g., image and text matching pairs in an image and text search scenario), thereby improving the expression effect of the multimodal expression model in a specific application scenario.

Specifically, according to some embodiments, the matching data pair includes first data and second data of a target data modal, the meanings of the first data and the second data match, and "fine-tuning the multimodal expression model based on the matching data pair in a pre-defined application scenario" includes fine-tuning the expression module corresponding to the target data modal based on the matching data pair.

According to another embodiment, the matching data pair includes first data of a first data modal and second data of a second data modal, the meanings of the first data and the second data match, and "performing fine-tuning on the multimodal representation model based on the matching data pair in the preset application scenario" includes generating a first data sample and a second data sample of the first data modal based on the first data, generating a third data sample and a fourth data sample of the second data modal based on the second data, constructing a data sample tuple based on the first data sample, the second data sample, the third data sample, and the fourth data sample, and performing fine-tuning on the multimodal representation model based on the data sample tuple.

The embodiment of fine-tuning the multimodal representation model based on data sample tuples is similar to the embodiment of training the multimodal representation model based on sample tuples in steps S110 to S180, and will not be described in detail here.

The multimodal representation model training method 100 according to an embodiment of the present disclosure can obtain a pre-trained (or pre-trained and fine-tuned) multimodal representation model. The pre-trained (or pre-trained and fine-tuned) multimodal representation model may be applied to multiple cross-modal data processing tasks, such as cross-modal search tasks, cross-modal inference tasks, etc. It should be understood that the pre-trained (or pre-trained and fine-tuned) multimodal representation model may also be applied to single-modal data processing tasks.

According to an embodiment of the present disclosure, a cross-modal search method is further provided. FIG. 4 is a flowchart illustrating a cross-modal search method 400 according to an embodiment of the present disclosure. The method 400 may be executed on a server or on a client device. That is, each step of the method 400 may be executed by a server or a client device.

As shown in FIG. 4, method 400 includes steps S410 to S440.

In step S410, the search target of the first data modal is obtained.

In step S420, the search target is input to a multimodal expression model to obtain a first expression corresponding to the search target. The multimodal expression model is obtained by training using the multimodal expression model training method according to the above embodiment.

In step S430, a second representation of each of the plurality of candidate objects of the second data modal is obtained, where the second representation is obtained by inputting the corresponding candidate object into a multimodal representation model.

In step 440, a target object that matches the search object is determined from the plurality of candidate objects based on the similarity between the first expression and each of the second expressions.

According to the embodiments of the present disclosure, accurate cross-modal search can be achieved.

According to some embodiments, the first data modal and the second data modal may be an image modal and a text modal, and thus, the method 400 can achieve cross-modal search for accurate images and text.

The similarity between the first and second representations may be, for example, cosine similarity.

According to some embodiments, the target object may be, for example, one or more candidate objects with the greatest similarity. According to other embodiments, the target object may be one or more candidate objects with a similarity greater than a threshold.

5 is a comparison diagram of the cross-modal retrieval effect of the multimodal representation model of the embodiment of the present disclosure (i.e., the present model) and the multimodal representation model of the prior art on images and texts in the public datasets Flickr30K and MSCOCO, where R@K indicates the proportion of all sample instances in which the correct answer appears in the previous K return results, and ^* indicates that the model is fine-tuned on the MSCOCO dataset.

According to an embodiment of the present disclosure, there is further provided a cross-modal inference method. FIG. 6 is a flowchart illustrating a cross-modal inference method 600 according to an embodiment of the present disclosure. The method 600 may be executed, for example, on a server or on a client device. That is, each step of the method 600 may be executed by a server or a client device.

As shown in FIG. 6, method 600 includes steps S610 to S630.

In step S610, a first target of the first data modal and a second target of the second data modal are obtained.

In step S620, the first object and the second object are input into the multimodal representation model, respectively, to obtain a first representation corresponding to the first object and a second representation corresponding to the second object. The multimodal representation model is obtained by training using the multimodal representation model training method according to the above embodiment.

In step S630, inference tags for the first object and the second object are determined based on the similarity between the first expression and the second expression.

According to the embodiments of the present disclosure, accurate cross-modal inference can be achieved.

According to some embodiments, inference tags include, for example, Entailment, Neutral, and Contradiction.

According to an embodiment of the present disclosure, there is further provided an apparatus for training a multi-modal expression model. Figure 7 is a block diagram of an apparatus 700 for training a multi-modal expression model according to an embodiment of the present disclosure. As shown in Figure 7, the apparatus 700 includes:
a first acquiring unit 710 configured to acquire a sample tuple, the sample tuple including a first sample and a second sample of the first data modal, and a third sample and a fourth sample of the second data modal, where the meanings of the samples included in the sample tuple match;
a first input unit 720 configured to input the first sample and the second sample to the first representation unit, respectively, to obtain a first representation of the first sample and a second representation of the second sample;
a second input unit 730 configured to input the third sample and the fourth sample to the second representation unit, respectively, to obtain a third representation of the third sample and a fourth representation of the fourth sample;
a first loss unit 740 configured to determine a first unimodal loss for the first data modal based on the first representation and the second representation;
a second loss unit 750 configured to determine a second unimodal loss for the second data modal based on the third representation and the fourth representation;
a third loss unit 760 configured to determine a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal based on a first target representation and a second target representation, where the first target representation is any one of the first representation and the second representation, and the second target representation is any one of the third representation and the fourth representation;
a fourth loss unit 770 configured to determine an overall loss of the multimodal representation model based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss;
An adjustment unit 780 configured to adjust parameters of the multi-modal representation model based on the overall loss.

According to an embodiment of the present disclosure, a sample tuple is used to train a multimodal representation model, where the sample tuple includes two samples of different data modalities, and the meanings of each sample in the sample tuple are matched. Based on the sample tuple, during the training process, not only single-modal losses for each data modal are calculated, but also cross-modal losses between different modalities are calculated, which can guide the multimodal representation model to simultaneously learn intra-modal and inter-modal semantic associations, thereby improving the generalizability and robustness of the model, and thus enabling the model to generate a unified representation of multimodal data.

According to some embodiments, the device 700 further includes a second acquisition unit configured to acquire a raw data pair, the raw data pair including first raw data of the first data modal and second raw data of the second data modal, the first raw data and the second raw data having matching meanings, a first generation unit configured to generate the first sample and the second sample based on the first raw data, and a second generation unit configured to generate the third sample and the fourth sample based on the second raw data.

According to some embodiments, the first generation unit includes a first extension subunit configured to generate the first sample by performing a first data extension process on the first raw data, and a second extension subunit configured to generate the second sample different from the first sample by performing a second data extension process on the first raw data.

According to some embodiments, the first sample and the second sample are the same as the first raw data, and the first representation unit is configured to make the first representation different from the second representation by adding random noise.

According to some embodiments, the first data modal is a text modal and the first representation unit is a text encoder including a dropout processing layer, the drop probability of the dropout processing layer being less than a pre-set threshold.

According to some embodiments, the first loss unit 740 is further configured to determine the first single-modal loss based on a first similarity between the first representation and the second representation and a second similarity between the first representation and a second representation of another sample tuple, where the first single-modal loss is negatively correlated with the first similarity and positively correlated with the second similarity.

According to some embodiments, the third loss unit 760 is further configured to determine the first cross-modal loss based on a third similarity between the first target representation and the second target representation and a fourth similarity between the first target representation and a second target representation of another sample tuple, where the first cross-modal loss is negatively correlated with the third similarity and positively correlated with the fourth similarity.

According to some embodiments, the third loss unit 760 is further configured to determine the second cross-modal loss based on a third similarity between the first target representation and the second target representation and a fifth similarity between the second target representation and the first target representation of another sample tuple, where the second cross-modal loss is negatively correlated with the third similarity and positively correlated with the fifth similarity.

According to some embodiments, the overall loss is a weighted sum of the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss.

According to some embodiments, the device 700 further comprises a fine-tuning unit configured to perform fine-tuning on the multimodal representation model based on matching data pairs in a predefined application scenario.

According to some embodiments, the matching data pair comprises first data and second data of a target data modal, the meanings of the first data and the second data matching, and wherein the fine-tuning unit is further configured to perform fine-tuning on a representation unit corresponding to the target data modal based on the matching data pair.

According to some embodiments, the matching data pair includes first data of a first data modal and second data of a second data modal, the meanings of the first data and the second data matching, and the fine-tuning unit includes a first generation subunit configured to generate a first data sample and a second data sample of the first data modal based on the first data, a second generation subunit configured to generate a third data sample and a fourth data sample of the second data modal based on the second data, a third generation subunit configured to construct a data sample tuple based on the first data sample, the second data sample, the third data sample, and the fourth data sample, and a fine-tuning subunit configured to perform fine-tuning on the multimodal representation model based on the data sample tuple.

According to some embodiments, the first data modal and the second data modal are any two of a text modal, an image modal, a video modal, and an audio modal.

According to an embodiment of the present disclosure, there is further provided a cross-modal search device. Fig. 8 is a block diagram showing a cross-modal search device 800 according to an embodiment of the present disclosure. As shown in Fig. 8, the device 800 includes:
A first acquisition unit 810 configured to acquire a search object of a first data modality;
a representation unit 820 configured to input the search subject into a multimodal representation model to obtain a first representation corresponding to the search subject;
a second acquisition unit 830 configured to acquire a second representation of each of a plurality of candidate objects of a second data modality, where the second representation is obtained by inputting the corresponding candidate object into the multimodal representation model;
a determination unit 840 configured to determine a target object matching the search object from the plurality of candidate objects based on a similarity between the first representation and each second representation, where the multimodal representation model is obtained by training with a training device based on the multimodal representation model.

According to the embodiments of the present disclosure, accurate cross-modal search can be achieved.

According to an embodiment of the present disclosure, there is further provided a cross-modal reasoning apparatus. Fig. 9 is a block diagram showing a cross-modal reasoning apparatus 900 according to an embodiment of the present disclosure. As shown in Fig. 9, the apparatus 900 includes:
an acquisition unit 910 configured to acquire a first object of a first data modal and a second object of a second data modal;
a representation unit 920 configured to input the first object and the second object into a multimodal representation model, respectively, to obtain a first representation corresponding to the first object and a second representation corresponding to the second object;
a determining unit 930 configured to determine inferred tags for the first object and the second object based on a similarity between the first representation and the second representation, where the multimodal representation model includes a multimodal representation model obtained by training with a training device based on the multimodal representation model.

According to the embodiments of the present disclosure, accurate cross-modal inference can be achieved.

It should be understood that, as shown in FIG. 7, each unit of the apparatus 700 may correspond to each step in the method 100 in FIG. 1, as shown in FIG. 8, each unit of the apparatus 800 may correspond to each step in the method 400 in FIG. 4, and as shown in FIG. 9, each unit of the apparatus 900 may correspond to each step in the method 600 in FIG. 6. Thus, the operations, features and advantages described for the method 100 above also apply to the apparatus 700 and the units included therein, the operations, features and advantages described for the method 400 above also apply to the apparatus 800 and the units included therein, and the operations, features and advantages described for the method 600 above also apply to the apparatus 900 and the units included therein. For the sake of brevity, some operations, features and advantages are not described here.

Although certain functionality is discussed above with reference to specific units, it should be noted that the functionality of each unit discussed herein may be split into multiple units and/or at least some functionality of multiple units may be combined into a single unit. For example, the first input unit 720 and the second input unit 730 described above may, in some embodiments, be combined into a single unit.

It should also be understood that various techniques may be described herein in the general context of software hardware elements or program units. With respect to each unit described in Figures 7-9 above, it may be implemented in hardware or in hardware in combination with software and/or firmware. For example, these units may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of units 710-930 may be implemented together in a System on Chip (SoC). An SoC may include integrated circuit chips (e.g., a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or one or more components in other circuits) that can optionally execute received program code and/or include embedded firmware to perform functions.

According to an embodiment of the present disclosure, there is provided an electronic device, the electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform at least one of a multimodal representation model training method, a cross-modal search method, and a cross-modal inference method according to an embodiment of the present disclosure.

According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to execute at least one of a multimodal representation model training method, a cross-modal search method, and a cross-modal inference method according to an embodiment of the present disclosure.

According to one aspect of the present disclosure, a computer program product is provided that includes a computer program that, when executed by a processor, realizes at least one of a multimodal representation model training method, a cross-modal search method, and a cross-modal inference method according to an embodiment of the present disclosure.

As shown in FIG. 10, a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure, which is an example of a hardware device applicable to various aspects of the present disclosure, is described here. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown in this specification are merely exemplary and do not limit the implementation of the present disclosure as described and/or claimed herein.

As shown in FIG. 10, the electronic device 1000 includes a computing unit 1001, which can perform various suitable operations and processes by computer programs stored in a read-only memory (ROM) 1002 or loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may further store various programs and data required to operate the device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to the bus 1004.

The components of the electronic device 1000 are connected to an I/O interface 1005, and include an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 may be any type of device capable of inputting information to the device 1000, and the input unit 1006 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackboard, a trackball, a control lever, a microphone, and/or a remote control. The output unit 1007 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 1008 may include, but is not limited to, a magnetic disk, an optical disk. The communications unit 1009 enables the device 1000 to exchange information/data with other devices via a computer network, e.g., the Internet, and/or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver, and/or a chipset, e.g., a Bluetooth ^™ device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communications device, and/or the like.

The computing unit 1001 may be any of a variety of general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 may include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes each of the methods and processes described above, such as methods 100, 400, and 600. For example, in some embodiments, the methods 100, 400, and 600 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, some or all of the computer program may be loaded and/or installed into the device 1000 via the ROM 1002 and/or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, it can perform one or more steps of the methods 100, 400 or 600 described above. Alternatively, in other embodiments, the computing unit 1001 is configured to perform at least one of the methods 100, 400, 600 in any other suitable manner (e.g., using firmware).

Various embodiments of the systems and techniques described herein above may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), software hardware, firmware, software, and/or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and/or interpreted by a programmable system that includes at least one programmable processor, which may be a dedicated or general purpose programmable processor, and may include receiving data and instructions from a storage system, at least one input device, at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

Program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus such that when executed by the processor or controller, the program code performs the functions/operations specified in the flowcharts and/or block diagrams. The program code may be fully executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or fully executed on a remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include portable computer disks , hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), portable compact disk read-only memories (CD-ROMs) , optical storage devices, magnetic storage devices, or any suitable combination of the above.

To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and may receive input from the user in any form (including sound input, speech input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes a backstage component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added, or removed using the various forms of flow described above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto so long as the technical solutions disclosed in this disclosure achieve the desired results.

Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely illustrative embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims (28)

1. A computer-implemented training method for a multi-modal representation model, the multi-modal representation model including a first representation module corresponding to a first data modality and a second representation module corresponding to a second data modality, the training method comprising:
obtaining a sample tuple, the sample tuple including a first sample and a second sample of the first data modal, and a third sample and a fourth sample of the second data modal, the meanings of the samples included in the sample tuple being consistent ;
inputting the first sample and the second sample into the first representation module, respectively, to obtain a first representation of the first sample and a second representation of the second sample;
inputting the third sample and the fourth sample into the second representation module, respectively, to obtain a third representation of the third sample and a fourth representation of the fourth sample;
determining a first unimodal loss for the first data modal based on a first similarity between the first representation and the second representation and a second similarity between the first representation and a second representation of another sample tuple , the first unimodal loss being negatively correlated with the first similarity and positively correlated with the second similarity ;
determining a second single-modal loss for the second data modal based on a sixth similarity between the third representation and the fourth representation and a seventh similarity between the third representation and a fourth representation of another sample tuple , the second single-modal loss being negatively correlated with the sixth similarity and positively correlated with the seventh similarity;
Based on a first target representation and a second target representation, a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal are determined, the first target representation being one of the first representation and the second representation, the second target representation being one of the third representation and the fourth representation , a third similarity between the first target representation and the second target representation, and a third similarity between the first target representation and the other sample data. determining the first cross-modal loss based on a fourth similarity between the first target representation and the second target representation of another sample tuple, the first cross-modal loss being negatively correlated with the third similarity and positively correlated with the fourth similarity; determining the second cross-modal loss based on a third similarity between the first target representation and the second target representation and a fifth similarity between the second target representation and a first target representation of another sample tuple, the second cross-modal loss being negatively correlated with the third similarity and positively correlated with the fifth similarity ;
determining an overall loss of the multimodal representation model based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss;
and adjusting parameters of the multi-modal representation model to reduce the overall loss.
method. obtaining raw data pairs, the raw data pairs including first raw data of the first data modal and second raw data of the second data modal, the first raw data and the second raw data being semantically consistent ;
generating the first sample and the second sample based on the first raw data;
The method of claim 1 , further comprising generating the third sample and the fourth sample based on the second raw data. generating the first sample and the second sample based on the first raw data,
performing a first data augmentation process on the first raw data to generate the first samples;
3. The method of claim 2, further comprising: performing a second data augmentation operation on the first raw data to generate the second samples, the second samples being different from the first samples. The method of claim 2, wherein the first sample and the second sample are the same as the first raw data, and the first representation module is configured to make the first representation different from the second representation by adding random noise. 5. The method of claim 4, wherein the first data modal is a text modal and the first representation module is a text encoder including a dropout processing layer, the drop probability of the dropout processing layer being less than a preset threshold. The method of claim 1, wherein the overall loss is a weighted sum of the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss. The method of claim 1, further comprising fine-tuning the multimodal representation model based on matching data pairs in a preset application scenario. The matching data pair includes a first data and a second data of a target data modality, and the meaning of the first data and the meaning of the second data are consistent . According to the matching data pair in a preset application scenario, fine-tuning the multi-modal expression model is performed.
The method of claim 7 , further comprising: making refinements to a representation module corresponding to the target data modal based on the matching data pairs. The matching data pair includes first data of a first data modality and second data of a second data modality, and the first data and the second data have the same meaning. The matching data pair in a preset application scenario is used to fine-tune the multimodal expression model.
generating a first data sample and a second data sample of the first data modal based on the first data;
generating a third data sample and a fourth data sample of the second data modality based on the second data;
constructing a data sample tuple based on the first data sample, the second data sample, the third data sample, and the fourth data sample;
and performing fine - tuning on the multi-modal representation model based on the data sample tuples. The method of claim 1, wherein the first data modal and the second data modal are two of a text modal, an image modal, a video modal, and an audio modal. 1. A computerized cross-modal search method, comprising:
Obtaining a search target of a first data modal;
inputting the search subject into a multimodal representation model to obtain a first representation corresponding to the search subject;
obtaining a second representation of each of a plurality of candidate objects of a second data modality, the second representation being obtained by inputting the corresponding candidate object into the multimodal representation model;
determining a target object that matches the search object from the plurality of candidate objects based on a similarity between the first representation and each of the second representations;
2. A method for cross-modal search, wherein the multi-modal representation model is obtained by training using the method of claim 1. 1. An apparatus for training a multimodal representation model, the multimodal representation model comprising a first representation unit corresponding to a first data modality and a second representation unit corresponding to a second data modality, the training apparatus comprising:
a first acquiring unit configured to acquire a sample tuple, the sample tuple including a first sample and a second sample of the first data modal, and a third sample and a fourth sample of the second data modal, the meanings of the samples included in the sample tuple being consistent ;
a first input unit configured to input the first sample and the second sample to the first representation unit, respectively, to obtain a first representation of the first sample and a second representation of the second sample;
a second input unit configured to input the third sample and the fourth sample to the second representation unit, respectively, to obtain a third representation of the third sample and a fourth representation of the fourth sample;
a first loss unit configured to determine a first unimodal loss of the first data modal based on a first similarity between the first representation and the second representation and a second similarity between the first representation and a second representation of another sample tuple , the first unimodal loss being negatively correlated with the first similarity and positively correlated with the second similarity ;
a second loss unit configured to determine a second unimodal loss of the second data modal based on a sixth similarity between the third representation and the fourth representation and a seventh similarity between the third representation and a fourth representation of another sample tuple , the second unimodal loss being negatively correlated with the sixth similarity and positively correlated with the seventh similarity ;
a third loss unit configured to determine a first cross-modal loss from the first data modal to the second data modal and a second cross-modal loss from the second data modal to the first data modal based on a first target representation and a second target representation, the first target representation being any one of the first representation and the second representation, the second target representation being any one of the third representation and the fourth representation, a third similarity between the first target representation and the second target representation, and a fourth similarity between the first target representation and the second target representation and a fourth similarity between the first target representation and the second target representation. determining the first cross-modal loss based on a fourth similarity between the first target representation and the second target representation of another sample tuple, the first cross-modal loss being negatively correlated with the third similarity and positively correlated with the fourth similarity; determining the second cross-modal loss based on a third similarity between the first target representation and the second target representation and a fifth similarity between the second target representation and the first target representation of another sample tuple, the second cross-modal loss being negatively correlated with the third similarity and positively correlated with the fifth similarity;
a fourth loss unit configured to determine an overall loss of the multimodal representation model based on the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss; and
an adjustment unit configured to adjust parameters of the multimodal representation model to reduce the overall loss;
Device. a second acquiring unit configured to acquire a raw data pair, the raw data pair including first raw data of the first data modal and second raw data of the second data modal, the first raw data and the second raw data being semantically consistent ;
a first generating unit configured to generate the first sample and the second sample based on the first raw data;
The apparatus of claim 12 , further comprising: a second generating unit configured to generate the third sample and the fourth sample based on the second raw data. The first generating unit is
a first enhancement subunit configured to perform a first data enhancement process on the first raw data to generate the first samples;
and a second enhancement subunit configured to perform a second data enhancement process on the first raw data to generate the second samples that differ from the first samples. 14. The apparatus of claim 13, wherein the first samples and the second samples are the same as the first raw data, and the first representation unit is configured to make the first representation different from the second representation by adding random noise. 16. The apparatus of claim 15 , wherein the first data modal is a text modal and the first representation unit is a text encoder including a dropout processing layer, the drop probability of the dropout processing layer being less than a pre-set threshold. 13. The apparatus of claim 12 , wherein the overall loss is a weighted sum of the first single-modal loss, the second single-modal loss, the first cross-modal loss, and the second cross-modal loss. The apparatus of claim 12 , further comprising a fine-tuning unit configured to perform fine-tuning on the multimodal representation model based on matching data pairs in a preset application scenario. The matching data pair includes first data and second data of a target data modal, and the first data and the second data have a matching meaning, and the fine-tuning unit further comprises:
The apparatus of claim 18 , configured to perform fine-tuning on a representation unit corresponding to the target data modal based on the matching data pairs. The matching data pair includes first data of a first data modal and second data of a second data modal, and the first data and the second data have a matching meaning, and the fine-tuning unit:
a first generating subunit configured to generate a first data sample and a second data sample of the first data modal based on the first data;
a second generating subunit configured to generate a third data sample and a fourth data sample of the second data modality based on the second data;
a third generating subunit configured to construct a data sample tuple based on the first data sample, the second data sample, the third data sample, and the fourth data sample;
and a fine-tuning subunit configured to perform fine-tuning on the multi-modal representation model based on data sample tuples . The apparatus of claim 12 , wherein the first data modal and the second data modal are any two of a text modal, an image modal, a video modal, and an audio modal. A cross-modal search device, comprising:
a first acquiring unit configured to acquire a search subject of a first data modality;
a representation unit configured to input the search subject into a multimodal representation model to obtain a first representation corresponding to the search subject;
a second acquisition unit configured to acquire a second representation of each of a plurality of candidate objects in a second data modality, the second representation being obtained by inputting the corresponding candidate object into the multimodal representation model;
a determining unit configured to determine a target object matching the search object from the plurality of candidate objects based on a similarity between the first representation and each second representation;
The multimodal representation model is obtained by training with the device according to claim 12 . An electronic device,
At least one processor;
a memory communicatively coupled to the at least one processor;
11. An electronic device comprising: an electronic device having stored therein instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 10 . An electronic device,
At least one processor;
a memory communicatively coupled to the at least one processor;
12. An electronic device comprising: an electronic device having a memory configured to store instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the method of claim 11. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to carry out the method of any one of claims 1 to 10 . A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of claim 11. A computer program which, when executed by a processor, causes the processor to carry out a method according to any one of claims 1 to 10 . A computer program which, when executed by a processor, causes the processor to perform the method of claim 11.