JP7581612B2 - Method, apparatus, electronic device and storage medium for training a semantic search model - Google Patents

Description

The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technologies such as machine learning and natural language processing, and in particular to methods, apparatus, electronic devices, and storage media for training semantic search models.

In the information age, people want to quickly find the information they need among a vast number of books, web pages, and documents. Recalling candidates from a large amount of data, and then re-sorting the recalled data to score its reliability, has become the mainstream pattern in information retrieval today.

In the recall stage of a search task, there are usually two different methods: sparse vector search and dense vector search. In the sparse vector search method, the query sentence and the candidate language materials are encoded into a sparse vector, and the dimension of this vector is generally roughly the same as that of a dictionary. This method mainly relies on the degree of character matching to perform similarity calculation. Common algorithms include BM25, but it is impossible to learn a semantic search model that corresponds to sparse vector search. This method has high transfer ability and is not limited to a specific field. In the dense vector search method, the query sentence and the candidate language materials are encoded into two vectors in the semantic space by the corresponding semantic search model, and the similarity is calculated based on the vector to recall the correlation result. In this method, it is necessary to train the semantic search model with training data, and the matching degree can be determined using semantic information, but the transfer ability is poor.

The present disclosure provides a method, apparatus, electronic device, and storage medium for training a semantic search model.

According to one aspect of the present disclosure, a method for training a semantic search model is provided, which includes obtaining a target query sentence type corresponding to each of at least two original semantic search models, the target query sentence type being the most accurate query sentence type when the original semantic search models process various types of query sentences, obtaining a distilled dataset based on the at least two original semantic search models, the target query sentence type corresponding to each of the original semantic search models, and a pre-constructed corpus, and training the target semantic search model based on the distilled dataset.

According to another aspect of the present disclosure, a training device for a semantic search model is provided, the training device including: a type acquisition module for acquiring a target query sentence type corresponding to each of at least two original semantic search models, the target query sentence type being the most accurate query sentence type when the original semantic search models process various types of query sentences; a data acquisition module for acquiring a distilled dataset based on the at least two original semantic search models and a pre-established corpus; and a training module for training the target semantic search model based on the distilled dataset.

According to yet another aspect of the present disclosure, there is provided an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing commands executable by the at least one processor, the commands, when executed by the at least one processor, causing the at least one processor to perform any of the methods of the above-mentioned aspects and possible embodiments.

According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform any of the methods of the above-described aspects and possible embodiments.

According to yet another aspect of the present disclosure, there is provided a computer program product including a computer program that, when executed by a processor, performs the method of any of the above aspects and possible embodiments.

According to the technology disclosed herein, the search capabilities of at least two original semantic search models can be integrated into a trained target semantic search model, thereby overcoming the shortcomings of a single semantic search model and improving the accuracy of semantic search.

It should be understood that the contents described in this section are not intended to identify key or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following specification.

The drawings are for better understanding of the technical solution and are not intended to limit the present disclosure.
FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure. FIG. 13 is a schematic diagram according to a third embodiment of the present disclosure. FIG. 1 is a configuration diagram of a method for training a semantic search model according to an embodiment of the present invention. FIG. 13 is a schematic diagram according to a fourth embodiment of the present disclosure. FIG. 13 is a schematic diagram according to a fifth embodiment of the present disclosure. FIG. 1 is a block diagram of an electronic device for implementing the method of an embodiment of the present disclosure.

Below, exemplary embodiments of the present disclosure will be described based on the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included and should be considered as merely examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of simplicity, the following description omits descriptions of well-known functions and structures.

Obviously, the described embodiments are only some of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments obtained based on the embodiments of the present disclosure under the premise that a person skilled in the art does not perform creative labor, fall within the scope of protection of the present disclosure.

It should be noted that the terminal device according to the embodiment of the present disclosure may include, but is not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablets. The display device may include, but is not limited to, devices with display capabilities such as personal computers and televisions.

Furthermore, the term "and/or" in this specification is simply a relational relationship describing related objects, and means that three relations can exist. For example, A and/or B can mean three situations: A exists alone, A and B exist simultaneously, and B exists alone. Also, the character "/" in this specification generally means that the related objects before and after it are in an "or" relationship.

In conventional technology, search methods based on sparse vectors and search methods based on dense vectors are generally used alone. Search methods based on sparse vectors can only model text matching, and lack semantic understanding of the content, making them less effective. On the other hand, search methods that use only dense vectors miss some text matching information. In other words, using either of the above search methods alone results in less accurate semantic search.

Figure 1 is a schematic diagram according to a first embodiment of the present disclosure. As shown in Figure 1, the present embodiment provides a method for training a semantic search model, and specifically includes the following steps:

In S101, a target query sentence type corresponding to each of at least two original semantic search models is obtained.

Here, the target query sentence type corresponding to the original semantic search model is the query sentence type that is most accurate when this original semantic search model processes various types of query sentences.

In S102, a distilled dataset is obtained based on at least two original semantic search models, target query sentence types corresponding to each original semantic search model, and a pre-constructed corpus.

At S103, a target semantic search model is trained based on the distilled dataset.

The entity performing the method for training a semantic search model of this embodiment may be a device for training a semantic search model. The device may be an electronic entity or an application that employs software integration and runs on a computing device when in use to realize training of the semantic search model.

The at least two semantic search models of this embodiment may include a sparse vector-based semantic search model and a dense vector-based semantic search model. Here, the sparse vector-based semantic search model may encode the literal semantic vector of any language material, such as the query sentence query and/or the candidate language material, based on a dictionary. On the other hand, the dense vector-based semantic search model realizes encoding the semantic vector of the query sentence query and the candidate language material at the semantic level based on a pre-trained neural network model. For example, the dense vector-based semantic search model of this embodiment may be realized based on the RocketQAv2, ColBERT, Phrase-BERT, or COIL model, etc.

That is, the at least two original semantic search models of this embodiment may include at least two of the BM25 model, RocketQAv2, ColBERT, Phrase-BERT, and COIL model, etc. In addition, the original semantic search models used in this embodiment are all known or trained.

Each original semantic search model processes different types of query sentences, and the accuracy is different accordingly. For query sentence types that the original semantic search model is good at processing, the accuracy of the processing is high. On the other hand, for query sentence types that the original semantic search model is not good at processing, the accuracy of the processing is low. Based on this, in this embodiment, when selecting a target query sentence type corresponding to each original semantic search model, the corresponding target query sentence type can be obtained based on the accuracy of each original semantic search model processing the query sentence corresponding to each query sentence type. For example, the most accurate query sentence type can be selected as the target query sentence type. Optionally, in this embodiment, one, two or more target query sentence types corresponding to each original semantic search model can be obtained according to actual needs. For example, when only one target query sentence type is obtained, more target query sentence types may be obtained because there is a shortage of language material for selectable target query sentences. This may be considered as the target query sentence type being a type of query sentence that the original semantic search model is good at. This allows the characteristics of the corresponding original semantic search model to be more embodied when distilled data is obtained based on the target query sentence type. Furthermore, when training the target semantic model based on such distilled data, the target semantic search model can learn the characteristics of the original semantic search model.

The target query sentence type in this embodiment may be an address-seeking type, an answer-seeking class, or a resource-seeking class. Alternatively, in practical applications, it is possible to divide the query sentence type according to the field, scenario, etc., and obtain the corresponding target query sentence type.

In this embodiment, a distilled dataset can be obtained based on at least two original semantic search models, the target query sentence types corresponding to each original semantic search model, and a pre-constructed corpus. This distilled dataset is screened based on at least two original semantic search models, so that the characteristics of each original semantic search model are compatible. Furthermore, by training the target semantic search model based on the distilled dataset, the characteristics of each original semantic search model can be integrated into the target semantic search model, overcoming the shortcomings of a single semantic search model being inaccurate, and performing more accurate semantic searches and improving the accuracy of recall results.

The method for training a semantic search model in this embodiment obtains a distilled dataset based on at least two original semantic search models and a pre-constructed corpus, and further trains a target semantic search model based on the distilled dataset, thereby integrating the search capabilities of the at least two original semantic search models into the trained target semantic search model, overcoming the shortcomings of a single semantic search model, and improving the accuracy of semantic search.

Figure 2 is a schematic diagram according to a second embodiment of the present disclosure. The method for training a semantic search model of this embodiment is described in more detail based on the technical solution of the embodiment shown in Figure 1 above. As shown in Figure 2, the method for training a semantic search model of this embodiment can specifically include the following steps:

In S201, a target query sentence type corresponding to each of at least two original semantic search models is obtained.

In S202, a target query sentence corresponding to each original semantic search model is obtained based on the target query sentence type corresponding to each original semantic search model.

In S203, a predetermined number of recall language materials are recalled from the corpus based on each original semantic search model and the target query sentence corresponding to each original semantic search model.

At S204, a distilled dataset is generated based on the target query sentences corresponding to each original semantic search model and the corresponding recalled predetermined number of recall linguistic materials.

In order for the distilled data in the generated distilled dataset to more accurately reflect the characteristics of each original semantic search model, in this embodiment, as many query sentences as possible that each original semantic search model excels at are selected as target queries to perform language material recall. Based on the target queries of each semantic search model and the recall of the corresponding recall language materials, the search ability and search characteristics of the corresponding semantic search model can be reflected. Therefore, in this embodiment, a distilled dataset is generated based on the target query sentences corresponding to each original semantic search model and a predetermined number of recalled recall language materials.

In this embodiment, in step S201, obtaining the target query sentence type corresponding to each original semantic search model may specifically include any of the following methods:

The first method obtains target query sentence types corresponding to each original semantic search model based on a test set corresponding to each type of query sentence that has been pre-constructed.

In the first method, a corresponding test set can be pre-constructed based on different types of query sentences. For example, the different types of query sentences can include address-seeking types, answer-seeking types, or resource-seeking types. Alternatively, the types of query sentences can be divided according to domains or scenarios.

In this method, a test set corresponding to each type of query sentence can be used to detect the accuracy rate of each original semantic search model. If the accuracy rate is greater than a preset accuracy rate threshold, such as 90%, 95% or other proportional value, the query sentence type can be the target query sentence type corresponding to the original semantic search model. In addition, the various query sentence types can be sorted in order of accuracy to obtain the most accurate query sentence type. In this embodiment, the target query sentence type corresponding to each original semantic search model can be one, two or more, and is not limited here.

The second method obtains a target query sentence type corresponding to each original semantic search model based on the attributes of each original semantic search model.

In this implementation method, the attributes of the original semantic search model can be predefined. Specifically, the attributes of the original semantic search model can be arranged based on the type, field, or scenario of the training set when training the model. The attributes can indicate the type, field, or scenario of the training data used when the original semantic search model was trained, indicating that the original semantic search model is good at processing query sentences in the type, field, or scenario, and has the highest accuracy in processing query sentences in the type, field, or scenario compared with processing query sentences of other types. Here, the type of training set can also include a type for requesting an address, a type for requesting an answer, or a type for requesting a resource. Based on this, the attributes of the original semantic search model can be set as the target query sentence type corresponding to the original semantic search model.

In this embodiment, whichever of the above methods is adopted, it is possible to accurately obtain the target query sentence type corresponding to each original semantic search model.

Next, a target query sentence corresponding to each original semantic search model is obtained based on the target query sentence type corresponding to each original semantic search model. For example, various types of query sentences can be collected and stored in a query sentence corpus based on the user's historical behavior data. At the time of use, any corresponding query sentence can be obtained from the query sentence corpus based on the target query sentence type and set as the target query sentence. Alternatively, the target query sentence may be obtained in a different manner. For example, any one query sentence of the corresponding type can be obtained from the user's log directly based on the target query sentence type as the target query sentence.

In this embodiment, when generating a distilled dataset, steps S202 and S203 may be repeated as described above depending on the size of the distilled dataset to be generated, to obtain each target query sentence corresponding to each original semantic search model and a predetermined number of recall linguistic materials recalled from the corpus for each target query sentence. That is, for one target query sentence and one original semantic search model, a predetermined number of recall linguistic materials can be recalled from the corpus. The predetermined number in this embodiment can be set according to actual needs. For example, it may be 100, 80, 50, 20, or other number of recall linguistic materials.

In this embodiment, the target query sentence corresponding to each original semantic search model and a predetermined number of recalled recall linguistic materials can be added directly to the distilled dataset as distilled data. Since all recall linguistic materials are obtained according to the target query sentence based on the original semantic search model, they are distilled data as positive samples in the distilled dataset. Therefore, it is necessary to mark this distilled data as a positive sample by arranging the correlation between the target query sentence and the recall linguistic materials as 1, and it may be indicated that the probability that the recall linguistic materials will be recalled when the target query sentence is searched is 1.

Conversely, when constructing distilled data of negative samples in a distilled dataset, non-recallable negative sample language material can be constructed by referring to the target query sentence corresponding to the original semantic search model and a predetermined number of recalled recall language materials, and the distilled data can be marked as a negative sample by arranging the negative sample language material so that the correlation between the target query sentence and the negative sample language material is 0, thereby indicating that the probability that the negative sample language material will be recalled when the target query sentence is searched is 0.

Steps S201 to S204 are an embodiment of step S101 in the embodiment shown in Figure 1 above.

At S205, a target semantic search model is trained based on the distilled dataset.

The method for training a semantic search model in this embodiment obtains a target query sentence corresponding to each original semantic search model and a predetermined number of recalled linguistic materials recalled accordingly to generate a distilled dataset, and further trains a target semantic search model based on the distilled dataset. The target query sentence corresponding to each original semantic search model obtained and the predetermined number of recalled linguistic materials recalled accordingly can fully express the capabilities and performance of each original semantic search model, so that the generated distilled dataset can retain the characteristics of each original semantic search model. Furthermore, when training a target semantic search model based on the distilled dataset, the search capabilities of at least two original semantic search models can be integrated into the trained target semantic search model, overcoming the shortcomings of a single semantic search model and effectively improving the accuracy of semantic search.

Figure 3 is a schematic diagram according to a third embodiment of the present disclosure. The method for training a semantic search model of this embodiment is described in more detail based on the technical solution of the embodiment shown in Figure 1 above. As shown in Figure 3, the method for training a semantic search model of this embodiment can specifically include the following steps:

In S301, a target query sentence type corresponding to each of at least two original semantic search models is obtained based on a pre-defined test set corresponding to each type of query sentence.

The specific implementation method can be seen in the description of the embodiment shown in Figure 2 above, so it will not be described in detail here.

In S302, a target query sentence corresponding to each original semantic search model is obtained based on the target query sentence type corresponding to each original semantic search model.

In S303, a predetermined number of recall language materials are recalled from the corpus based on each original semantic search model and the target query sentence corresponding to each original semantic search model.

At S304, a pre-trained fine sorting model is used to screen the target query sentences corresponding to each original semantic search model and a predetermined number of recalled linguistic materials to generate a distilled dataset.

Steps S301 to S304 are an embodiment of step S101 in the embodiment shown in Figure 1 above.

Unlike the embodiment shown in FIG. 2 above, in this embodiment, when generating a distilled dataset by screening, a fine sorting model is further used to screen the target query sentences and a predetermined number of recalled linguistic materials corresponding to each original semantic search model to generate a distilled dataset. For example, FIG. 4 is a configuration diagram of a training method for a semantic search model in this embodiment. Accordingly, the configuration diagram of the embodiment shown in FIG. 2 may be the screening of the recall results excluding the fine sorting model in FIG. 4.

The original semantic search model and the target semantic search model using the neural network structure of this embodiment both use a double-tower structure model to realize the encoding of the query sentence query and the candidate linguistic materials, and further calculate the vector similarity between the two based on the encoding result. The fine sort model of this embodiment is obtained by pre-training using annotation data. The fine sort model can model the interaction information between the query sentence query and the candidate linguistic materials, and its ability is superior to that of the corresponding double-tower structure model. Therefore, by using the fine sort model to screen and filter the recall results of each original semantic search model, the quality of the distilled data in the distilled dataset can be effectively improved.

In one embodiment of the present disclosure, step S304, using a pre-trained fine sorting model to screen the target query sentences corresponding to each original semantic search model and the recalled predetermined number of recall linguistic materials to generate a distilled dataset, may specifically include the following steps:

(a1) Screen positive sample data based on the fine sorting model, the target query sentence corresponding to each original semantic search model, and a predetermined number of recalled linguistic materials.

(b1) Screening negative sample data based on the fine sorting model, the target query sentence corresponding to each original semantic search model, and a predetermined number of recalled linguistic materials.

(c1) Store the positive sample data and negative sample data in a distilled dataset.

For example, in this embodiment, based on the fine sorting model, the positive sample data and the negative sample data can be separately screened and compiled as distilled data to form a distilled dataset. Specifically, the ratio of positive sample data to negative sample data in the distilled dataset can be set according to actual needs, and may be, for example, 1:1, 1:2, 1:3, 1:4, or other ratios, and is not limited here. The corresponding sample data may be screened according to the number of desired samples.

In this embodiment, when a distilled dataset is generated, the positive sample data and the negative sample data may not be labeled in the method of integrating samples or in the distilled data. For example, each distilled data may include one query sentence, at least two recall language materials, and a ranking of the correlation degree between the at least two recall language materials and the query sentence. This type of distilled data may be soft-labeled data. By training the target semantic search model with this type of distilled data, the target semantic search model can learn a score ranking of the correlation degree between different recall language materials and the same query sentence. In this embodiment, in step S304, the target query sentence corresponding to each original semantic search model and a predetermined number of recall language materials that have been recalled are screened using a pre-trained fine sorting model to generate a distilled dataset in the following three cases.

The first case is direct integration. In this case, it is necessary to obtain positive sample data and negative sample data according to the above-mentioned method based on each original semantic search model, and then directly integrate the positive sample data and negative sample data generated by each of at least two original semantic search models to obtain a final distilled dataset. That is, the distilled dataset obtained in this case sufficiently contains the positive sample data and negative sample data distilled for each original semantic search model.

For example, step (a1) in this case may accordingly include the following steps:

(a2) For each target query sentence corresponding to each original semantic search model, calculate a correlation score between the target query sentence and each of a predetermined number of recall linguistic materials recalled correspondingly using the fine sort model.

(b2) Removing recall language materials whose correlation scores are less than a preset threshold from the first N of a predetermined number of recall language materials, where N is a positive integer greater than 1.

(c2) Construct positive sample data based on the target query sentence and each of the remaining recall language materials among the first N in the predetermined number of recall language materials.

For example, the positive sample data constructed at this point may include the target query sentence, and the first N recall language materials in a predetermined number of recall language materials that have a correlation score equal to or greater than a preset threshold. In order to train the target semantic search model using these samples as positive samples, the correlation score label of the positive sample data at this point can be rearranged to 1, thereby allowing the target semantic search model to learn the ability to recall the recall language materials in the corresponding positive sample data based on the target query sentence.

For example, in this case step (b1) may accordingly include the following steps:

(a3) For a target query sentence corresponding to each original semantic search model, select recall language materials whose correlation scores are smaller than a preset threshold from among the N+1th and subsequent recall language materials among a predetermined number of recall language materials. N is a positive integer greater than 1.

(b3) Construct negative sample data based on the target query sentence and recall language materials selected from the N+1th or later recall language materials in a predetermined number of recall language materials, the correlation score of which is less than a preset threshold value.

Similarly, the negative sample data constructed at this point may include the target query sentence and recall language materials from the N+1th and subsequent recall language materials in the predetermined number of recall language materials whose correlation scores are smaller than a preset threshold. In order to train the target semantic search model using this sample as a negative sample, the correlation score label of the negative sample data at this point is rearranged to 0, so that the target semantic search model can learn the ability to not recall the recall language materials in the corresponding negative sample data based on this target query sentence.

For example, when the predetermined number is 100, N is 20. The quality of the positive sample can be improved by selecting recall language materials having a correlation score smaller than a preset threshold from the first 20 of the first 100 recall language materials recalled by each original semantic search model for any one target query sentence. On the other hand, the negative sample data can be constructed by selecting recall language materials having a correlation score smaller than a preset threshold from the 20th to 100th recall language materials. The preset threshold can be empirically set to 0.1, 0.2, or other numerical values. By adopting this method, the quality of the distilled data in the distilled dataset can be effectively improved.

The second case is cross-integration. In this case, the positive sample data generated for each original semantic search model can be taken and stored in a distilled dataset. Meanwhile, the negative sample data can be screened from all recall results from at least two original semantic search models. That is, the distilled dataset obtained in this case contains sufficient positive sample data distilled for each original semantic search model, but the negative sample data may be distilled from only a portion of the original semantic search models.

For example, depending on this case, in step (b1), screening the negative sample data based on the fine sorting model, the target query sentence corresponding to each original semantic search model, and the recalled predetermined number of recall linguistic materials may specifically include the following steps:

(a4) For each target query sentence corresponding to each original semantic search model, calculate a correlation score between the corresponding target query sentence and each recall linguistic material in the predetermined number of recall linguistic materials recalled accordingly using the fine sort model.

(b4) Based on the correlation score between each target query sentence and each recall language material, negative sample data is screened from all recall language materials from at least two original semantic search models according to a predetermined method.

For example, the required number of negative sample data can be screened in ascending order of correlation score. Alternatively, all recall language materials whose correlation scores are smaller than a preset threshold can be directly obtained. Alternatively, the negative sample data can be screened in other ways, but this is not limited thereto. In this way, by referring to all recall language materials from at least two original semantic search models, higher quality negative sample data can be selected, and the quality of the distilled dataset can be further improved.

The third case is soft-label data integration. In this case, unlike the above two cases, the target query sentences corresponding to at least two original semantic search models and a predetermined number of recalled linguistic materials recalled accordingly are integrated without distinguishing between positive and negative samples, while retaining the original scores or rankings of the original semantic search models. In this case, when training the target semantic search model using the distilled dataset, the target semantic search model is trained to learn the scores or rankings for different sample data by the original semantic search model.

In practical applications, one can choose any of the above three cases to generate a distilled dataset as needed. Either way, an accurate, reasonable, and effective distilled dataset can be obtained.

In this embodiment, when generating a distilled dataset, the above method is used to generate distilled data of a related type by adopting the target query sentence type that each original semantic search model is good at as much as possible, thereby making the most of the advantages of each original semantic search model to generate advantageous distilled data. For example, since the RocketQAv2 model may be better for problem-based queries, the distribution of query sentences when generating the distilled dataset in this step is adjusted, and the proportion of problem-based query sentences handled by the RocketQAv2 model is increased to obtain more effective distilled data. The same applies to other original semantic search models, and in order to obtain more effective distilled data, a query sentence type that is good at processing is selected, but this will not be described in detail here.

At S305, a target semantic search model is trained based on the distilled dataset.

The target semantic search model in this embodiment is a double tower structure model.

Different training methods can be used for different distilled dataset generation methods. For example, for the first and second cases of generating distilled datasets, i.e., the direct integration and cross integration generation methods, the hard label method can be used for training, and a common comparative learning training method can be used to introduce within-lot negative sampling using cross entropy loss. For the soft label data integration method, i.e., the third case of generating the distilled dataset, the margin MSE method can be used for training to learn the scoring between sample pairs. These two methods can be flexibly selected according to the effect of training.

The double-column target semantic search model trained using the integrated distilled data can integrate the features of multiple teachers, i.e., the original semantic search models, and control the data distribution of different teachers during the distilled data generation process, retaining only the parts that are advantageous and have large differences, thereby making the most of the generated weakly labeled training data.

After training on weakly labeled data, the effectiveness of the double-tower target semantic search model can exceed the results of training on annotated data, and has stronger generalization, reducing over-fitting to the annotated data, and partially resolving the robustness problem caused by data deviation.

The method for training a semantic search model in this embodiment uses a fine sorting model to obtain the target query sentences of each original semantic search model and a predetermined number of recalled linguistic materials recalled accordingly to generate a distilled dataset, and further effectively improve the quality of the distilled dataset. Furthermore, by training the target semantic search model based on the distilled dataset, the accuracy of the trained target semantic search model can be improved.

The method for training a semantic search model in this embodiment can integrate the capabilities of at least two types of original semantic search models into a target semantic search model with a double-tower structure, thereby effectively improving the search capability of the target semantic search model.

The training method for the semantic search model in this embodiment includes a sparse vector search model in at least two original semantic search models, so that the target semantic search model learns a certain degree of sparse vector search ability, improves the resolution ability of the fine sort matching scenario of the lexical text, improves the generalization ability of the target semantic search model, and gives it better zero-sample domain transfer performance.

In addition, in the embodiment of the present disclosure, for existing double-tower structured semantic search models, the characteristics of rapid deployment and high search efficiency of the double-tower model are maintained without changing the structure of the double-tower model, and it can be widely applied to large-scale search scenarios.

The method for training a semantic search model in this embodiment can be flexibly expanded when applied using a scheme that integrates the capabilities of multiple types of original semantic search models. When a model or system with higher capabilities in some respect emerges, this scheme can be adopted to effectively absorb the advantages of that system and improve the performance of the target semantic search model.

Figure 5 is a schematic diagram according to a fourth embodiment of the present invention. As shown in Figure 5, this embodiment provides a semantic search model training device 500 including: a type acquisition module 501 for acquiring a target query sentence type corresponding to each of at least two original semantic search models, the target query sentence type being the most accurate query sentence type when the original semantic search models process various types of query sentences; a data acquisition module 502 for acquiring a distilled dataset based on at least two original semantic search models, the target query sentence types corresponding to each original semantic search model, and a pre-constructed corpus; and a training module 503 for training the target semantic search model based on the distilled dataset.

The implementation principle and technical effect of the semantic search model training device 500 of this embodiment, which uses the above modules to realize training of a semantic search model, are similar to those of the implementation of the embodiment of the related method described above. For details, please refer to the description of the embodiment of the related method described above, and a detailed description will not be given here.

Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure. This embodiment provides a training device 600 for a semantic search model, which further describes the technical solution of the present disclosure in addition to the technical solution of the embodiment shown in Figure 5 above. As shown in Figure 6, this embodiment provides a training device 600 for a semantic search model, which includes a type acquisition module 601, a data acquisition module 602, and a training module 603, which are modules with the same names and functions as those shown in Figure 5.

Here, the data acquisition module 602 includes a sentence acquisition unit 6021 that acquires a target query sentence corresponding to each original semantic search model based on a target query sentence type corresponding to each original semantic search model, a language material acquisition unit 6022 that recalls a predetermined number of recall language materials from the corpus based on each original semantic search model and the target query sentence corresponding to each original semantic search model, and a generation unit 6023 that generates a distilled dataset based on the target query sentence corresponding to each original semantic search model and the predetermined number of recall language materials recalled accordingly.

Furthermore, in one embodiment of the present disclosure, the type acquisition module 601 acquires a target query sentence type corresponding to each original semantic search model based on a pre-set test set corresponding to each type of query sentence.

Furthermore, in one embodiment of the present disclosure, the type acquisition module 601 acquires a target query sentence type corresponding to each original semantic search model based on the attributes of each original semantic search model.

Furthermore, in one embodiment of the present disclosure, the generation unit 6023 uses a pre-trained fine sorting model to screen the target query sentences corresponding to each original semantic search model and a predetermined number of recalled linguistic materials recalled accordingly to generate a distilled dataset.

Furthermore, in one embodiment of the present disclosure, the generation unit 6023 screens positive sample data based on the fine sorting model, the target query sentence corresponding to each original semantic search model, and a predetermined number of recall linguistic materials recalled accordingly, and screens negative sample data based on the fine sorting model, the target query sentence corresponding to each original semantic search model, and a predetermined number of recall linguistic materials recalled accordingly, and stores the positive sample data and the negative sample data in a distilled dataset.

Furthermore, in one embodiment of the present disclosure, the generation unit 6023 uses a fine sorting model to calculate a correlation score between the target query sentence and each of a predetermined number of recall linguistic materials recalled correspondingly for a target query sentence corresponding to each original semantic search model, deletes recall linguistic materials having a correlation score smaller than a preset threshold from the first N (N is a positive integer greater than 1) of the predetermined number of recall linguistic materials, and constructs positive sample data based on the target query sentence and each of the remaining recall linguistic materials among the first N of the predetermined number of recall linguistic materials.

Furthermore, in one embodiment of the present disclosure, for a target query sentence corresponding to each original semantic search model, the generation unit 6023 selects recall language materials having a correlation score smaller than a preset threshold from among the N+1th (N is a positive integer greater than 1) or later recall language materials among a predetermined number of recall language materials, and constructs negative sample data based on the target query sentence and the recall language materials having a correlation score smaller than a preset threshold selected from the N+1th or later recall language materials among the predetermined number of recall language materials.

Furthermore, in one embodiment of the present disclosure, the generation unit 6023 uses a fine sorting model to calculate, for each target query sentence corresponding to each original semantic search model, a correlation score between the corresponding target query sentence and each of a predetermined number of recall language materials recalled accordingly, and screens negative sample data from all recall language materials from at least two original semantic search models in a predetermined manner based on the correlation scores of each target query sentence and each recall language material.

The implementation principle and technical effect of the semantic search model training device 600 of this embodiment, which uses the above modules to realize training of a semantic search model, are similar to those of the implementation of the above-mentioned related method embodiment. For details, please refer to the description of the above-mentioned related method embodiment, and a detailed description will not be given here.

In the technical solution disclosed herein, the acquisition, storage, application, etc. of personal information of the users involved conforms to the provisions of relevant laws and regulations and is not contrary to public order and morals.

According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

FIG. 7 shows a schematic block diagram of an exemplary electronic device 700 capable of implementing embodiments of the present disclosure. The electronic device represents various forms of digital computers, such as laptops, desktop computers, workbenches, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as PDAs, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present disclosure as described and/or claimed herein.

As shown in FIG. 7, the device 700 includes a computing means 701 capable of performing various appropriate operations and processes according to a computer program stored in a read only memory (ROM) 702 or loaded from a storage means 708 into a random access memory (RAM) 703. The RAM 703 may store various programs and data necessary for the operation of the device 700. The computing means 701, the ROM 702, and the RAM 703 are connected via a bus 704. An input/output (I/O) interface 705 is also connected to the bus 704.

Several components of the device 700 are connected to the I/O interface 705, including input means 706, e.g., a keyboard, a mouse, etc., output means 707, e.g., various types of displays, speakers, etc., storage means 708, e.g., a magnetic disk, an optical disk, etc., and communication means 709, e.g., a network card, a modem, a wireless communication transceiver, etc. The communication means 709 allows the device 700 to exchange information/data with other devices via a computer network, e.g., the Internet, and/or various telecommunication networks.

The computing means 701 may be various general-purpose and/or dedicated processing components having processing and computing capabilities. Some examples of the computing means 701 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing means 701 performs various methods and processes described above, such as the methods of the present disclosure. For example, in some embodiments, the methods of the present disclosure may be implemented as a computer software program physically embodied in a machine-readable medium, such as the storage means 708. In some embodiments, some or all of the computer program may be loaded and/or installed in the device 700 via the ROM 702 and/or the communication means 709. When the computer program is loaded into the RAM 703 and executed by the computing means 701, one or more steps of the above methods of the present disclosure may be performed. Alternatively, in other embodiments, the computing means 701 may be configured in any other suitable manner (e.g., via firmware) to perform the methods of the present disclosure.

Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being embodied in one or more computer programs. The one or more computer programs may be executed and/or interpreted on a programmable system that includes at least one programmable processor. The programmable processor may be a dedicated or general purpose programmable processor that is capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

Program codes for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the program codes perform the functions/operations specified in the flowcharts and/or block diagrams. The program codes can be executed entirely on the machine, partially on the machine, as a stand-alone package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

In the context of this disclosure, a machine-readable medium is a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) for providing input to the computer by the user. Other types of devices may also be used to provide interaction with a user. For example, feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form (including sound input, speech input, or haptic input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and an internetwork.

The computer system may include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship between the clients and servers arises by computer programs running on the corresponding computers and having a client-server relationship with each other. The servers may be cloud servers, servers in a distributed system, or servers in combination with a blockchain.

It should be understood that steps can be rearranged, added, or removed using the various types of flows shown above. For example, each step described in this disclosure may be performed in a parallel order or sequential order, or may be performed in a different order, and is not limited thereto, so long as the desired results of the technical proposal disclosed in this disclosure are achieved.

The above specific embodiments do not constitute limitations on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure should be included within the scope of protection of the present disclosure.

Claims (21)

1. A computer-implemented method for training a semantic search model, comprising:
Obtaining a target query sentence type corresponding to each of at least two original semantic search models, the target query sentence type being the most accurate query sentence type when the original semantic search models process various types of query sentences;
Obtaining a distilled dataset based on at least two original semantic search models, a target query sentence type corresponding to each of the original semantic search models, and a pre-constructed corpus;
training a target semantic search model based on the distilled dataset;
How to train a semantic search model. Obtaining a target query sentence type corresponding to each of the at least two original semantic search models includes:
Obtaining the target query sentence type corresponding to each of the original semantic search models based on a pre-defined test set corresponding to various types of query sentences;
The method for training a semantic search model according to claim 1 . Obtaining a target query sentence type corresponding to each of the at least two original semantic search models includes:
obtaining the target query sentence type corresponding to each of the original semantic search models based on attributes of each of the original semantic search models;
The method for training a semantic search model according to claim 1 . Obtaining a distilled dataset based on at least two original semantic search models, a target query sentence type corresponding to each of the original semantic search models, and a pre-constructed corpus includes:
Obtaining a target query sentence corresponding to each of the original semantic search models according to a target query sentence type corresponding to each of the original semantic search models;
Recalling a predetermined number of recall language materials from the corpus based on each of the original semantic search models and a target query sentence corresponding to each of the original semantic search models;
generating the distilled dataset based on target query sentences corresponding to each of the original semantic search models and the predetermined number of recall linguistic materials recalled accordingly;
A method for training a semantic search model according to any one of claims 1 to 3. generating the distilled dataset based on the target query sentences corresponding to each of the original semantic search models and the predetermined number of recall linguistic materials recalled accordingly,
using a pre-trained fine sorting model to screen the target query sentences corresponding to each of the original semantic search models and the predetermined number of recalled linguistic materials recalled accordingly to generate the distilled dataset;
The method for training a semantic search model according to claim 4. Using a pre-trained fine sorting model to screen the target query sentences corresponding to each of the original semantic search models and the predetermined number of recall linguistic materials recalled accordingly to generate the distilled dataset,
Screening positive sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly;
Screening negative sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly;
storing the positive sample data and the negative sample data in the distilled data set.
The method for training a semantic search model according to claim 5. Screening positive sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly,
For each target query sentence corresponding to the original semantic search model, calculating a correlation score between the target query sentence and each of the predetermined number of recall linguistic materials recalled correspondingly using the fine sort model;
removing recall language materials having the correlation score smaller than a preset threshold from the first N (N is a positive integer greater than 1) of the predetermined number of recall language materials;
constructing the positive sample data based on the target query sentence and each remaining recall linguistic material among the first N in the predetermined number of recall linguistic materials;
The method for training a semantic search model according to claim 6. Screening negative sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly,
For each target query sentence corresponding to the original semantic search model, select recall language materials having a correlation score smaller than a preset threshold from among the N+1th (N is a positive integer greater than 1) or later recall language materials in the predetermined number of recall language materials;
and constructing negative sample data based on the target query sentence and recall language materials selected from the N+1th or later recall language materials in the predetermined number of recall language materials, the correlation score of which is less than a preset threshold value.
The method for training a semantic search model according to claim 6. Screening negative sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly,
For each target query sentence corresponding to the original semantic search model, calculating a correlation score between the corresponding target query sentence and each of the predetermined number of recall linguistic materials recalled correspondingly using the fine sort model;
and screening the negative sample data from all recall language materials according to the at least two original semantic search models based on a correlation score between each of the target query sentences and each of the recall language materials in a preset manner.
The method for training a semantic search model according to claim 6. An apparatus for training a semantic search model, comprising:
a type acquisition module for acquiring a target query sentence type corresponding to each of at least two original semantic search models, the target query sentence type being the most accurate query sentence type when the original semantic search model processes various types of query sentences;
A data acquisition module for acquiring a distilled dataset based on at least two original semantic search models, a target query sentence type corresponding to each of the original semantic search models, and a pre-constructed corpus;
a training module for training a target semantic search model based on the distilled dataset;
A training device for semantic retrieval models. The type acquisition module includes:
Obtaining the target query sentence type corresponding to each of the original semantic search models according to a pre-defined test set corresponding to various types of query sentences;
An apparatus for training a semantic search model according to claim 10. The type acquisition module includes:
obtaining the target query sentence type corresponding to each of the original semantic search models based on the attributes of each of the original semantic search models;
An apparatus for training a semantic search model according to claim 10. The data acquisition module includes:
a sentence acquisition unit that acquires a target query sentence corresponding to each of the original semantic search models according to a target query sentence type corresponding to each of the original semantic search models;
a language material acquisition unit that recalls a predetermined number of recall language materials from the corpus based on each of the original semantic search models and a target query sentence corresponding to each of the original semantic search models;
A generator that generates the distilled dataset based on a target query sentence corresponding to each of the original semantic search models and the predetermined number of recall linguistic materials recalled accordingly.
An apparatus for training a semantic search model according to any one of claims 10 to 12. The generation unit is
Using a pre-trained fine sorting model, screen the target query sentences corresponding to each of the original semantic search models and the predetermined number of recalled linguistic materials recalled accordingly to generate the distilled dataset.
An apparatus for training a semantic search model according to claim 13. The generation unit is
Screening positive sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly;
Screening negative sample data based on the fine sorting model, the target query sentence corresponding to each of the original semantic search models, and the predetermined number of recall linguistic materials recalled accordingly;
storing the positive sample data and the negative sample data in the distilled data set;
An apparatus for training a semantic search model according to claim 14. The generation unit is
For each target query sentence corresponding to the original semantic search model, calculate a correlation score between the target query sentence and each of the predetermined number of recall linguistic materials recalled correspondingly using the fine sort model;
Removing recall language materials having correlation scores smaller than a preset threshold from the first N (N is a positive integer greater than 1) of the predetermined number of recall language materials;
constructing the positive sample data based on the target query sentence and each remaining recall linguistic material among the first N of the predetermined number of recall linguistic materials;
An apparatus for training a semantic search model according to claim 15. The generation unit is
For each target query sentence corresponding to the original semantic search model, select recall language materials having a correlation score smaller than a preset threshold from among the N+1th (N is a positive integer greater than 1) or later recall language materials in the predetermined number of recall language materials;
constructing negative sample data based on the target query sentence and recall language materials selected from the N+1th or later recall language materials in the predetermined number of recall language materials, the correlation score of which is less than a preset threshold value;
An apparatus for training a semantic search model according to claim 15. The generation unit is
For each target query sentence corresponding to the original semantic search model, calculate a correlation score between the corresponding target query sentence and each of the predetermined number of recall linguistic materials recalled correspondingly using the fine sort model;
Screening the negative sample data from all the recall language materials according to the at least two original semantic search models based on a correlation score between each of the target query sentences and each of the recall language materials in a preset manner;
An apparatus for training a semantic search model according to claim 15. At least one processor;
a memory communicatively coupled to the at least one processor;
An electronic device having stored in the memory instructions executable by the at least one processor, the commands, when executed by the at least one processor, causing the at least one processor to perform the method for training a semantic search model described in any one of claims 1 to 3. A non-transitory computer-readable storage medium storing computer commands for causing a computer to execute the method for training a semantic search model according to any one of claims 1 to 3. A computer program that, when executed by a processor, realizes the method for training a semantic search model according to any one of claims 1 to 3.

Images