JP6345638B2 - System and method for maintaining speech-to-speech translation in the field - Google Patents

Description

(Cross-reference of related applications)
This application is filed in US provisional patent application no. 61/045, 079, US provisional application no. 61 / 092,581, and US provisional patent application no. Claim priority of 61 / 093,898.

  The present invention is generally directed to speech-to-speech translation systems for interlingual communication, and more particularly, a new vocabulary (in which a user does not require linguistic or technical or technical knowledge in the field) It is directed to a method and apparatus for field maintenance that allows new vocabulary) items to be added and to improve and modify the content and usage of the user's system.

  Automatic speech recognition (ASR) and machine translation (MT) technologies have been well developed to develop practical speech translation systems on laptops or mobile devices for limited and non-limited areas. Has become feasible. In particular, domain-specific speech-to-speech conversion systems have been developed for a variety of application areas, including tourism, medical deployment, and military, in the research field and in the laboratory. Such a system has been known for some time in the following work: A. Waibel, C. Fugen, "Spoken language translation" in Signal Processing Magazine, IEEE May 2008; 25 (3): 70-79. , In Proc. HLT, 2003, and Nguyen Bach, Matthias Eck, Paisarn Charoenpornsawat, Thilo Kohler, Sebastian Stuker, ThuyLinh Nguyen, Roger Hsiao, Alex Waibel, Stephan Vogel, Tanja Schultz and Alan W. Black, TransTac 2007 eyes-free and hands-free two-way speech-to-speech translation system, "In Proc. Of the IWSLT, Trento, Italy, October 2007. However, these studies have been defined in advance by system developers and are limited to work with the limited vocabulary determined by the domain of application and the intended location where the system will be used. Vocabulary and language usage is therefore largely determined based on example scenarios and by data collected or estimated in such scenarios.

  However, in the field situation, the actual language and language usage deviates from the expected scenario in the laboratory. Even in simple areas such as tourism, language usage changes dramatically in the field as users move to different locations, interact with different people and pursue different goals and needs. To do. Therefore, new words (new words) and new phrases always arise. Such a new word, the speech recognition term “unknown word” (OOV), is misrecognized as a word in the dictionary and translated incorrectly. The user may try to paraphrase, but if he / she cannot enter a definitive word or concept (such as a person name or city name) or cannot communicate, communication may be disrupted due to lack of words or phrases .

  Despite the need for a user-correctable speech-to-speech translation system, no actual solution has been proposed so far. While it may seem easy to add words to the system, it has proven to be extremely difficult to make such modifications. Appropriate modifications must be made to many component modules throughout the system, and most modules must be retrained to restore component balance and integrated functionality. In practice, roughly 20 different modules will have to be modified or reoptimized to learn new words. Such modifications require specialized knowledge and experience with the components of the speech translation system. As a result, according to the inventor's understanding, such modifications have so far been made only by specialists within the laboratory, requiring human expertise, time and cost.

  For example, assume a system designed for European users does not include the name “Hong Kong” in its vocabulary. When a speaker speaks the sentence “Let's go to Hong Kong”, the system recognizes the most similar phonetic similar words in the dictionary and generates “Let's go to home call”. Become. At this point it is not clear whether the error is the result of a recognition error or the result of this word loss in the entire speech-to-speech translation system. Therefore, the user then begins to modify the system. This can be done by one of several modification techniques. Most simply, it may speak or type again, but instead, cross-modal error correction as described in other disclosures and prior art (Waibel et al., US Pat. No. 5,855,000). It can be done more effectively with technology. Once the correct spelling with the desired word sequence is established ("Let's go to Hong Kong"), the system performs the translation. If "Hong Kong" is in the dictionary, the system will continue normally from there and perform translation and synthesis. However, if it is not in the recognition and translation dictionary, the system will need to verify whether this word is a proper expression. Finally, and most importantly, if the user speaks the same word the next time, even if the name or word can be properly translated into the output language without being learned through user intervention, the system will It will break down again.

  Unfortunately, new word learning cannot be addressed by simply typing in a new word in the word list and requires changes at roughly 20 different points and at all levels of the speech translation system. To do. Currently, new word learning includes manual tagging and editing of entries, extensive database collection including required words, retraining probabilities of language and translation models, and reoptimization of the entire system. To attempt to reestablish consistency between all components and component dictionaries, and to repair the statistical balance between words, phrases and concepts in the system (probability is total 1 and therefore all words will be affected by the addition of a single word).

  As a result, current speech translation systems generally required the use of advanced computational tools and linguistic resources found within the laboratory, even for minor modifications. However, when used in a real field, requiring any modifications made in the laboratory is not acceptable because it takes too much time, effort and cost. Instead, it hides all complexity from the user and performs all critical actions and language processing steps semi-autonomously or autonomously behind the scenes, and through a simple and intuitive interface whenever possible There is a need for learning and customization modules that interact with human users in a non-confusing manner that completely eliminates the need for linguistic or technical expertise in the field. In the present invention, we provide a detailed explanation that the learning and customization module satisfies these needs.

  Unfortunately, many of the translation systems are very complex and user access is not feasible or utilized. Therefore, while utilizing machine translation technology, the user correction function allows to provide interlingual communication without the need for linguistic or technical knowledge or expertise, thereby overcoming language barriers What is needed is a system and method that allows people to be more intimate.

  In various embodiments, the present invention solves the above-described problems by providing a method and apparatus for updating a vocabulary of a speech translation system. In various embodiments, a method is provided for updating a vocabulary of a speech translation system that translates a first language, including written and spoken language, into a second language. The method includes adding a new word in the first language to a first recognition lexicon in the first language and associating a description including pronunciation and word class information with the new word. These new words and descriptions are then updated in a first machine translation module associated with the first language. The first machine translation module includes a first tagging module, a first translation model, and a first language module, and is configured to translate the new word into a corresponding translated word in the second language.

  Optionally, for bi-directional translation, the method additionally includes translating the translated word back from the second language to a new word in the first language and associating the new word with a corresponding translated word in the second language. Adding a translated word and a description of the translated word to the second recognition dictionary of the second language. The second machine translation module associated with the second language is then updated with this translation and description. The second machine translation module includes a second tagging module, a second translation model, and a second language module.

  In embodiments, the method includes inputting a first word into a text-phonetic dictionary associated with the first language and inputting a second word into a text-phonetic dictionary associated with the second language. Is further included. These input signals are of different modalities (eg, voice spelling and verbal spelling, voice spelling and verbal spelling, written and voice, etc.) (referred to herein as “cross modal”). Or the same modality (voice and redo voice, written and rewritten text, etc.).

  An embodiment of the present invention is directed to a speech-to-speech translation system that communicates between a first language and a second language, which is a class-based system that can be maintained in the field. The system includes two speech recognition units, each speech recognition unit configured to receive speech containing a spoken language of a first language or a second language and generate text corresponding to the spoken language. A unit and two corresponding machine translation units, each machine translation unit configured to receive text from one of the speech recognition units and output the translation of the text into text in the other language Including translation units. The system also includes a user site customization module that enables the system to learn new words in cooperation with the user. The user site customization module is configured to receive a user selection input that includes speech or text corresponding to one or both of the languages, and appropriately updates the machine translation unit with the user selection input.

  In one embodiment, the system is equipped with four main features to provide a class-based speech-to-speech translation system that can be maintained in the field. The first feature includes a speech translation framework that allows new words to be added to the active system vocabulary, or allows switching between multiple vocabularies specific to a place or task. The speech translation framework results in dynamically adding words to the speech recognition module without requiring restart of the module. The system includes a multilingual system dictionary and language independent word-class across all system components in a speech-to-speech translation device, and class-based machine translation (phrase-based statistical MT, syntactic MT, example-based MT Etc.) and multi-language word-class tagging based on a combination of single language taggers during model training, and alignment from known tagged languages via parallel corpus Use word-class tagging in new languages. The second feature is that a multi-modal interactive interface allows non-experts to add new words to the system. A third feature is that the system is designed to adapt ASR and SMT model adaptation using multimodal feedback provided by the user. And the fourth feature is that this system has a network function that enables correction or word sharing.

  In another embodiment, a multimodal interactive interface is disclosed that allows a user to add new words to a speech-to-speech translation device in the field and without technical expertise. This includes the following examples: (1) how to automatically classify the class of words or phrases added to the system and automatically generate pronunciation and translation of the words, (2) speaking To input new words in a cross-modal manner by one or more of typing, spelling, handwriting, browsing, or paraphrasing. (3) Whether a linguistically untrained user has proper transliteration and translation of speech. Multi-modal feedback to help determine the decision, via multiple text formats (ie written in roman letters and other languages) and text-to-speech (TTS, ie whether it sounds properly) Feedback including speech format, (4) method of setting language models and translation probabilities for new words, and (5) user behavior, interests and usage history Against new words learned on the basis of relevance to raise the probability of language model and translation or discount it.

  In another embodiment, an online system is disclosed that modifies via multimodal feedback by the user in the field. This includes examples such as: (1) an interface and method that allows a user to modify the automatic speech recognition results and adapt the speech recognition component using this feedback information; (2) Interfaces and methods that allow a user to modify a machine translation hypothesis and use this feedback information to improve the machine translation component, and (3) the language for the correct or modified word based on the user modification A method of automatically adjusting (increasing or decreasing) the probability of models, dictionaries and translation models.

  In another embodiment, an Internet application is disclosed that allows users to share modifications or new word additions made in the field across multiple devices. These include the following examples: (1) how to upload, download and edit models for use in speech-to-speech translation devices via the World Wide Web, (2) new words across the user community A method of collating additions and modifications in the field, and (3) a method of uploading, downloading and editing a place or task specific vocabulary for use in a speech-to-speech translation device

  The accompanying drawings illustrate examples in embodiments of the present invention. A brief description of such drawings is as follows.

FIG. 1 is a block diagram illustrating a speech-to-speech translation system configured in accordance with one embodiment of the present invention.

FIG. 2 illustrates an example of a graphic user interface displayed to the user via the tablet interface.

FIG. 3 is a flow chart illustrating the steps of speech-to-speech translation performed in accordance with one embodiment of the present invention in FIG.

FIG. 4 is a flowchart illustrating the steps the system learns from modifications made by the user (modification and repair module).

FIG. 5 is a flowchart illustrating steps by which a user can add new words to the system (user site customization module).

FIG. 6 is a flow chart illustrating an example of how the device automatically generates translations and pronunciations for new words that the user wishes to add to the system.

FIG. 7 is a flowchart illustrating an example of a method for verifying a new word input via a multimodal interface.

FIG. 8 illustrates an example of a visual interface that displays automatically generated word information.

FIG. 9 is a flowchart illustrating the steps necessary to train a class-based MT model.

FIG. 10 is a flowchart illustrating the steps of applying the class-based MT to the input sentence.

FIG. 11 is a diagram illustrating features that may be used during word-class tagging by statistical or machine learning approaches.

  Various embodiments of the present invention describe methods and systems for speech-to-speech translation. Embodiments may be used to adapt to a user's voice and speech through model adaptation. In further embodiments, the user can correct recognition errors, and the system can clearly learn from the errors corrected by the user, thereby reducing the likelihood that these errors will occur again in the future. By either adding new words to the system or selecting a predefined dictionary optimized for a specific location or task, the present invention allows the user to add the vocabulary to his or her individual needs and environment. Make it customizable. When adding new words, the multimodal interface allows the user to modify and verify automatically generated translations and pronunciations. This allows the user to add new words to the system when the user has no knowledge of other languages. In some embodiments, the system is further configured to send any new vocabulary entered by the user to the user's community. This data is verified and a dictionary is automatically generated and can then be downloaded by any user.

FIG. 1 illustrates a block diagram overview of an example of a speech-to-speech translation system that can be maintained in the field according to the present invention. This example, the system operates between two languages L _a and L _b. This speech - including speech translation - speech conversion a typical implementation of a dialog system, both the sound from La to L _b and from L _b to L _a. However, bidirectionality in this configuration is not a requirement for the present invention. A one-way system from L _a to L _b , or several languages L ₁ . . . A multi-directional system including L _n would equally benefit from the present invention. System has two ASR modules 2 and 9, ASR modules 2 and 9, using the speech models 18, ASR class-based language model 19 and a recognition dictionary model 20 (shown in FIG. 3), _{L a} And L _b are recognized, and texts corresponding to L _a and L _b are generated. In this example, we used a speech recognition system “Ninja” developed by Mobile Technologies, LLC. Other types of ASR modules that may be used include speech recognizers developed by IBM Corporation (IBM Corporation), SRI, BBN, or in Cambridge or Aachen.

The system also includes two mechanical translation module 3 and 8, the machine translation module 3 and 8, the text respectively translating from _{L a} to _{L b} and from _{L b} to _{L a} (module 11). The MT module used in this example was a “PanDoRA” system developed by Mobile Technologies LLC. For other MT modules, modules developed by IBM Corporation, SRI, BBN, etc., or at Aachen University etc. can be used.

The two text-to-speech engines 4 and 7 each correspond to one of the machine translation modules 3 and 8 and are configured to receive text generated from the corresponding ASR unit. Output text is transferred to the corresponding MT module 3 or 8, MT modules 3 and 8 respectively translating _{L b} into and text text from _{L a} from _{L b} to _{L a.} The TTS module generates a speech output, and at least one text word of L _a to speech through an output device 5 such as a speaker, and at least one text word of L _b such as a device 5 or a speaker 6 Each audio is converted to audio via another output device. In this example, a Cepstral TTS module was used. Any TTS module that supports the Windows® SAPI (speech application programming interface) convention can be used as well.

  The modification and repair module 11 allows the user to modify the system's output via a number of modalities including voice, gestures, writing, antennae, touch sensitive and keyboard interface, and the system learns from the user's modifications. Make it possible to do. The correction and repair module may be of the type as disclosed in US Pat. No. 5,855,000. The user site customization module 12 provides an interface for the user to add new vocabulary to the system and can also select a system vocabulary appropriate to the current situation. For example, there is an interface that is triggered by a change of location, as determined by GPS coordinates indicating the current location of the device, or by explicit selection of a task or location by the user.

A user accesses the user site customization module 12 and through a graphical user interface display displayed on the screen (or active touch screen) of the device 13 and a pointing device 14 including a mouse or pen, the system Can communicate with. An example of a graphical user interface is shown in FIG. In this example, the device 13 displays the text and corresponding text for audio input L _a window 15. Machine translation in the second language _{L b} of text _{L a} is displayed in the window 16.

  In one embodiment, the same microphone and speaker are available in both languages. Therefore, the microphones 1 and 10 can be a single physical device and the speakers 5 and 6 can be a single physical device.

A flowchart illustrating an example operation in the method of the present invention is shown in FIG. Initially, in step 15b, the speech recognition system is activated by the user. For example, a button on a graphic user interface (FIG. 2) or an external physical button (not shown) can be selected. Then the voice of the user (item 25), by one of the ASR module in step 27, i.e., by the module 2 if the user is speaking the L _a, if the user is talking L _b is Recognized by module 9. The ASR modules 2 and 9 apply three models: a speech model 18, an ASR class-based language model 19 and a recognition dictionary model 20. These models are language specific and each ASR module includes its own set of models. In step 28, the text resulting from the user's voice is displayed via the GUI on the device screen 13.

  Next, translation is performed using the MT module 3 or 8 based on the input language (step 29). MT modules 3 and 8 have three main models: tagging or parsing to identify word classes [Collins02] model (model 22), class-based translation model (model 23), and class-based language model ( Model 24) is applied. The tagging model 22 may be any suitable type of tagging or parsing model, such as the types described in the following prior art documents: J. Lafferty, A. McCallum, and F. Pereira, " Conditional random fields: Probabilistic models for segmenting and labeling sequence data, "In Proceedings of 18th International Conference on Machine Learning, pages 282-289, 2001 (" Lafferty01 ") or Michael Collins," Parameter estimation for statistical parsing models: Theory and practice of distribution-free methods "(2004) In Harry Bunt, John Carroll, and Giorgio Satta, editors, New Developments in Parsing Technology, Kluwer. Other models applied during machine translation include distortion models and sentence length models that constrain how words are rearranged during translation. A detailed description of class-based machine translation is given below. The resulting translation is displayed on the device 13 via the GUI as shown in step 30.

  To help the user determine whether the translation output is correct, the automatically generated translation (FIG. 2, item 16) is translated back to the input language via the MT module 3 or 8. Then, for example, as shown in FIG. 2, item 15a, it is displayed in parentheses immediately below the original input. If the reliability of both speech recognition and translation is high (step 31) as determined by ASR model 2 or 9 and MT module 3 or 8, then via TTS module 4 or 7 (step 33), speaker 5 Or, via 6, an audio output (item 26) is generated. Otherwise, the system indicates that the translation may be wrong via GUI, voice and / or careful feedback. The particular TTS module used in step 33 is selected based on the output language.

  Thereafter, if the user is dissatisfied with the generated translation, the user may intervene during or after the speech-to-speech translation process in any of steps 27-33. Thereby, the correction and repair module 11 is called (step 35). The correction and repair module 11 records and logs any corrections that can be made by the user. This modification can later be used to update ASR modules 2 and 9 and MT modules 3 and 8, which are described in more detail further below this document. If the modification includes a new vocabulary item (step 36), the user enters a field customization mode in step 15c and explicitly adds a new word to the system, or in step 15d, Thomas Schaaf, “Detection of OOV words using generalized. User site customization if new words are automatically detected in the input speech using confidence standards or new word models, such as the methods described in word models and a semantic class language model, "in Proc. of Eurospeech, 2001 Module 12 is called. This module 12 provides a multi-modal interface and allows the user to add new words to the active system vocabulary. As new words or new phrases (new phrases) are added by the user, the ASR, MT and TTS models (items 17, 21 and 33a) are updated on demand. The functionality of this module is described further below for both languages.

  A common set of classes (eg, person names, place names, and organization names) is used in both ASR and MT in both languages. This common set provides a system-wide set of semantic slots that allows new words to be added to the system. The names, special terms and phrases that occur within these classes are highly variable terms depending on the various user locations, locations, behaviors, habits and tasks and therefore require the most user customization. .

  In the preferred example, the particular class used depends on the application area of the system. Class includes a semantic class for proper expression, i.e. person name, place name and organization name, or task specific noun phrase, e.g. food name, disease name, drug name, and a word or phrase that does not apply to any predefined class Another class that is open may be included. Syntactic classes such as synonyms or word equivalence classes can be used as well. Examples of application areas include, but are not limited to, travelers, medical practices, peacekeeping and the like. In one example, the classes required in a traveler's application area include names of people, cities, food, and the like. In another example, application classes required by medical professionals include disease names, drug names, anatomical names, and the like. In another example, classes required for peacekeeping applications include the names of weapons, vehicles, and the like. In order to enable on-site customizable speech translation, the system can correct errors and later learn from these errors through the operation of the correction and repair module 11 in combination with the user field customization module 12. To do.

[Correction and repair module]
The correction and repair module 11 allows the user to intervene at any time during the speech-to-speech translation process. The user may identify the error and log it, or correct the error in the speech recognition or translation output if he / she wants. Such user intervention is of considerable value because it provides immediate correction in the person-to-person communication process and the system provides the opportunity to adapt to the user's needs and concerns and learn from mistakes. A flow diagram illustrating the ability to feed back such errors is shown in FIG. If the user is dissatisfied with the translation of the statement (ie, an error has occurred), the user may log the current input (step 40). The system saves the current speech and other information in a log file. This allows the user to access and correct at a later time. Alternatively, the user uploads to the community database, allowing the expert user to identify and correct the error.

  Users can also modify speech recognition or machine translation output through a number of modalities. The user can modify the entire utterance by speaking again or by entering a sentence via the keyboard or handwriting interface. Alternatively, the user can highlight the erroneous part of the output hypothesis via the touch screen, mouse or cursor keys, using the keyboard, handwriting, or voice, or a word Correct only that phrase or word by spelling it clearly. The user selects the erroneous part in the output hypothesis via the touch screen and selects the competing hypothesis in the automatically generated drop-down list, or re-enters by voice, or It can also be modified by any other complementary modality (eg handwriting, spelling, paraphrasing, etc.). These methods, and how to properly combine complementary repair actions, are based on the method proposed by Wabel et al. In US Pat. No. 5,855,000 for multimodal speech recognition correction and repair. Here, the proposed method is applied to the speech recognition and translation module of the interactive speech translation system.

If the user modifies the speech recognition output (step 43), the system first determines whether this modification includes a new word (step 44). By matching the word language L _a and L _b and respective recognition dictionary model 20 associated, this determination is made. If the word is not found, the system prompts the user to add a new word to the active system vocabulary if desired (FIG. 5, step 50). Otherwise, the probability in the ASR model (FIG. 3, item 17) is updated again to reduce the likelihood that the same error will occur. This can be done in a characteristic way in which the probability of the modified word sequence is increased and the probability of similar competitive hypotheses is reduced.

  The user can modify the machine translation output if he has sufficient language expertise. The same modalities that are used in the case of ASR can be used. If the machine translation output is modified by the user (step 45), and if the modification includes a new word, the user is prompted by the dialog to add the new word to the active system vocabulary (FIG. 5, step 50). If the modification only includes words that are already in the active system vocabulary, the machine translation model (FIG. 3, item 21) is updated. Specifically, the phrase can be extracted from the corrected sentence pair and put into a translation model. The target language model used can be updated in a similar way as in ASR.

[User field customization module]
The user site customization module 12 enables the system to learn new words in cooperation with the user. Conventional systems do not allow users to modify vocabulary within a speech-to-speech translation system. Unlike conventional systems, the user site customization model 12 allows the user to make sequential modifications in the execution system, which requires minimal knowledge of computer speech and language processing techniques or linguistics. It is relatively easy to implement by non-experts who have none or none. Model 12 provides such field customization, which provides and receives some understandable feedback from the user and autonomously derives all necessary parameters and system configuration based on this feedback. Made by. The field customization module 12 accomplishes this through: 1) an intuitive interface for user-customization, and 2) automatically estimating all internal parameters and settings required for user customization, An internal tool that reduces the burden on the user.

For one-way translation, the system processes at least four pieces of information about a word or phrase and adds a new word or phrase to the active system vocabulary. This information includes:
-Class (ie the semantic or syntactic class of the new entry)
- word in the language _{L a} (ie written form in the _{L a)}
- word pronunciation of the L _a - word translation in _{L b} (ie written form in _{L b)}
For bidirectional translation, the system also requires the input of the new word pronunciation in L _b. L _b, on the contrary, TTS generates an audio output, ASR module for _{L b} is it possible to recognize new words.

  A flowchart illustrating the operational steps of the user site customization model 12 is shown, for example, in FIG. When the system encounters a new word, based on the correction and correction interventions made via the repair model 11 described in the previous section, the system will determine whether the user (FIG. 5, step 50) should “learn” this word. That is, it will prompt you to decide whether to add to the active system vocabulary. In that case, the word learning mode is activated and the on-site customization module 12 begins to operate. It should be noted here that field customization or learning new words does not require only the results of the error correction dialog. Specifically, the user may enter word learning mode from a pull-down menu and choose to add a new word or a priori new word list. Even if various words such as technical terms, names, places, etc. are suddenly needed by external events, learning new words can be started as well. However, in all such cases, the system must collect the information described above.

  After the user indicates that he wants to add a new word to the system vocabulary (step 50), the system first looks up a large external dictionary. This dictionary is either contained locally on the device, is a dictionary service accessible via the Internet, or a combination of both. The external dictionary is composed of entries of word translation pairs. Each entry includes pronunciation and word-class information that allows new words to be easily added to the active system vocabulary. Each entry also includes a description of each of the word pairs in both languages. This allows the user to select an appropriate translation of the word without knowledge of the target language. If the new word is included in an external dictionary (step 51), the system displays an alternative translation list of words that includes each explanation (step 52). If the user selects one of the predefined translations from the dictionary (step 53), the user can verify the pronunciation and other information provided by the dictionary (step 53a) and edit if necessary. The new word is then added to the active system vocabulary.

  In order to add a new word to the active system vocabulary, three steps are required (steps 59, 59a, 59b). Initially, the word and its translation are added to the ASR recognition dictionary of modules 2 and 9 (step 59). This word is added to this recognition dictionary 20 along with the pronunciation (s) provided by the external dictionary. If the user simply inputs this word, the probability of occurrence of this word is set to be larger than competing elements of the same class in the ASR class-based language model 19. This increases the possibility of words added by the user. Next, the word and its translation are added to the MT model (FIG. 3, item 21), allowing the system to translate the new word in both translation directions. Finally, this word is registered in the TTS pronunciation model (FIG. 3, model 33a), allowing the system to pronounce this word correctly in both languages.

  If the new word entered by the user is not included in the external dictionary, the system will automatically generate the information needed to register the word into the active system vocabulary and verify this information with the user. To do. First, the context of the surrounding word is used if available (step 54), and the new word class is estimated via the tagging model (FIG. 3, model 22). Next, pronunciations and translations of new words are automatically generated via either the rule-based model or the statistical model (step 55). The resulting information is then presented to the user via a multimodal interface (step 58). The system prompts the user to verify (step 58) or modify (step 57) the automatically generated translation or pronunciation. Finally, after the user verifies this information, the new word is added to the active system vocabulary (steps 59, 59a, 59b). To dynamically add (59) new words (specifically “word + pronunciation + word class”) to the ASR vocabulary 59, it is typically organized as a tree structure within ASR module 2 or 9. Stored) is retrieved and then updated to include new words. This allows new words to be dynamically added to the recognition vocabulary, and therefore immediately recognized if there is a next verbal statement. ASR systems do not require reinitialization or restart as in conventional systems.

  Similarly, new words (specifically “word + translation + wordclass”) can be added to the MT translation model (59a) and stored as a hash-map in translation model 23 (which is in MT module 3 and / or 8). Can be searched), and a new translation pair containing the new word and its translation, as well as the word class are added. This allows new words to be dynamically added to the MT modules 3 and / or 8, and the new words will be accurately translated in subsequent utterances. MT systems do not require re-initialization or restart as in conventional research.

  Estimating all of this information automatically is essential to allow non-expert users in the field to perform customization tasks. In the following, it will be described in detail how this critical information about the word is automatically estimated and then how it can be obtained from the user or intuitively verified.

[New word pronunciation and translation generation]
Users of speech-to-speech translation systems usually have limited or no knowledge of phonetics, linguistics, linguistics, and often even little knowledge of words and their usage in other languages. As such, the user cannot be expected to provide translations and all relevant information (pronunciation, orthography, word usage, etc.) for each new word that he wishes to add to the system. Therefore, when the user enters a new word, the system estimates the word-class and automatically generates word translation and pronunciation information in both languages.

  To register a new word in the active system vocabulary, a translation of this word and the pronunciation of both this word and its translation are required. The generation of this information can be executed as a three-step process, for example, as shown in FIG. First, the pronunciation of a word is generated (step 60). A translation is generated based on the word and its character sequence (step 61). Next, a new word pronunciation in the target language is generated using the information generated in the preceding step (step 62). Two examples of generating this information using various techniques in a Japanese-English field maintainable S2S translation system are shown on the left side of FIG. In order to add the new English word “Wheeling” (item 61) to the system, English pronunciation is first generated via machine learning (step 65). Machine learning may be performed by any suitable technique as described in the prior art literature: Damper, RI (Ed.), Data-Driven Techniques in Speech Synthesis. Dordrecht, The Netherlands: Kluwer Academic Publishers (2001). Next, a transliteration of this word in Japanese is automatically generated via statistical machine transliteration (step 66), and then a Japanese pronunciation is generated via manually defined rules (step 66). 67). Transliteration may be accomplished using any suitable statistical machine transliteration engine. For example, examples can be found in the following prior art documents: K. Knight and J. Graehl, Machine transliteration. Computational Linguistics 24 4 (1998), pp. 599-612, and HLT / NAACL-2007. Bing Zhao, Nguyen Bach, Ian Lane, and Stephan Vogel, "A Log-linear Block Transliteration Model based on Bi-Stream HMMs". The resulting information (item 68) is then verified by the user through the sound player and with the audio string before registering the word in the active system vocabulary.

  Similarly, in order to add a new Japanese word “Wakayama” (item 70) to the system, a Japanese pronunciation is first generated via a manually defined rule (step 71). Next, a transliteration of this word in Japanese is automatically generated via a rule-based transliteration (step 72), and then an English pronunciation is generated via a manually defined rule (step 73). ). Rule-based transliteration may be performed using the methods of the prior art literature: Mansur Arbabi, Scott M. Fischthal, Vincent C. Cheng, and Elizabeth Bar, "Algorithms for Arabic name transliteration," IBM Journal. of research and Development, 38 (2): 183-193, 1994. The resulting information (item 74) is then verified by the user before registering the word in the active system vocabulary.

  The user can verify the pronunciation by the generated translation and voice output. An alternative configuration is to give the native language if the user considers it more appropriately (ie, if the user is an English speaker, “Hanyu Pinyin” for Chinese or “Romaji” for Japanese) The description format may be used. The user may edit the translation and / or pronunciation if necessary. Once approved by the user, words and word features are added to the multilingual system dictionary.

  The system also eliminates the need for translation of each new word added to the dictionary by automatically generating the necessary information with the help of interactive user input. An example of the user interface is shown in FIGS.

[Interactive user interface]
The system then consults the user to confirm and verify the estimated linguistic information. This is done in an intuitive manner so as not to infer any special linguistic or technical knowledge. An appropriate interface is therefore used. In the following, the interaction with the user during the learning of new words will be described.

  In this interface, the user may select a “new word” mode from the menu, or can invoke a new word learning mode after a new / unknown word has been generated by user modification. In the window pane that appears, he / she can type the desired new word, name, special term, concept, wording. This is based on orthographic input in the user's language (which may be a character set different from English, eg, Chinese, Japanese, Russian, etc.). Next, the system generates a Roman transliteration and words predicted pronunciations. This is done by conversion rules learned from handwritten or extracted speech data or learned from transliterated speech data. The user can then inspect the automatic conversion and play the sound of the pronunciation generated via TTS. The user may iteratively correct any of these representations (letters in any language, transliterations written in Roman letters, phonetic notation, and phonetic phonetics) and other corresponding entries. Are regenerated as well (hence, a modified phonetic notation in one language may correct the phonetic notation in the other language).

  The system also automatically determines the most likely word class to which the new word belongs based on co-occurrence statistics of other words (with known classes) in similar contexts. select. The new word pane also allows this class specific manual selection (and / or modification), but allows the user to overrule any class judgment thus estimated.

In summary, given a new word / phrase from the user, the system operates as follows.
-Automatically classify the semantic class of entries (used by ASR and MT components).
- (used in ASR and TTS for _{L 1)} word automatically generates pronunciation.
Automatically generate translations of words (used by both MT components)
- (used in ASR and _{L 2} for the TTS) automatically generate pronunciation translation.
-Allows the user to modify / edit data automatically generated on demand.
-Provide other modalities that allow the user to verify that the automatically generated translation is correct (ie listen to the pronunciation of the word via TTS).

  If the user enters a word that does not match any of the predefined classes in the system, the user can assign this word to an “unknown” class. For ASR, an “unknown” class is defined by words found in the training data and not found in the recognition dictionary. For SMT, bilingual entries not found in the translation dictionary are set to unknown tags in the target language model.

[Increase in-class probability and relevance]
None of these input methods require linguistic training and provide an intuitive way for the user to determine whether a new word is properly represented. The user may then receive an entry for the new word by adding the new word to a “multilingual system-dictionary” or user-specific dictionary. The entire system integrates a standard dictionary and a customized dictionary into the user's runtime dictionary.

In addition to the five entries described above, an in-class probability P (w | C) is also defined. In this way, the system can identify multiple words belonging to the same class. Therefore, words that are closer to the user's tasks, preferences and habits are preferably assigned a higher in-class probability. Based on the relevance to the user, this increase to increase the in-class probability is determined, in which case the relevance is evaluated by observing the following points:
-New word entry and its up-to-date * Because he / she indicated that the user wanted a new word by entering a new word, the new word entered is of course likely to be used in the near future, and Hence the intra-class probability is raised (increased) over the alternative existing class entry
-Correlation between new words and user behavior, interests and tasks, including: * Distances to places such as city names, landmarks and interesting places * Past usage history * Co-occurrence statistics (Sushi is more than Bogotá) More related to Tokyo)
-General saliency of new words, including:
* Urban population * Recent mention in the media

  Such observation and relevance statistics are based on the location, history or behavior observed by the user and / or alternatively by observing the occurrence of new words in the system in a wide background language resource such as the Internet. Collected. Such statistics may be collected in a single language in a data rich language and applied to translation dictionaries and translation language models.

  New user actions and tasks over time reduce the similarity of such words and / or new information (such as appearing in different cities) subclasses of words In the case where the relevance of the word is reduced, the relevance of the raised words may be attenuated over time.

[Crossmodal entry]
Optionally, the new word is entered by one of the following:
-Speaking: The user speaks a new word. All information such as pronunciation and transliteration is estimated by a new word model, a translation model, and a background dictionary, as described above, except based on voice input. The system may interact verbally and select class identity and other relevant information.
-Spelling: The user spells a new word by voice. This input method generally improves the likelihood of an accurate transliteration while speaking. Spelling may be used complementary to the manner of speaking and other input modalities.
-Handwriting: The user enters a new word by hand. This input method generally improves the likelihood of an accurate transliteration while speaking. Handwriting may be used complementary to speaking, spelling, or other input modalities.
-Browsing: New words may be selected by interactive browsing. Here, the system searches the Internet for text that contains a history of the user's recent usage and / or a statistical profile similar to the recently selected and entered new word, and related related new words. May be proposed.

[Remote new word learning and shared dictionary development via the Internet]
All of the methods described in the previous section aim to allow an individual user to customize a speech translation system to his / her needs and tasks in the field. However, many of these user customizations can be useful to other users as well. In some embodiments, user customizations are uploaded to a large database of communities and names, special terms, or phrases are shared among stakeholders. Lexical entries, translations and class tags are collected and associated with similar interested communities. The user can then download these shared community resources and add them as resources to the user's own system.

  As an alternative configuration, the user may simply upload the translation in an incomplete state and may choose to request manual translation from the community. Other users provide online corrections and translations on a volunteer (or paid) basis for such inaccurate or incomplete pre-translational words or sentences, and their missing or inaccurate translations can do. The resulting modifications and translations are once again submitted to the updated and shared community translation database.

[Unmanaged adaptation]
After correction, repair, and new word learning, finally we get a corrected transcript and hence a true transcript or translation of the verbal sentence. A speech-to-speech translation device or system automatically provides the fact that such a ground truth is introduced, further adapting the ASR module (FIG. 1, module 2 or 9) to the primary user of the device. Can be used. Such adaptation is designed to improve the accuracy and usefulness of the device. Two unique methods for adaptation are performed. The first is the adaptation of the system to better recognize the user's speech; the speech model and pronunciation model adaptation; and the second is the adaptation to the user's speech style for language model adaptation. Profiles are used to store adaptation data for specific users and can be switched in the field.

[Class-based machine translation]
In the previous section, we explained error repair and new word learning. In these modules, the standard was class-based machine translation. Next, we will explain the detailed functions of such class-based machine translation.

[approach]
State-of-the-art machine translation systems perform translation on a word-level. This is evident from previous translation systems, including those described in the following three documents; (1) P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, 'Moses: Open source toolkit for statistical machine translation', In Proc. ACL, 2007 ("[Koehn07"); (2) D. Chiang, A. Lopez, N. Madnani, C. Monz, P. Resnik and M. Subotin, "The Hiero machine translation system: extensions, evaluation, and analysis, ", In Proc. Human Language Technology and Empirical Methods in Natural Language Processing, pp. 779-786, 2005 (" Chiang05 "); and (3) K. Yamada and K. Knight" A decoder for syntax-based statistical MT ". In Proc. Association for Computational Linguistics, 2002 (" Yamada02 "). Alignment is performed word-to-word; translation examples, ie phrase pairs, are matched at the word level; and a word-based language model is applied. The hierarchical translation module described in Chang 05 and the like, and the syntax-based translation model described in Yamada 02 and the like extend this by introducing an intermediate structure. However, these approaches still require accurate word matching. Since each word is treated as a separate entity, these models are not generalized to the first-looked word.

One embodiment of the class-based machine translation is a class-based statistical machine translation as follows, where the foreign language sentence f ^J ₁ = f ₁ , f ₂ ,. . . , _{F J} are different language ^e _I 1 ₌ ^e 1 by a hypothesis having a maximum likelihood searching the hypothesis ^{^ e} _{I 1} given _{by:, e} 2,. . . , Translated into e _I :
^ E ^I ₁ = argmaxP (e ^I ₁ | f ^J ₁ ) = argmaxP (f ^J ₁ | e ^I ₁ ) · P (e ^I ₁ )
A class may be a semantic class such as a proper expression, a syntactic class or a class composed of equivalent words or phrases. As an example, we describe a case where a named entity class is incorporated into the system.

The two largest information models applied during translation are the target language model P (e ^I ₁ ) and the translation model P (f ^J ₁ | e ^I ₁ ). In the class-based statistical machine translation framework, P (f ^J ₁ | e ^I ₁ ) is a class-based translation model (Fig. 3, model 23), and P (e ^I ₁ ) is a class-based language model (Fig. 3 and model 24).

  A class-based model for a statistical machine translation framework can be trained using the approach shown in FIG. Initially, training a corpus of sentence pairs is normalized (step 100), and a tagging model (FIG. 3, model 22) is used to tag the corpus (step 101). One approach to do this is described in Latterty01. In this step, sentences combined to form a training pair can be tagged independently, tagged together, or tags from one language can be inferred to the other language. After the entire training corpus is tagged, the words in the sentence pair are aligned (step 102). Alignment is accomplished using current approaches described in prior art documents such as: Franz Josef Och, Christoph Tillmann, Hermann Ney: "Improved Alignment Models for Statistical Machine Translation"; pp. 20-28; Proc. of the Joint Conf. of Empirical Methods in Natural Language Processing and Very Large Corpora; University of Maryland, College Park, MD, June 1999, and Brown, Peter R, Stephen A. Delia Pietra, Vincent J. Delia Pietra, and RL Mercer 1993. "The mathematics of statistical machine translation: Parameter estimation," Computational Linguistics, vol 19 (2): 263-311. In this step, collocation phrases within a tagged entity (ie, “New York”) are treated as a single token. Next, phrases are extracted using a method such as Koehn07 (step 103), and a class-based translation model is generated (FIG. 3, model 23). The tagged corpus is also used to train a class-based target language model (FIG. 3, model 24). Training may be accomplished using the methods described in prior art documents such as: B. Suhm and W. Waibel, "Towards better language models 'for spontaneous speech' in Proc. ICSLP-1994, 1994 (Suhm94 (Step 104).

  In order to translate the input sentence, the method illustrated in FIG. 11 is applied. Initially, the input sentence is normalized (step 105) and tagged using a manner as applied to the training corpus (step 106). Input sentences are tagged using a monolingual tagger (FIG. 3, model 22). Next, the input sentence is decrypted using the class-based MT model (FIG. 3, models 23 and 24). For class-based statistical machine translation, decryption is performed using the same method used in standard statistical machine translation, however, phrase pairs are as shown in the following example: Can be matched at class-level rather than word.

クラス内の語又は句（即ち：@PLACE.city{Wheeling}、 @TIME{4:30}）は、数／時間の事例の場合、直接通過するか、翻訳が翻訳モデルから決定される。ユーザは、「ユーザ現場カスタマイズモジュール」を介して翻訳モデルに新語を加えることができる（図１、モジュール１２）。次いで、（図６の例で詳述しているように）ユーザがあらかじめ都市名「Ｗｈｅｅｌｉｎｇ」を加えた場合、翻訳モデルは次の句も含むことになる：

Assume a tagged input statement as follows:
the train to @ PLACE.city {Wheeling} leaves at @TIME {4:30}
The following phrases can be matched:

The words or phrases in the class (ie: @ PLACE.city {Wheeling}, @TIME {4:30}) are passed directly in the case of a number / hour or the translation is determined from the translation model. The user can add new words to the translation model via the “user site customization module” (FIG. 1, module 12). Then, if the user has previously added the city name “Wheeling” (as detailed in the example of FIG. 6), the translation model will also contain the following phrase:

Assuming translation model probability P (f ^J ₁ | e ^I ₁ ) (FIG. 3, model 23) and MT class-based language model probability P (e ^I ₁ ) (FIG. 3, model 24), the search is the maximum likelihood. It is executed to find a translation hypothesis having a degree P (f ^J ₁ | e ^I ₁ ) · P (e ^I ₁ ).

これは入力文の正確な翻訳である。 Assuming the input sentence and phrase above, the resulting translation is as follows:

This is an accurate translation of the input sentence.

  In this example, the word “Wheeling” did not appear in the training corpus, but after the user entered the word via the “User Field Customization Module” (FIG. 1, module 12), the system Can be translated into Furthermore, since the word-class is known (“@ PLACE.city” in this example), the system can select a better translation for multiple words in the vicinity, and select multiple words in the translation output. It will be arranged in order exactly.

[Parallel tagging of multilingual corpus]
In one embodiment, a labeled parallel corpus is obtained by tagging each side of a training corpus having a single language tagger independently and then removing inconsistent labels from each sentence pair. In this approach, for each sentence pair (Sa, Sb), a label-sequence pair (Ta, Tb) having the maximum conditional probabilities P (Ta, Sa) and P (Tb, Sb) is selected. If the total number of occurrences of any class-tag is different between P (Ta, Sa) and P (Tb, Sb), the class-tag is removed from the label-sequence pair (Ta, Tb). One way to estimate P (Ta, Sa) and P (Tb, Sb) is by applying a conditional random field-based tagging model Luffy01. An example of a feature set used during monolingual tagging is shown in FIG.

  In some embodiments, in addition to single language features, the use of target words extracted from word-alignment (wb, j in FIG. 11) further increases the consistency of labeling across multiple sentence pairs. Can be improved.

単一言語モデルの性能が著しく異なる場合、λａ及びλｂを最適化して、二言語タグ付け性能を改善できる。 In another embodiment, both sentences in a translation pair are labeled together while applying the constraint that the set of class-tags must be equivalent. Specifically, for a sentence pair (Sa, Sb), we search for a label-sequence pair (Ta, Tb) that maximizes the combined maximum conditional probability.

If the performance of the single language model is significantly different, λa and λb can be optimized to improve bilingual tagging performance.

  In one embodiment, if a manually annotated corpus is not available for a particular language, the labels are not annotated from a known first language across all the sentence pairs in the training corpus A label can be generated by projecting the label into a language. One approach to do this is described in the prior art literature: D. Yarowsky, G. Ngai and R. Wicentowski, “Inducting Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora,” In Proc. HLT, pages 161-168, 2001 ("Yarowsky01").

[Class-based machine translation system and evaluation example]
Through experimental evaluation, we show that detailed class-based machine translation improves translation performance compared to traditional approaches. Furthermore, we show that using the parallel tagging approach described in section 2.2.2 further improves the accuracy of the translation.

The system for translation between Japanese and English developed for the tourist domain was evaluated. A description of the training data and test data is shown in Table 1.

Accurate and consistent tagging across multiple sentence pairs is extremely important to achieve an effective class-based SMT. We studied the following two approaches to improve tagging quality; first, the introduction of bilingual features from word-alignment; and second, both sides of a sentence pair are tagged together Bilingual tagging. 14,000 sentence pairs from the parallel training corpus were manually tagged using the 16 class labels shown in Table 2.

  From this manually labeled set, we selected 10% (1400 sentence pairs) containing one or more tags as held-out data to assess tagging accuracy.

Initially, the performance of the baseline monolingual CRF-based tagger was evaluated. Each side of the held-out set was independently labeled using a language-dependent model. The output was then compared to manual criteria. The tagging accuracy for the various metrics is shown in Table 3.

  In bilingual tagging, a tag is considered accurate if the entity is correctly labeled on both sides of the corpus. The right column shows the percentage of sentence pairs that are correctly tagged on both sides. Despite the fact that F scores above 0.90 for each language, the accuracy of bilingual tagging has dropped significantly to 0.84, and only 80% sentence pairs are correctly tagged. It's just that. Incorporating alignment features into a single language tagger improved accuracy for both languages and significantly improved recall for the Japanese side, but slightly increased the percentage of correctly tagged sentence pairs. I just did it. Removing inconsistent tags across all sentence pairs improved accuracy, but did not improve the number of correctly tagged sentence pairs.

  Next, the effectiveness of bilingual tagging was evaluated using the approach described above. The tagging accuracy of this approach and the case of incorporating word alignment features are shown in the bottom two rows of Table 3. Compared to the monolingual case, bilingual tagging significantly improved tagging accuracy. Not only is tagging consistency improved (F score for bilingual tagging increased from 0.84 to 0.95), but also tagging accuracy on both English and Japanese sides Improved. Incorporating word-alignment features resulted in a smaller improvement in tagging accuracy for all measurements.

  In addition, the effectiveness of the system was evaluated by comparing the performance of three class-based systems with a baseline system that did not use a class model.

For the baseline system, a phrase-based translation model was trained using the Moses toolkit, such as described in Koehn05 and GIZA ++ (which is used in prior art documents such as: Franz Josef Och) "A Systematic Comparison of Various Statistical Alignment Models", Computational Linguistics, volume 29, number 1, pp.19-51 March 2003. The 3-gram language model is the SRILM toolkit described in the following prior art documents: A. Stolcke "SRILM-an extensible language modeling toolkit", In Proc. Of ICSLP, pp. 901-904, 2002. Decoding was performed using a PanDoRA decoder. This decoder is described in the prior art document: Ying Zhang, Stephan Vogel, "PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-held Devices," In the Proceedings of MT Summit. XI, Copenhagen, Denmark, Sep. 10-14 2007. The system was created using the training set listed in Table 1 for both translation directions J → E (from Japanese to English) and E → J (from English to Japanese). The data used to train the target language model was limited to this corpus. The translation quality of the baseline system was evaluated on a test set of 600 sentences. One standard was used during the evaluation. The BLEU scores for the J → E and E → J systems were 0.4381 and 0.3947, respectively. BLEU scores are described in the prior art literature: Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu "BLEU: a Method for Automatic Evaluation of Machine Translation," In Proc. Association for Computational Linguistics, pp 311-318, 2002. Translation quality using three different tagging methods was evaluated:
+ Num: 8 classes related to number and time + NE-class: above class + 8 classes for proper expression + Bi-tagging: 16 classes above, training corpus tagged in two languages

  Monolingual tagging was applied to the + num and + NE-class cases, removing inconsistent tags across sentence pairs. In the + Bi-tagging case, bilingual tagging that incorporates word alignment features was used. For each of the tagging methods, the entire training corpus was tagged with an appropriate set of class-labels. The class-based translation model and language model were then trained using a method equivalent to that used in the baseline system. During the test, input sentences were tagged using a monolingual tagger. All named expressions in the test set were entered into the user dictionary for use during translation.

Table 4 shows the performance of the 600 sentence test set for the baseline and class-based systems for the J → E and E → J system BLEU scores.

  A class-based SMT system using number and time tags (+ num) obtained improved translation quality for both translation directions compared to the baseline system. For these models, BLEU scores of 0.4441 and 0.4104 were obtained. Translation quality was significantly improved when a class-based system using specific expressions in addition to number and time tags was applied. BLEU scores of 0.5014 for the J → E system and 0.4464 for the E → J case were obtained. When bilingual tagging was used to tag the training corpus (+ Bi-tagging), an additional 0.8 point gain was obtained for both translation directions in BLEU. In 14% of the sentences in the test set that contain one or more named expressions, the (+ Bi-tagging) system will replace the system tagged in a single language ("+ NE-class") with a maximum of 3.5 BLEU points. Was excellent.

  Although very detailed details have been described above, it should be understood that the drawings and detailed embodiments have been provided for purposes of illustration and not limitation. Variations in design and configuration may be made within the principles of the invention. Those skilled in the art will appreciate that such changes or modifications or combinations of elements, variations, equivalents or improvements of the invention still exist within the scope of the invention as defined in the appended claims. It will be.

Claims (19)

Receiving from a user a utterance in a first language that includes a first term associated with a term in the field;
Translating the speech from the first language into the second language in a speech translation system;
Receiving an instruction to add the first term relating to the field to a first recognition dictionary of a first automatic speech recognition module of the speech translation system;
Adding the first term related to the field and the translation of the statement into the second language to a first machine translation module related to the first language in the speech translation system;
Adding the first term associated with the field and the translation of the utterance into the second language to a shared database for a community associated with the field and associated with the field of the first term;
Including methods. Receiving a second term related to the field from the shared database;
Adding the second term associated with the field from the shared database to the first recognition dictionary and the first machine translation module associated with the first language in the speech translation system;
The method of claim 1 comprising: If the translation to the second language added to the shared database includes an inaccurate or incomplete translation;
The first term comprises the step of requesting to correct translation for the first term related to the field method of claim 1 associated with the field applied to the shared database. From the shared database, and obtaining a corrected translation of the first term associated with the field,
Adding the corrected translation from the shared database to the first machine translation module associated with the first language in the speech translation system;
The method of claim 1 comprising:   The method of claim 1, wherein a corrected translation of the first term associated with the field in the shared database is provided by a second user of the community.   6. The method of claim 5, wherein the shared database stores terms related to one or more fields, including one or more of names, wordings, and special terms.   Displaying simultaneously at least (i) the utterance of the received first language and (ii) the translation of the utterance into the second language on a user interface display of the speech translation system in text. Item 2. The method according to Item 1.   Translating the statement from the first language into the second language includes generating a translation into the second language via either a rule-based model or a statistical model; The method of claim 1. When an instruction to add the first term is received by the processor, word class information, pronunciation in the first language, translation into the second language, pronunciation in the second language for the first term A step of determining
Adding the determined word class information of the first term and the determined pronunciation in the first language to the first recognition dictionary of the first language in the first automatic speech recognition module. The method of claim 1. A computer program product comprising a non-transitory computer-readable recording medium containing computer program code,
The computer program code is
Receiving from a user a utterance in a first language that includes a first term associated with a domain;
Translating the speech of the first language into a second language in a speech translation system;
The first recognition dictionary for the first language of the first automatic speech recognition module of the speech translation system, comprising the steps of receiving an indication to apply said first term related to the field of in the first language,
Adding the first term related to the field and the translation of the statement into the second language to a first machine translation module related to the first language in the speech translation system;
Adding the first term associated with the field and the translation of the utterance into the second language into a shared database for the community associated with the field and associated with the field of the first term; ,
Is for running
The computer program product, wherein the first term relating to the field added to the shared database is accessible by the community. Receiving a second term related to the field from the shared database;
Adding the second term associated with the field from the shared database to the first recognition dictionary and the first machine translation module associated with the first language in the speech translation system;
The computer program product of claim 10, further comprising computer program code for executing The was added to a shared database, the translation into the second language, if it contains insufficient translation, in the first term associated with the field applied to the shared database, associated with the field The computer program product of claim 10, further comprising computer program code for performing the step of requesting that a translation be corrected for the first term. From the shared database, and obtaining a corrected translation of the first term associated with the field,
Adding the corrected translation from the shared database to the first machine translation module associated with the first language in the speech translation system;
The computer program product of claim 10, further comprising computer program code for executing The first term of the corrected translation is given by a second user of the community, computer program product of claim 10 associated with the field in the shared database.   The computer program product of claim 14, wherein the shared database stores terms related to one or more fields, including one or more of names, wordings, and special terms.   To simultaneously display at least (i) the utterance of the received first language and (ii) the translation of the utterance into the second language in text on a user interface display of the speech translation system. The computer program product of claim 10, further comprising computer program code.   Translating the statement from the first language into the second language includes generating a translation into the second language via either a rule-based model or a statistical model; The computer program product of claim 10. When an instruction to add the first term is received by the processor, word class information, pronunciation in the first language, translation into the second language, pronunciation in the second language for the first term A step of determining
Adding the determined word class information of the first term and the determined pronunciation in the first language to the first recognition dictionary of the first language in the first automatic speech recognition module; The computer program product of claim 10, further comprising computer program code. A speech translation system,
A computer processor;
Memory,
An automatic speech recognition module that receives from a user a utterance in a first language that includes a first term associated with a term in the field;
A translation module for translating the statement from the first language to the second language;
An instruction to add the first term related to the field to the first recognition dictionary of the first automatic speech recognition module of the speech translation system is received, and the first term related to the field and the second of the remarks. A user site customization module that adds language translations to a first machine translation module associated with the first language in the speech translation system;
With
The user site customization module relates to the first term associated with the field and the translation of the utterance into the second language for the community associated with the field of the first term. A speech translation system further configured to add to a shared database.

Images