JP5244663B2 - Speech recognition processing method and system for inputting text by speech - Google Patents

Description

  The present invention relates to a speech recognition processing method and system for inputting text by speech.

  There is software that converts a voice uttered by a user into text data using a terminal having a relatively high processing capability such as a personal computer. The terminal acquires the voice uttered by the user through the microphone. The voice is encoded into voice data. The voice data is converted into text data by voice recognition processing.

  In addition, in the case of a terminal having a relatively low processing capability such as a portable terminal, it is possible to recognize a voice of about several thousand vocabulary keywords. However, from the viewpoint of processing capability, a sentence corresponding to a large vocabulary of more than tens of thousands of words such as dictation cannot be recognized as speech.

  For this purpose, there is a technique for executing a voice recognition process by connecting a portable terminal to a voice recognition server via a network. According to this technology, the portable terminal transmits the encoded voice data to the voice recognition server in a batch by HTTP (HyperText Transfer Protocol). The voice recognition server converts voice data into text data by voice recognition processing. The converted text data is returned to the mobile terminal. Thereby, a process with a heavy load in the voice recognition process can be executed by the server. That is, even a portable terminal with low processing capability can perform high-accuracy speech recognition of a large vocabulary.

JP 2005-283972 A

  According to the prior art, the portable terminal transmits voice data to the voice recognition server in a batch using an HTTP request. On the other hand, the voice recognition server also converts the voice data into text data at once. Then, all the text data is returned in a batch with an HTTP response. By using an HTTP request and response sequence, it is possible to access a single voice recognition server from a plurality of portable terminals.

  However, the user cannot visually recognize the text data on the display while speaking to the microphone. In particular, the longer the input text is, the more delay occurs until the text data is displayed, resulting in less convenience. In addition, in the case of HTTP, since TCP (Transmission Control Protocol) is used as a lower protocol, it is error free but has a large overhead and a heavy load on the network.

  Accordingly, an object of the present invention is to provide a voice recognition method and system that can recognize voice data in real time and can reduce the load on the network as much as possible.

According to the present invention,
A session control server;
A terminal that activates call connection means for the session control server, text processing application, and voice input interface means for inputting voice data from the user;
A speech recognition processing method in a system having a call connection means for a session control server and a speech recognition server having speech recognition processing means for converting speech data into text data,
When the terminal activates the voice input interface means for the text processing application, the call connection means of the terminal recognizes the call connection request including the voice recognition parameters including the codec information and the voice recognition type through the session control server. After being transmitted to the server and decoded by the voice recognition server based on the codec information, the dictionary is switched using the voice recognition type, and the terminal receives the call connection acceptance response from the voice recognition server. Establishing a first session for speech data and a second session for text data with a speech recognition server;
A second step in which the terminal transmits voice data of a predetermined unit uttered by the user to the voice recognition server via the first session;
A third step in which the speech recognition server transmits the primary candidate text data converted using the speech recognition processing means to the terminal via the second session;
A fourth step in which the terminal and the voice recognition server continuously repeat the second step and the third step until the voice input by the user is completed;
When there is other candidate text data other than the primary candidate text data already transmitted by the voice recognition server when the voice input is finished by the user, one or more other candidates associated with the primary candidate text data text data, possess a fifth step of transmitting to the terminal,
The terminal is characterized in that the user selects and determines whether the primary candidate text data is the primary candidate text data or any other candidate text data .

  According to another embodiment of the speech recognition processing method of the present invention, for the first step, the first session for speech data is established by RTP (Realtime Transport Protocol) and the second for text data. The session is preferably established by TCP (Transmission Control Protocol).

According to the present invention,
In a system in which a terminal and a voice recognition server are call-connected by a session control server,
The terminal
A text processing application;
Voice input interface means for inputting voice data from a user;
When the voice input interface means for the text processing application is activated, a call connection request including a voice recognition parameter including codec information and a voice recognition type is transmitted to the voice recognition server via the session control server, and the call is received from the voice recognition server. Call connection means for establishing a first session for voice data and a second session for text data with the voice recognition server after receiving the connection acceptance response ;
Voice data transmitting means for transmitting voice data of a predetermined unit acquired by the voice input interface means to the voice recognition server via the first session;
The speech recognition server
Call connection means to the session control server;
Voice recognition processing means for performing decoding processing based on the codec information, switching a dictionary using a voice recognition type, and converting voice data into text data;
Text data transmitting means for transmitting the primary candidate text data to the terminal via the second session;
Voice recognition control means for continuously repeating the voice recognition processing means and the text data transmission means until the voice input by the user is completed;
When there is other candidate text data other than the primary candidate text data already transmitted by the voice recognition server when the voice input is finished by the user, one or more other candidates associated with the primary candidate text data text data, possess the other candidate storing means for transmitting to the terminal,
The terminal is characterized in that the user selects and determines whether the primary candidate text data is the primary candidate text data or any other candidate text data .

  According to another embodiment of the system of the present invention, it is also preferable that the first session for voice data is established by RTP and the second session for text data is established by TCP.

  According to the voice recognition method and system of the present invention, the mobile terminal transmits a predetermined unit of voice data as an RTP data stream and receives text data converted by voice recognition as a TCP data stream. As a result, the load on the network can be reduced as much as possible compared to the case of HTTP in which voice data and text data are transmitted and received in a batch.

  In addition, during the voice input, the mobile terminal displays the first candidate text data obtained by sequentially converting the voice data on the display, and displays the other candidate text data on the display after the voice input is completed. As a result, the voice data is recognized in real time during the voice input from the viewpoint of the user, and the optimum text data can be selected after the voice input is completed.

It is a 1st system block diagram in this invention. It is a functional block diagram of the terminal and voice recognition server in this invention. It is a flowchart in this invention. It is a 1st display screen example of the terminal in this invention. It is a 2nd example of a display screen of the terminal in this invention. It is an example of description of SDP of INVITE request. It is an example of description of SDP of INVITE response. It is a 2nd system block diagram in this invention. It is a 3rd system block diagram in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a first system configuration diagram according to the present invention.

  According to FIG. 1, the terminal 1 makes a call connection with the voice recognition server 2 via the session control server 3. The terminal 1 is a mobile terminal having a relatively low processing capability such as a mobile phone. The session control server 3 is a SIP (Session Initiation Protocol) server as a call control protocol, and is connected to a control network such as an IMS / MMD (IP Multimedia Subsystem / Multimedia Domain) network. The mobile terminal 1 is connected to the IMS / MMD network via an access network such as a mobile phone network.

  According to FIG. 1, the mobile terminal 1 activates, for example, mail software as a text processing application. Here, the user can input text into the editor of the mail software by speaking into the microphone of the mobile terminal 1.

  The voice recognition server 2 is call-connected from the portable terminal 1 via the SIP server 3. The voice recognition server 2 converts the voice data received from the portable terminal 1 into text data by voice recognition processing. The converted text data is returned to the portable terminal 1.

  Between the portable terminal 1 and the voice recognition server 2, an RTP session for voice data and a TCP session for text data as a recognition candidate are established. RTP is a protocol for transmitting data such as audio or moving images by streaming. TCP is a protocol for transmitting data such as files without error.

  RTP uses UDP (User Datagram Protocol) as a lower protocol. Therefore, it is also preferable to add an error correction code of FEC (Forward Error Correction) or MFT (Missing Feature Theory) to the RTP packet. Thereby, the influence on the recognition performance due to the packet loss is reduced.

  FIG. 2 is a functional configuration diagram of the terminal and the voice recognition server in the present invention.

  The mobile terminal 1 includes, as hardware, a communication interface unit 101, a microphone 102 that acquires voice uttered by a user, a display 103 that displays text data, and a key operation unit 104 that selects an operation and text. Have.

  The mobile terminal 1 includes, as software, a call connection unit 111, a transport interface unit 112, a text processing application 113, a voice input interface unit 114, a voice data transmission unit 121, a text data reception unit 122, And another candidate selection unit 123. These functional components are realized by executing a program that causes a computer mounted on the mobile terminal to function.

  The text processing application 113 has a text editor function, and may be, for example, a mail application. The text processing application 113 activates the voice input interface unit 114 with the voice recognition parameter as an argument. The speech recognition parameters are at least codec information for decoding processing and speech recognition type for dictionary switching. After completing the voice input, the text processing application 113 acquires text data from the voice input interface unit 114.

  The voice input interface unit 114 functions as a user interface. The voice input interface unit 114 acquires voice uttered by the user from the microphone 102 and encodes it into voice data using AMR (Adaptive Multi-Rate), EVRC (Enhanced Variable Rate Codec), or the like. Or you may convert into the audio | voice data which extracted the characteristic by signal processing. When the voice input is completed, the voice input interface unit 114 notifies the call connection unit 111 of the end of the voice input.

  Note that the voice input interface unit 114 functions in a superimposed manner with the text processing application 113. That is, the voice input interface unit 114 is implemented as middleware that can be commonly used by various applications. Therefore, when designing the text processing application 113, it is not necessary to consider speech recognition processing. Also,

  The call connection unit 111 functions as a client for the SIP server 3. When the voice input interface unit 114 is activated, the call connection unit 111 transmits an INVITE message (call connection request) including a voice recognition parameter to the voice recognition server 2 via the SIP server 3. The call connection unit 111 transmits an INFO message including control information for starting or ending voice input to the voice recognition server 2 via the SIP server 3 in accordance with an instruction from the voice input interface unit 114.

  The transport interface unit 112 establishes an RTP data stream for voice data and a TCP data stream for text data with the voice recognition server 2.

  The voice data transmission unit 121 transmits the predetermined unit of voice data acquired by the voice input interface unit 114 to the voice recognition server 2 via the RTP data stream.

  The text data receiving unit 122 receives text data obtained by voice recognition from the voice recognition server 2. During the voice input, the primary candidate text data is sequentially received. Further, after the voice input is completed, a combination of the primary candidate text data and one or more other candidate text data is received. The received text data is output to the text processing application 113.

  The other candidate selection unit 123 causes the user to select other candidate text data. After completing the voice input, the text processing application searches for primary candidate text data corresponding to the other candidate text data from the text data already displayed on the display. For the matched primary candidate text data, the other candidate text data is displayed on the display, and the user is allowed to select it.

  The voice recognition server 2 includes a communication interface unit 201, a call connection unit 211, a transport interface unit 212, a voice recognition processing unit 221, a text data transmission unit 222, another candidate accumulation unit 223, and a voice recognition control unit. 224. These functional components are realized by executing a program that causes a computer mounted on the server to function.

  The call connection unit 211 functions as a client for the SIP server 3. The call connection unit 211 acquires a voice recognition parameter from the received INVITE message. The speech recognition parameters are output to the speech recognition processing unit 221.

  The transport interface unit 212 establishes an RTP data stream for voice data and a TCP data stream for text data with the mobile terminal 1.

  The voice recognition processing unit 221 receives voice data via an RTP data stream, and converts the voice data into text data by voice recognition. Here, the speech recognition processing unit 221 outputs provisional primary candidate text data in the middle of utterance to the text data transmission unit 222 and outputs other candidate text data to the other candidate storage unit 223.

The voice recognition processing unit 221 converts the voice data into text data with reference to the dictionary and the language model. The speech recognition processing unit 221 includes, for example, an N-gram model that recognizes a sentence including “Tenanoha”. The N-gram model is a language model that statistically calculates probabilities from sample data. When N = 3 (trigram) is used to estimate the appearance probability P (w ₁ w ₂ ... W _n ) of a given word string w ₁ w ₂ ... W _n , P (w ₁ w ₂ ... W _n ) = ΠP (w _i | w _i−2 , w _i−1 ) × P (w ₁ w ₂ ) P (w _i | w _i−2 , w _i−1 ) on the right side represents the conditional probability that w _i comes next when the word w _i−2 and w _i−1 come. All products of P (w _i | w _i-2 , w _i-1 ) are calculated, and a combination of word strings in which P (w ₁ w ₂ ... W _n ) has the largest value is determined as a recognition result. To do.

  In the N-gram model, correlation with words before and after is used for speech recognition of a certain part during speech. For this reason, in order to obtain the speech recognition result of a certain part, the subsequent part of speech is required. The speech recognition result is determined after obtaining an utterance up to several words ahead of the utterance of the part. In other words, the voice recognition result is confirmed by a delay of several words after the portion is uttered.

  Therefore, the speech recognition processing unit 221 outputs the primary candidate text data in the N-gram model to the text data transmission unit 222. In addition, the speech recognition processing unit 221 outputs a combination of the primary candidate text data and other candidate text data determined several words ahead of the primary candidate text data to the other candidate storage unit 223.

  The text data transmission unit 222 transmits temporary primary candidate text data in the middle of utterance to the mobile terminal 1 via a TCP data stream.

  The voice recognition control unit 224 causes the voice recognition processing unit 221 and the text data transmission unit 222 to repeatedly function until the voice input by the user is completed.

  The other candidate accumulation unit 223 transmits a combination of the primary candidate text data and one or more other candidate text data to the mobile terminal 1 when the voice input by the user is completed.

  FIG. 3 is a flowchart in the present invention. Corresponding to the sequence of FIG. 3, FIG. 4 is a first display screen example of the terminal in the present invention. FIG. 5 is a second display screen example of the terminal according to the present invention.

(S301) The speech recognition server 2 registers the location information (AOR (Address-Of-Record), contact address) of the server in the SIP server 3 using the REGISTER method.

(S302) According to FIG. 4A, the text processing application is mail software, and the user is going to input text into the “body” of the mail.

(S303) According to FIG. 4B, the mail software editor is activated for the portable terminal 1. Then, the user selects the item “recognition start”.

(S304) According to FIG. 4C, the mobile terminal 1 instructs the user to speak into the microphone. At this time, the text processing application delivers the speech recognition parameters to the speech input interface unit. As a result, the voice input interface unit is activated.

(S305) The mobile terminal 1 registers the location information (AOR, contact address) of the terminal in the SIP server 3 using the REGISTER method. AOR is a logical address representing the location of the terminal in SIP. Here, AOR represents the address of the speech recognition server. The contact address is a real address of the mobile terminal and is associated with AOR. Thereby, a contact address can be searched from AOR. The AOR and the contact address are not necessarily one-to-one. By assigning a plurality of contact addresses to one AOR, it is possible to simultaneously send to a plurality of voice recognition servers.

(S311) The portable terminal 1 transmits a call connection request (INVITE) to the voice recognition server 2 via the SIP server 3. Here, in the SDP (Specification Description Protocol) of the INVITE request, an RTP data stream for voice data and a TCP data stream for text data are established between the mobile terminal 1 and the voice recognition server 2. It is described accordingly.

  FIG. 6 is a description example of the SDP of the INVITE request.

  “m” represents a data stream type, and “a” represents a parameter for the data stream. In the SDP of the present invention, an audio data stream (m = audio) and a text data stream (m = message) are described. Also, the payload type and codec / format are mapped. Furthermore, voice recognition parameters are set. As the speech recognition parameters, for example, a speech recognition type, a packet size, a transfer interval, and the number of text data output candidates are set.

  According to FIG. 6, various parameters are also set. For example, according to “gps”, a dictionary for voice recognition can be switched based on position information of a mobile terminal. Further, for example, according to “user”, the dictionary for voice recognition can be switched based on personal identification information or personal history information.

  The voice recognition server 2 determines the voice recognition parameters when receiving the INVITE request. If the voice recognition server 2 can accept the voice recognition parameter, the voice recognition server 2 returns an INVITE response.

  FIG. 7 is a description example of the SDP of the INVITE response.

  In the INVITE response, the port number on the voice recognition server side is described for each data stream.

(S312) An RTP data stream for voice data and a TCP data stream for text data are established between the mobile terminal 1 and the voice recognition server 2.

  If a session with the speech recognition server 2 has already been established at the start of speech input, the REGISTER method (S305) and the INVITE method (S311) are omitted.

(S321) The portable terminal 1 transmits an INFO message including control information for starting voice input to the voice recognition server 2 via the SIP server 3. The INFO method is used to change the setting of voice recognition parameters and notify control information (start / end / stop, error, etc.) of voice recognition processing.

(S322) The portable terminal 1 transmits a predetermined unit of voice data uttered by the user to the voice recognition server 2 via an RTP data stream. On the other hand, the voice recognition server 2 converts it into text data by voice recognition processing, and returns temporary primary candidate text data in the middle of utterance to the portable terminal 1 via the TCP data stream. Here, the text data sequentially returned is the primary candidate by the speech recognition process. The mobile terminal 1 receives the text data via the TCP data stream and simultaneously displays the text data on the display so that the user can visually recognize the text data.

  The predetermined unit in the audio data may be a transfer size designated by a parameter, and may be variably controlled by a buffering size or the like according to the network state.

According to FIG. 4D, “Good morning” recognized from the user's utterance is displayed.
According to FIG. 4E, “today” recognized from the user's utterance is displayed.
According to FIG. 4 (f), “avoidance” recognized from the user's utterance is displayed. In fact, although the user uttered in the meaning of “meeting”, it was recognized as “avoidance” as the primary candidate by the voice recognition processing.
According to FIG. 5 (a), “From 3rd PM” recognized from the user's utterance is displayed. In fact, the user was recognized as “primary from the afternoon” as the primary candidate by the voice recognition process, even though the user uttered in the meaning of “from 3 pm”.
According to FIG. 5B, “Start at the usual place” recognized from the user's utterance is displayed.

(S323) When the voice input by the user is completed, the mobile terminal 1 transmits an INFO message including the control information of the voice input end to the voice recognition server 2 via the SIP server 3.

  The voice recognition server 2 that has received the INFO message indicating the completion of voice input transmits the other candidate text data to the mobile terminal 1 when there is other candidate text data other than the primary candidate that has already been transmitted.

  According to FIG. 5C, the mobile terminal 1 receives the other candidate text data “Membership is” and “Meeting is” for the primary candidate text data “Avoid”. At this time, the primary candidate text data “Avoid is” already displayed on the display is searched, and an anchor is displayed at that position. Then, the user selects which is the correct text data among “evasion is”, “membership is” and “meeting”. Here, “Meeting” is selected.

  According to FIG. 5 (d), the mobile terminal 1 receives the other candidate text data “Compliance” and “Third” for the primary candidate text data “Third”. At this time, the primary candidate text data “tertiary” already displayed on the display is searched, and an anchor is displayed at that position. Then, the user is made to select which is the correct text data among “tertiary”, “compliance”, and “3 o'clock”. Here, “3 o'clock” is selected.

(S324) When the user finishes selecting text data, the mobile terminal 1 delivers the text data to the text processing application. This completes the input of text data to the text processing application.

(S325) The mobile terminal 1 disconnects from the voice recognition server 2 by the BYE method, and ends the session.

(S326) Finally, the mobile terminal 1 deletes the location registration of the mobile terminal using the REGISTER method.

  FIG. 8 is a second system configuration diagram according to the present invention.

  The system shown in FIG. 8 is an application example to an IP telephone or a telephone conference system. For example, the user's own or the other party's utterance is recognized as speech to obtain text data. The text data can be stored as a memo or minutes, or transferred to a third party by e-mail.

  FIG. 9 is a third system configuration diagram in the present invention.

  The system of FIG. 9 is an example applied to a television caption system. For example, when a broadcast program or a video streaming program is viewed by an IP television receiving terminal, the sound of the broadcast content is recognized and captions are displayed as text data on a portable terminal.

  As described above in detail, according to the speech recognition method and system of the present invention, the mobile terminal transmits a predetermined unit of speech data as an RTP data stream, and transmits text data converted by speech recognition to the TCP. Received in the data stream. As a result, the load on the network can be reduced as much as possible compared to the case of HTTP in which voice data and text data are transmitted and received in a batch.

  In addition, during the voice input, the mobile terminal displays the first candidate text data obtained by sequentially converting the voice data on the display, and displays the other candidate text data on the display after the voice input is completed. As a result, the voice data is recognized in real time during the voice input from the viewpoint of the user, and the optimum text data can be selected after the voice input is completed.

  In the various embodiments of the present invention described above, various changes, modifications, and omissions in the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Mobile terminal, terminal, mobile phone 101 Communication interface unit 102 Microphone 103 Display 104 Key operation unit 111 Call connection unit 112 Transport interface unit 113 Text processing application 114 Voice input interface unit 121 Voice data transmission unit 122 Text data reception unit 123 Other Candidate selection unit 2 Voice recognition server 201 Communication interface unit 211 Call connection unit 212 Transport interface unit 221 Speech recognition processing unit 222 Text data transmission unit 223 Other candidate storage unit 224 Speech recognition control unit 3 SIP server, session control server

Claims (4)

A session control server;
A terminal that activates call connection means for the session control server, a text processing application, and voice input interface means for inputting voice data from a user;
A speech recognition processing method in a system comprising: a call connection means for the session control server; and a speech recognition server having speech recognition processing means for converting the speech data into text data,
When the terminal activates the voice input interface means for the text processing application, the call connection means of the terminal sends the call connection request including a voice recognition parameter including codec information and a voice recognition type to the session. The voice recognition server transmits to the voice recognition server, the voice recognition server performs a decoding process based on the codec information, and the dictionary is switched using the voice recognition type. A first step of establishing a first session for voice data and a second session for text data with the voice recognition server after receiving a call connection acceptance response from the voice recognition server ;
A second step in which the terminal transmits voice data of a predetermined unit uttered by a user to the voice recognition server via a first session;
A third step in which the voice recognition server transmits the primary candidate text data converted by using the voice recognition processing means to the terminal via a second session;
A fourth step in which the terminal and the voice recognition server continuously repeat the second step and the third step until voice input by the user is completed;
When speech input by the user is completed, if there is other candidate text data other than the primary candidate text data that the speech recognition server has already transmitted , one or more corresponding to the primary candidate text data other candidate text data, have a a fifth step of transmitting to said terminal,
A speech recognition processing method, characterized in that the terminal makes a user select and determine whether the primary candidate text data is the primary candidate text data or which other candidate text data .   Regarding the first step, the first session for voice data is established by RTP (Realtime Transport Protocol), and the second session for text data is established by TCP (Transmission Control Protocol). The speech recognition processing method according to claim 1. In a system in which a terminal and a voice recognition server are call-connected by a session control server,
The terminal
A text processing application;
Voice input interface means for inputting voice data from a user;
When the voice input interface means for the text processing application is activated, the call connection request including a voice recognition parameter including codec information and a voice recognition type is transmitted to the voice recognition server via the session control server, Call connection means for establishing a first session for voice data and a second session for text data with the voice recognition server after receiving a call connection acceptance response from the voice recognition server ;
Voice data transmitting means for transmitting voice data of a predetermined unit acquired by the voice input interface means to the voice recognition server via a first session;
The voice recognition server
Call connection means for the session control server;
Voice decoding processing means for performing decoding processing based on the codec information, switching a dictionary using the voice recognition type, and converting the voice data into text data;
Text data transmitting means for transmitting primary candidate text data to the terminal via a second session;
Voice recognition control means for continuously repeating the voice recognition processing means and the text data transmission means until the voice input by the user is completed;
When speech input by the user is completed, if there is other candidate text data other than the primary candidate text data that the speech recognition server has already transmitted , one or more corresponding to the primary candidate text data other candidate text data, possess the other candidate storing means for transmitting to said terminal,
The system wherein the terminal determines and confirms whether the primary candidate text data portion is the primary candidate text data or which other candidate text data . 4. The system according to claim 3 , wherein the first session for voice data is established by RTP and the second session for text data is established by TCP.

Images