JP7500057B2 - Communication management device and method - Google Patents

Description

An embodiment of the present invention relates to technology that supports communication (sharing awareness, communicating, etc.) using voice and text.

One example of voice communication is a transceiver. A transceiver is a wireless device that has both radio wave transmission and reception functions, allowing one user to talk to multiple users (one-way or two-way information transmission). Examples of transceivers in use can be seen at construction sites, event venues, hotels, inns, and other facilities. Another example of a transceiver in use is a taxi radio.

JP 2013-187599 A

The objective is to provide a communication system that forms a communication group including agents that communicate status or situation changes, and supports the communication of information between multiple users.

In the communication system of the embodiment, a user's speech is broadcast to the mobile communication terminals of other users through a mobile communication terminal carried by each of the users. The communication system includes a communication management device to which each of the mobile communication terminals is connected by wireless communication, and an agent device to which detection information output from a state detection device to be monitored is input and connected to the communication management device. The communication management device includes a communication control unit having a first control unit that broadcasts speech data received from the mobile communication terminal to each of the other multiple mobile communication terminals, and a second control unit that accumulates the speech recognition results obtained by speech recognition processing of the received speech data in chronological order as a communication history between users and controls text distribution so that the communication history is displayed synchronously on each of the mobile communication terminals. The agent device includes a speech text transmission unit that generates an agent speech text based on the detection information and transmits it to the communication management device. The communication control unit then broadcasts the synthetic voice data of the agent utterance text generated by the synthetic voice processing to each of the multiple mobile communication terminals, and stores the received agent utterance text in chronological order by including it in the communication history between users, and controls the text distribution to each of the mobile communication terminals.

FIG. 1 is a network configuration diagram of a communication system according to a first embodiment. 1 is a block diagram illustrating the configuration of a communication management device, an agent device, and a user terminal according to a first embodiment. 5A and 5B are diagrams illustrating an example of user information and group information according to the first embodiment. 5 is an example of a screen displayed on a user terminal according to the first embodiment. FIG. 4 is a diagram illustrating an example of setting management information according to the first embodiment. FIG. 2 is a diagram showing a processing flow of the communication system of the first embodiment. FIG. 2 is a diagram showing a processing flow based on a first example of the communication system of the first embodiment. FIG. 11 is a network configuration diagram of a communication system according to a second embodiment. FIG. 11 is a block diagram showing the configuration of a communication management device, an agent device, and a user terminal according to a second embodiment. FIG. 11 is a diagram showing a processing flow based on a second example of the communication system of the second embodiment. 13 is an example of a screen displayed on a user terminal according to the second embodiment. FIG. 13 is a diagram for explaining an example of an individual call mode interrupt process during a group call mode in the third embodiment. FIG. 13 is a block diagram showing the configuration of a communication management device, an agent device, and a user terminal according to a third embodiment. FIG. 13 is a diagram illustrating an example of specific notification setting information according to the third embodiment. FIG. 13 is a diagram showing a processing flow based on a third case of the communication system of the third embodiment.

First Embodiment
1 to 7 are diagrams for explaining the first embodiment. FIG. 1 is a network configuration diagram of a communication system according to this embodiment. The communication system provides an information transmission support function using voice and text, centered around a communication management device (hereinafter referred to as a management device) 100. Below, an aspect in which the communication system is applied will be described, taking facility management as an example.

The management device 100 is connected via wireless communication to user terminals (mobile communication terminals) 500 carried by each of a number of users, and broadcasts the user's speech to the other user terminals 500.

The user terminal 500 is, for example, a portable mobile terminal such as a multi-function mobile phone such as a smartphone, a PDA (Personal Digital Assistant), or a tablet terminal. The user terminal 500 has a communication function, a calculation function, and an input function, and connects to the management device 100 via wireless communication through an IP (Internet protocol) network or a mobile communication network to perform data communication.

The range in which the user's speech is broadcast to multiple other user terminals 500 (or the range in which the communication history, described later, is displayed synchronously) is set as a communication group, and each of the user terminals 500 of the target users (on-site users) is registered. As shown in FIG. 1, in this embodiment, the agent device 300 that receives detection information output from a state detection device (sensor device 1) of a monitored object in facility management and connects to the management device 100 by wireless or wired communication is registered as a member (agent) of the communication group in which multiple users are registered.

When the object to be monitored is a hot spring, the state of the hot spring is, for example, the temperature. In this case, the state detection device is a measuring device such as a temperature sensor 1. The temperature sensor 1 outputs the detected temperature to the agent device 300 as detection information. When the detected temperature is input, the agent device 300 generates an agent utterance text based on the detected temperature and transmits it to the management device 100. In other words, the agent device 300 is a device that makes utterances based on the detection information as a member of a communication group, similar to the user who carries the user terminal 500, and is positioned as a speech agent that speaks on behalf of the state detection device.

The agent device 300 is a desktop computer, tablet computer, or laptop computer. The agent device 300 has a data communication function for wireless or wired communication via an IP network or a mobile communication line network, and a calculation function (CPU, etc.). The agent device 300 can also be configured to have a display device (or a touch panel type display device) and character input means. The agent device 300 may also be a dedicated device equipped with each of the functions of this embodiment.

The communication system of this embodiment supports the transmission of information for sharing awareness and communicating, on the premise that multiple users can converse hands-free. At the same time, a communication group is formed including an agent that communicates the status or changes in situation of monitored objects in facility management, and the agent's speech function helps to improve the efficiency of obtaining and transmitting information on the status or changes in situation of monitored objects, which was previously done manually.

In particular, facility equipment management is supported by human labor, and there is always work to operate and control equipment. Such equipment operation and control requires continuous checking of the equipment's status and condition, which requires the user to visit the equipment to check its status or to visit the installation site of the status detection device to check the detection information. This requires a great deal of labor. Meanwhile, in recent years, attention has been focused on the linkage between sensor devices and the operation and control of equipment through the Internet of Things (IoT), but due to cost issues and other factors, as mentioned above, the reality is that this is supported by human labor.

In this embodiment, when equipment is operated and controlled manually, a mechanism is introduced in which sensor devices, etc. that output detection information to grasp the state and situation of the equipment, make speech based on the detection information as a member of a user communication group, thereby reducing the labor of the user. At the same time, a simple and low-cost system configuration is realized in which existing sensor devices and other status detection devices can easily join a user communication group simply by installing an agent device 300 that receives detection information at the facility management site.

Figure 2 is a block diagram of the configuration of the management device 100, the agent device 300, and the user terminal 500.

The management device 100 includes a control device 110, a storage device 120, and a communication device 130. The communication device 130 manages communication connections and controls data communications between the multiple user terminals 500, and controls broadcast communications to simultaneously send the same spoken voice and spoken text to the multiple user terminals 500.

The control device 110 includes a user management unit 111, a communication control unit 112, a voice recognition unit 113, and a voice synthesis unit 114. The storage device 120 includes user information 121, group information 122, communication history (communication log) information 123, a voice recognition dictionary 124, and a voice synthesis dictionary 125.

The agent device 300 is connected wirelessly or wired to a status detection device (sensor device 1) installed in the equipment to be managed, and includes a sensor information acquisition unit 320 that receives detection information output from the status detection device via a communication unit 310. The agent device 300 also includes a control unit (determination unit) 330, a spoken text transmission unit 340, a setting management unit 350, and a storage unit 360.

The user terminal 500 includes a communication/call unit 510, a communication App control unit 520, a microphone 530, a speaker 540, a display input unit 550 such as a touch panel, and a memory unit 560. Note that the speaker 540 is actually composed of earphones or headphones (wired or wireless), etc.

Figure 3 is a diagram showing an example of various information, where user information 121 is user registration information for users who use this communication system. The user management unit 111 controls the setting of user IDs, user names, attributes, and groups through a specified management screen. The agent device 300 is also user-registered. Group information 122 is group identification information that partitions communication groups. The sending and receiving of communication information and broadcast distribution are controlled by communication group ID, and information is controlled to not be mixed between different communication groups. In the user information 121, the communication groups registered in the group information 122 can be linked to each user.

The user management unit 111 of this embodiment provides a function to set up a communication group in which multiple users are registered, and which is subject to the first control (broadcast distribution of spoken voice data) and the second control (broadcast distribution of agent spoken text and/or text of user spoken voice recognition results) described below, and also provides a function to register the agent device 300 in the communication group.

Regarding grouping, a facility can be divided into multiple departments for management depending on the facility in which the communication system of this embodiment is introduced. For example, in the case of an accommodation facility, bellpersons (luggage carriers), concierges, and housekeepers (cleaning) can each be set in different groups, and a communication environment can be created in which guest room management is subdivided for each group. From another perspective, there may be cases where communication is not required due to roles. For example, food servers and bellpersons (luggage carriers) can be separated into groups because they do not need to communicate directly. There may also be cases where communication is not required due to geographical reasons. For example, if branch A and branch B are geographically separated and there is no need for frequent communication, the groups can be separated.

Therefore, it is possible to set up a mixture of various communication groups, such as a communication group in which an agent device 300 is registered, a communication group in which an agent device 300 is not registered, and a communication group in which multiple agent devices 300 are registered. If there are multiple pieces of equipment to be managed within a facility, an agent device 300 can be installed separately for each piece of equipment. Also, if multiple status detection devices are installed in the same facility, an agent device 300 can be installed for each status detection device and registered in one communication group.

The communication control unit 112 of the management device 100 functions as each of the first and second control units. The first control unit controls the broadcast distribution of the speech data received from the user terminal 500 to each of the other multiple user terminals 500. The second control unit accumulates the speech recognition results obtained by processing the received speech data into speech recognition data in chronological order as the communication history 123 between users, and controls text distribution so that the communication history 123 is displayed synchronously on each user terminal 500.

The function of the first control unit is to broadcast speech data. The speech data includes speech data artificially generated by speech synthesis processing from text (e.g., agent utterance text) and speech data uttered by the user. The speech synthesis unit 114 uses the speech synthesis dictionary 125 to synthesize speech data corresponding to the characters of the agent utterance text, generating speech synthesis data. At this time, the material of the speech data that constitutes the speech synthesis data is arbitrary.

The function of the second control unit is to broadcast the agent's spoken text and the user's spoken voice recognition results. In this embodiment, all voices input to the user terminal 500 and voices played back on the user terminal 500 are converted to text and stored in chronological order in the communication history 123, and are controlled so that they are displayed synchronously on each user terminal 500. The voice recognition unit 113 performs voice recognition processing using the voice recognition dictionary 124, and outputs text data as the spoken voice recognition result. Publicly known technology can be applied for the voice recognition processing.

The agent device 300 is equipped with a speech text sending unit 340 that generates agent utterance text based on the detection information output from the state detection device and sends it to the management device 100. As a first control function, the communication control unit 112 of the management device 100 performs synthetic speech processing on the agent utterance text received from the speech text sending unit 340 to generate synthetic speech data of the agent utterance text, and distributes it to each of the multiple user terminals 500. At the same time, as a second control function, the agent utterance text received from the speech text sending unit 340 is included in the communication history 123 between users, accumulated in chronological order, and controls the distribution of text to each user terminal 500.

The communication history information 123 is text-based log information in which the content of each user's utterance and the agent utterance text of the agent device 300 are accumulated in chronological order along with time information. The audio data corresponding to each text can be stored as an audio file in a specified storage area, and the communication history 123 records, for example, the storage location of the audio file. The communication history information 123 is generated and accumulated for each communication group.

Figure 4 is a diagram showing an example of the communication history 123 displayed on each user terminal 500. Each user terminal 500 receives the communication history 123 from the management device 100 in real time or at a specified timing, and the display is synchronized between multiple users, allowing past communication logs to be referenced in chronological order.

In addition, in the display field D, a voice mark M can be displayed for text that corresponds to the synthetic voice data, and a microphone mark H can be displayed for the speaker's own spoken text in the display field D.

As shown in the example of FIG. 4, each user terminal 500 displays the content of the agent device 300's speech in chronological order in the display field D along with the user's own speech and the speech of other users, and the communication history 123 accumulated in the management device 100 is shared as log information.

Figure 5 is a diagram showing an example of setting management information used by the agent device 300. The conditions under which the agent device 300 will speak and the contents of the spoken text are registered as setting management information. The control unit 330 functions as a judgment unit that judges whether or not the detection information satisfies the judgment conditions based on the judgment conditions set in the setting management information.

In the example of FIG. 5, in "setting 1," the condition is set as follows: temperature is less than 36 degrees, and the agent utterance text is "The temperature has fallen below 36 degrees." In "setting 2," the condition is set as follows: temperature is 42 degrees or more, and the agent utterance text is "The temperature has exceeded 42 degrees." The control unit 330 uses the detection information acquired at any time interval by the sensor information acquisition unit 320 to match it with the judgment conditions set in the setting management information, and determines whether the judgment conditions are met.

When the control unit 330 determines that the judgment condition is satisfied, the spoken text sending unit 340 extracts the spoken text from the setting management information, generates agent spoken text data, and sends it to the management device 100.

The setting management information can be input through a management information registration screen provided on the agent device 300, or a setting management information file in which multiple pairs of different judgment conditions and speech texts are recorded can be created on another computer device and stored in the agent device 300.

Figure 6 shows the processing flow of the communication system of this embodiment.

Each user starts the communication app control unit 520 on the user terminal 500, and the communication app control unit 520 performs connection processing with the management device 100. Then, the user enters his/her user ID and password on a specific login screen to log in to the management device 100. The login authentication processing is performed by the user management unit 111. After logging in, each user terminal 500 performs information acquisition processing with the management device 100 at any timing or at a specific time interval.

When user A speaks, the communication app control unit 520 collects the spoken voice and transmits the spoken voice data to the management device 100 (S501a). The voice recognition unit 113 of the management device 100 performs voice recognition processing on the received spoken voice data (S101) and outputs the voice recognition result of the spoken content. The communication control unit 112 stores the voice recognition result in the communication history 123 and stores the spoken voice data in the storage device 120 (S102).

The communication control unit 112 broadcasts the speech data of user A to each of the other user terminals 500 other than the user A who made the speech. In addition, the communication control unit 112 transmits the speech content (text) of user A stored in the communication history 123 to each user terminal 500 in the communication group including user A himself/herself for display synchronization (S103).

The communication app control unit 520 of each user terminal 500 other than user A performs an automatic playback process of the received speech voice data, and while outputting the speech voice (S502b, S502c), displays the speech content in text format corresponding to the speech voice output in the display field D.

Next, the agent device 300 monitors the detection information output from the status detection device, and if it is determined that the detection information satisfies the judgment condition, the speech text sending unit 340 generates agent utterance text based on the judgment result and sends it to the management device 100 (S301).

At this time, the agent utterance text may or may not include detection information such as sensor values. In other words, it is sufficient if the content notifies that the state satisfies the judgment condition, and it may be utterance text that does not include the sensor value itself, such as "The temperature is dropping" or "The temperature is too high." In addition, the agent utterance text can be generated to include the sensor value, such as "The temperature has dropped below 36 degrees. The current temperature is 35.1 degrees." By including the actual measured value, it is possible to inform the user whether emergency action is required or whether there is time to respond.

The communication control unit 112 of the management device 100 stores the received agent utterance text in the communication history 123 (S104), and the voice synthesis unit 114 generates synthetic voice corresponding to the agent utterance text (S105), and stores the generated synthetic voice data in the storage device 120.

The communication control unit 112 broadcasts the spoken voice data of the agent device 300 to each of all user terminals 500 registered in the communication group. It also transmits the agent spoken text stored in the communication history 123 to each user terminal 500 in the communication group for display synchronization (S106).

The communication app control unit 520 of each user terminal 500 performs automatic playback processing of the received agent's speech voice data, and while outputting the speech voice (S503a, S503b, S503c), displays the agent's speech content in text format corresponding to the speech voice in the display field D.

Figure 7 shows the processing flow based on the first example in which the communication system of this embodiment is applied.

As shown in FIG. 7, the sensor information acquisition unit 320 of the agent device 300 acquires hot spring temperature information output by the status detection device (sensor device 1) at any timing or at a predetermined time interval (S3001). Each time hot spring information is acquired, the control unit 330 performs a process of determining whether the hot spring temperature satisfies the judgment condition registered as the setting management information (S3002).

If the temperature of the hot spring meets the judgment condition (YES in S3003), the speech text sending unit 340 extracts the speech text set in the setting management information and generates agent speech text data "The temperature has fallen below 36 degrees" (S3004). The speech text sending unit 340 sends the generated agent speech text to the management device 100 (S3005).

The speech synthesis unit 114 of the management device 100 generates synthetic speech data of the received agent utterance text (S1001). In addition, the communication control unit 112 of the management device 100 stores the agent utterance text received from the agent device 300 in chronological order by including it in the communication history 123 between users (S1002).

To synchronize the display, the communication control unit 112 transmits the agent utterance text in text format to the user terminal 500 (S1003), and simultaneously distributes the synthesized voice data of the agent utterance content to each of the multiple user terminals 500 (S1004).

The communication app control unit 520 of each user terminal 500 displays the agent's utterance content in text format in the display field D, and performs automatic playback processing of the synthetic voice data to output the voice. At this time, the same agent's utterance content is displayed synchronously in the display field D of each user terminal 500, and the agent's utterance content "The temperature has fallen below 36 degrees" is output as voice.

Next, when User C, having heard the agent's speech, says, "I can't take my hands off the phone," the communication App control unit 520 collects the speech and transmits the speech data to the management device 100. The speech recognition unit 113 of the management device 100 performs speech recognition processing on the received speech data (1005) and outputs the speech recognition result of the speech content. The communication control unit 112 stores the speech recognition result in the communication history 123, and stores the speech data in the storage device 120 (S1006).

The communication control unit 112 broadcasts the speech data of user C to each of the other user terminals 500 other than the user C who made the utterance (1008). In addition, the content of user C's utterance, "I'm busy right now," stored in the communication history 123, is transmitted to each user terminal 500 in the communication group including user C himself for display synchronization (S1007).

The communication app control unit 520 of each user terminal 500 performs an automatic playback process of the received speech voice data, outputs the speech voice of "I can't take my hands off the phone," and displays the text-format speech content "I can't take my hands off the phone" corresponding to the speech voice output in display field D. Note that the management device 100 controls the user terminal 500 of user C who made the speech so that the speech voice data uttered by the user C is not transmitted.

When User B, who has heard User C's statement, says, "I'm right next to you, so I'll handle it," the communication app control unit 520 collects the spoken voice and transmits the spoken voice data to the management device 100. The voice recognition unit 113 of the management device 100 performs voice recognition processing on the received spoken voice data (1009) and outputs the voice recognition result of the spoken content. The communication control unit 112 stores the voice recognition result in the communication history 123, and stores the spoken voice data in the storage device 120 (S1010).

The communication control unit 112 broadcasts the speech data of user B to each of the other user terminals 500 other than user B who made the utterance (1012). In addition, the communication control unit 112 transmits the speech content of user B stored in the communication history 123, "I'm right next to you, so I'll handle it," to each user terminal 500 in the communication group including user B himself, for display synchronization (S1011).

The communication app control unit 520 of each user terminal 500 performs automatic playback processing of the received speech voice data, outputs the speech voice "I'm right next to you, so I'll take care of it," and displays the text-format speech content "I'm right next to you, so I'll take care of it" corresponding to the speech voice output in display field D. At this time, too, the management device 100 controls the user terminal 500 of user B who made the speech so that the speech voice data uttered by the user himself is not transmitted.

Second Embodiment
8 to 11 are diagrams for explaining the second embodiment. Fig. 8 is a network configuration diagram of a communication system according to this embodiment. In contrast to the first embodiment, the communication system of this embodiment provides an agent function in response to a question from a user spoken at a user terminal 500. Note that the same components as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

Figure 9 is a block diagram of the communication management device 100, agent device 300, and user terminal 500 of this embodiment. Compared to Figure 2 of the first embodiment, the configuration of the agent device 300 has been partially changed and added, and when triggered by a user's utterance on the user terminal 500, the agent device 300 generates agent utterance text based on the detection information and transmits it to the management device 100.

Specifically, the communication control unit 111 of the management device 100 has a function of transmitting the voice recognition result of the spoken voice received from the user terminal 500 to the agent device 300. The agent device 300 has a text receiving unit 370 that receives the voice recognition result of the user's spoken voice, a text analysis unit 380 that analyzes the voice recognition result in text format, and a control unit (information providing unit) 330A that determines whether to provide agent utterance text based on the analysis result of the text analysis unit 380. The utterance text sending unit 340 generates agent utterance text based on the determination result of the control unit 330A and sends it to the management device 100.

Figure 10 shows the processing flow based on the second example of the communication system of this embodiment.

As shown in FIG. 10, when user C says, "Tell me the current temperature of hot spring B," the communication app control unit 520 collects the spoken voice and transmits the spoken voice data to the management device 100. The voice recognition unit 113 of the management device 100 performs voice recognition processing on the received spoken voice data (1005) and outputs the voice recognition result of the spoken content. The communication control unit 112 stores the voice recognition result in the communication history 123, and stores the spoken voice data in the storage device 120 (S1006).

The communication control unit 112 broadcasts the speech data of user C to each of the other user terminals 500 other than the user C who made the utterance (1008). Meanwhile, the speech content of user C stored in the communication history 123, "Tell me the current temperature of hot spring B," is transmitted to each user terminal 500 in the communication group including user C himself for display synchronization, and the speech content in text format, "Tell me the current temperature of hot spring B," is also transmitted to the agent device 300 (S1007A).

The agent device 300 receives the spoken text "Tell me the current temperature of hot spring B" via the text receiving unit 370. The received spoken text is analyzed by the text analysis unit 380, for example, by performing a well-known morphological analysis to extract keywords (S3101). For example, the keywords "Hot spring B," "temperature," and "tell me" are extracted.

The control unit (information provision unit) 330A of the agent device 300 performs information provision determination processing using keywords that are the analysis results of the text analysis unit 380 (3102). For example, as setting management information, the name of the object managed by the agent device 300 (Hot Spring B), detection attributes (temperature) detected by a status detection device connected to the agent device 300, and example question sentence information ("Tell me," "Huh?", "How many," "I want to know") are registered. Note that in this embodiment as well, the registration processing of this setting management information is performed through the setting management unit 350.

The control unit (information provision unit) 330A determines whether the voice recognition result of user C includes keywords related to the state detection device or a question about the detection information, and if it is determined that the keywords are included (YES in S3103), it acquires the detection information through the sensor information acquisition unit 320 (3001). In the above example, the voice recognition result of user C includes "B Hot Spring", and also includes the detection attribute "temperature" and the question "tell me", so the control unit 330A outputs "OK" as the information provision judgment result.

In the above explanation, it is assumed that multiple agent devices 300 are registered in a communication group, and in order for each agent device 300 to determine whether a question is directed at itself, the inclusion of the name of the object managed by the agent device 300 is used as a determination factor. However, for example, if only one agent device 300 appears in a communication group, it is possible to configure the system so that detection information is obtained from the status detection device in response to a user utterance such as "Tell me the temperature." In addition, the name of the status detection device (temperature sensor) etc. can be registered as information provision determination information, and the agent device 300 can be configured to make a statement based on the detection information in response to a question from user C such as "What's the temperature sensor reading?".

When the control unit 330A judges that the temperature is acceptable, the sensor information acquisition unit 320 of the agent device 300 acquires the hot spring temperature information output by the state detection device (sensor device 1) (S3001). The speech text transmission unit 340 extracts the speech text set in the setting management information, and generates agent speech text data "The current temperature is 37.5 degrees" (S3004). The speech text transmission unit 340 transmits the generated agent speech text to the management device 100 (S3005). At this time, the standard phrase "The current temperature is XX degrees" can be registered as setting management information, and the "XX" part can be replaced with the detection information "37.5" to generate the agent speech text.

The speech synthesis unit 114 of the management device 100 generates synthetic speech data of the received agent utterance text (S1001). In addition, the communication control unit 112 of the management device 100 stores the agent utterance text received from the agent device 300 in chronological order by including it in the communication history 123 between users (S1002).

To synchronize the display, the communication control unit 112 transmits the agent utterance text in text format to the user terminal 500 (S1003), and simultaneously distributes the synthesized voice data of the agent utterance content to each of the multiple user terminals 500 (S1004).

The communication app control unit 520 of each user terminal 500 displays the agent's utterance content in text format in the display field D, and performs automatic playback processing of the synthetic voice data to output the voice. At this time, the same agent's utterance content is displayed synchronously in the display field D of each user terminal 500, and the agent's utterance content "The current temperature is XX degrees" is output as voice.

Next, when User C, who has heard the agent's speech, says, "It's higher than the reference temperature, please turn on the boiler," the communication App control unit 520 collects the speech and transmits the speech data to the management device 100. The speech recognition unit 113 of the management device 100 performs speech recognition processing on the received speech data (1009) and outputs the speech recognition result of the speech content. The communication control unit 112 stores the speech recognition result in the communication history 123, and stores the speech data in the storage device 120 (S1010).

The communication control unit 112 broadcasts the speech data of user C to each of the other user terminals 500 other than the user C who made the utterance (1012). In addition, the content of user C's utterance stored in the communication history 123, "It's higher than the reference temperature, but please turn on the boiler," is transmitted to each user terminal 500 in the communication group including user C himself for display synchronization (S1012).

Figure 11 is an example of a screen displayed on the user terminal 500 of this embodiment. As shown in Figure 11, each user terminal 500 chronologically displays in display field D the contents of the user's own speech and the contents of speech of other users, as well as the contents of speech in response to questions or calls made to the agent device 300, and the contents of speech made by the agent device 300 triggered by the questions or calls. The communication history 123 stored in the management device 100 is then shared as log information.

In this embodiment, the agent device 300 understands the user's questions and calls, and generates and provides agent utterance text based on the detection information of the status detection device each time. Therefore, the agent device 300 appears as a pseudo user in the communication group, and a communication environment that is closer to a conversation in which users communicate information between each other can be provided.

In addition to the facilities listed above, other examples include building facilities in the security industry and berths (departure and arrival points) in the logistics industry. In addition to temperature sensors, various other detection devices can be used as condition detection devices depending on the application scenario of this communication system.

For example, a camera is an example of a status detection device. Images captured by the camera are used to analyze and determine people's movements and the degree of congestion. Analysis results such as "a large number of people have moved to the bath area" or "there is a line at the front desk" are used as triggers for the agent device 300 to send agent utterance text in response to these analysis results to the management device 100, and a synthetic voice notification and a text display notification can be made to the user terminal 500. As another example of congestion, for example, the degree of congestion in a parking lot can be analyzed and determined, and a synthetic voice notification and a text display notification can be made to the user terminal 500, such as "the parking lot will soon be full" or "please prepare the second parking lot."

The agent device 300 can also be configured to have a function for extracting a specific person from a camera image. In this case, for example, a matching process can be performed between a preregistered person image and a captured image, and an analysis result such as "who arrived where" can be obtained using information on the location where the camera, which is a status detection device, is installed. Such an analysis result can be used as a trigger to cause the agent device 300 to output agent utterance text such as "Mr./Ms. ~ is at ~", and a notification can be sent to each user terminal 500 in a synthetic voice via the management device 100.

As another example, a weight sensor can be used as a status detection device. For example, by linking with a weight sensor used in elevators, etc., and detecting five or more occurrences of weight exceeding the limit within a ten-minute period, the agent device 300 will output an agent speech text such as "The elevator is crowded," and notify each user terminal 500 (each user) by synthetic voice via the management device 100. Each user can then head out to direct pedestrian traffic as necessary.

Furthermore, a GPS device (position information detection device) can be applied as the state detection device. For example, a GPS device is attached to a cart pulled by human power, and the agent device 300 is configured to be able to obtain the position information of the cart from the GPS device. The agent device 300 can match the current position of the cart with a preset route or a prohibited entry place, and detect a predetermined range of deviation from the route or an intrusion into a prohibited entry place. When the agent device 300 detects these, it outputs an agent speech text such as "Is the route wrong?" or "That is a prohibited entry place," and notifies each user terminal 500 (each user) by synthetic voice through the management device 100. At this time, it is assumed that not only the user of the user terminal 500 but also facility users will intrude into the prohibited entry place. In this case, the user of each user terminal 500 who receives the notification can head to the prohibited entry place and appropriately guide the facility users.

The communication management device 100 can also be configured to have the functions of the agent device 300. That is, the functions of the agent device 300 shown in FIG. 2 or FIG. 9 are provided as an agent unit within the communication management device 100. Then, the detection information by the status detection device is configured to be transmitted to the communication management device 100. In this case, the status detection device may be internally equipped with a data communication function, or the status detection device may be connected to a separate data communication device so that the detection information of the communication management device 100 can be transmitted via the data communication device. The agent unit of the communication management device 100 can receive detection information output from the monitored status detection device, generate agent utterance text based on the detection information, and operate as a member of a communication group, similar to the above embodiment.

Third Embodiment
12 to 15 are diagrams for explaining the third embodiment. Note that the same components as those in the first and second embodiments are given the same reference numerals and the description thereof will be omitted.

The communication management device 100 of this embodiment has an individual call function in addition to the group call function described above. FIG. 12 is a diagram for explaining an example of an individual call mode interrupt process during group call mode in this embodiment. As shown in FIG. 12, a synthetic voice based on the agent utterance text sent from the agent device 300 is transmitted only to a specific user in the communication group during the group call.

As described above, the agent device 300 is registered as a member (agent) in a communication group. In this embodiment, an individual call function is provided between the agent and a specific user through the management device 100.

Figure 13 is a block diagram of the management device (communication management device) 100, agent device 300, and user terminal 500 of this embodiment. As shown in Figure 13, the first control unit and second control unit described in the first and second embodiments are shown as a group call control unit 112A. The communication control unit 112 is configured to include a group call control unit 112A and an individual call control unit 112B.

The management device 100 generates and stores a group member list that includes multiple users registered in a communication group. The individual call control unit 112B selects a relevant user from the group member list based on an individual call request sent from the agent device 300.

The individual call control unit 112B provides an individual call function that transmits speech voice data to a specific user only among users in a communication group that is broadcast in a group call. The individual call control unit 112B performs call processing to make a call to a specified user so that the agent device 300 can make a one-to-one call with the specific user through the management device 100 during group call mode. The call processing is an interrupt process for the maintained group call mode, and when the user responds to the call processing, a call connection process (establishment process of an individual call communication channel) is performed. This starts the distribution process of speech voice data from the agent to the specific user only through the established call channel. All of these processes are performed as individual call interruption processing to make a call while maintaining the group call state within the communication group and separating the specific user from other users in the communication group.

The individual call function of this embodiment can also be applied between two users other than agents. That is, the management device 100 can distribute a group member list including multiple users registered in a communication group to each user terminal 500 in advance. Then, when a user to be an individual call partner is selected from the group member list, the user terminal 500 can send an individual call request including the selected user to the management device 100. The individual call control unit 112B performs call processing for the selected user and can establish an individual call communication channel based on the response action of the called user.

In addition, even if the individual call control unit 112B is not in group call mode, it can accept an individual call request, open an individual call channel with a specified or selected user, and provide a one-to-one call function.

After the individual call ends, an automatic return process to the group call mode maintained within the communication group can be performed. The automatic return process is performed by the communication control unit 112. When a disconnection operation for the individual call mode is performed on the user terminal 500, the communication control unit 112 performs a disconnection process of the established individual call channel, and automatically returns to the call channel of the active group call mode. In addition, it may be configured to automatically return to the group call mode in conjunction with the disconnection process of the individual call communication channel from the individual call control unit 112B side.

The call duration in the individual call mode (call start time, call duration after call answering, and call end time) is stored in the management device 100 as an individual call mode execution history, together with the history of the individual call partner. In addition, as in the group call mode, the voice data uttered during the individual call can be processed through voice recognition and converted into text, and stored individually in the communication history information 123, or linked to the passage of time in the communication history information 123. The voice data uttered during the individual call mode can also be stored in the storage device 120.

In this way, the management device 100 (communication device 130) of this embodiment performs broadcast communication control to simultaneously send speech voice data by one user and text information of the speech content (text information obtained by processing the speech voice data into voice recognition) to multiple user terminals 500 in response to the group call function. Also, in response to the individual call function, it performs individual distribution communication control of speech voice data between specific users (individual call users).

Next, the agent device 300 can hold in advance the specific notification setting information shown in FIG. 14. As shown in FIG. 14, the situation determination conditions are set, and specific users to be contacted in individual calls are determined for each condition. In addition, the content of the contact (agent utterance text) is also set in advance.

The specific notification setting information shown in FIG. 14 is information in which the user to be contacted (specific user, user characteristics) and the line type (individual call, group call) as the contact method are added to the setting management information in FIG. 5 in the first and second embodiments. The judgment conditions in FIG. 5 correspond to the situation judgment conditions in FIG. 14.

Figure 15 shows the processing flow based on the third example of the communication system of this embodiment.

The control unit (determination unit) 330 of the agent device 300 receives detection information output from the sensor device (status detection device) 1 to be monitored (S3001) and matches it with the "status determination condition" of the specific notification setting information (S3002). It determines whether the received detection information satisfies the status determination condition (S3003), and if it is determined that the status determination condition is satisfied (YES in S3003), it extracts a pre-set speech text (S3004) and sends a contact request including information on the user to be contacted, the line type, and the speech text to the management device 100 (S3005).

When the management device 100 receives a contact request from the agent device 300, the voice synthesis unit 114 generates voice synthesis data for the received agent utterance text (S1001).

Next, the communication control unit 112 checks whether an individual call setting is made to the specific user by referring to the line type and the specific user who is the contact included in the accepted contact request (S1001A). If the line type is "group call", the process proceeds to step S1002, and the contact process is made in group call mode, not individual call mode (S1003, S1004). At this time, the spoken text, etc. are accumulated in chronological order in the communication history 123 (S1002).

In step S1001A, if it is determined that an individual call setting for a specific user exists (YES in S1001A), the individual call control unit 112B performs individual call mode (interrupt) processing for the specific user included in the contact request for the current group call mode (S1001B). Specifically, a call process for the specific user is performed using an individual call communication channel (1001C). The specific user who receives the call performs an operation to answer the incoming call (S504a). When the specific user performs an operation to answer the incoming call, the management device 100 performs call processing to establish an individual call line between the management device 100 and the specific user through the individual call communication channel (S1001D). The individual call control unit 112B delivers voice synthesis data of the agent's utterance text to the user terminal 500 of the specific user through the individual call line. Thus, contact between the agent and the specific user through an individual call line is realized.

Note that a specific user who has switched to individual call mode is treated the same as a "hold" on the group call channel, and can automatically return to the group call communication channel after the individual call ends. The communication control unit 112 also stores the contact history of the specific user who used the individual call mode in the communication history 123 (S1002).

In addition, there may be two or more individual call partners for an agent. In this case, a separate individual call channel can be established for each specific user, and voice synthesis data based on the agent's speech text can be delivered to each of them. Also, different agent speech texts can be set for different individual call partners. That is, as in the example of Figure 14, the agent speech text can be set for the floor manager as "The temperature has fallen below the threshold. A notification will be sent to the set user requesting action," and the agent speech text can be set for a qualified person (e.g., a boiler engineer) as "Please adjust the temperature as soon as possible." Voice synthesis data based on different speech texts will be delivered to the floor manager and the qualified person for the same situation determination condition.

In addition, the contact user does not have to be a preset user. As in the example of FIG. 14, the location information of each user (user terminal) can be known in advance, and one or more users close to the location where the event occurred can be determined as specific users to respond to the event that occurs when the situation determination conditions are met. In the example of FIG. 14, when entry into a no-entry area is detected, a specific user can be selected according to the user's location information, and speech synthesis data for the spoken text "Sensor detection has occurred in a no-entry area. Please respond as a nearby user" can be delivered to the selected specific user.

As described above, the management device 100 can be configured to have the functions of the agent device 300. As a modification of this embodiment, the management device 100 is configured to have an agent function unit equivalent to the agent device 300. The management device 100 can then be configured to receive detection information from the sensor device 1, perform steps S3002, S3003, and S3004 in FIG. 15, and communicate in individual call mode during a group call.

The present embodiment has been described above, but each function of the communication management device 100 and the agent device 300 can be realized by a program, and computer programs prepared in advance to realize each function are stored in an auxiliary storage device, and a control unit such as a CPU reads the program stored in the auxiliary storage device into a main storage device, and the control unit executes the program read into the main storage device, thereby operating the functions of each unit.

The above program can also be provided to a computer in a state in which it is recorded on a computer-readable recording medium. Examples of computer-readable recording media include optical disks such as CD-ROMs, phase-change optical disks such as DVD-ROMs, magneto-optical disks such as MO (Magnet Optical) and MD (Mini Disk), magnetic disks such as floppy (registered trademark) disks and removable hard disks, and memory cards such as Compact Flash (registered trademark), Smart Media, SD memory cards, and memory sticks. Also included as recording media are hardware devices such as integrated circuits (IC chips, etc.) that are specially designed and configured for the purposes of the present invention.

Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

100 Communication management device 110 Control device 111 User management unit 112 Communication control unit (first control unit, second control unit)
112A Group call control unit (first control unit, second control unit)
112B Individual call control unit 113 Voice recognition unit 114 Voice synthesis unit 120 Storage device 121 User information 122 Group information 123 Communication history information 124 Voice recognition dictionary 125 Voice synthesis dictionary 130 Communication device 300 Agent device 310 Communication unit 320 Sensor information acquisition unit 330 Control unit (determination unit)
330A Control Unit (Information Providing Unit)
340 Speech text transmission unit 350 Setting management unit 360 Storage unit 370 Text reception unit 380 Text analysis unit 500 User terminal (mobile communication terminal)
510 Communication/talk unit 520 Communication App control unit 530 Microphone (sound collection unit)
540 Speaker (audio output unit)
550 Display/input unit 560 Memory unit D Display field

Claims (9)

A communication system in which a user's speech is broadcast to other users' mobile communication terminals through a mobile communication terminal carried by each of a plurality of users,
a communication management device to which each of the mobile communication terminals is connected by wireless communication; and an agent device to which detection information output from a state detection device of a monitoring target is input and which is connected to the communication management device,
The communication management device includes:
a communication control unit including a first control unit that broadcasts speech data received from a mobile communication terminal to each of a plurality of other mobile communication terminals, and a second control unit that accumulates a speech recognition result obtained by performing speech recognition processing on the received speech data in chronological order as a communication history between users, and controls text distribution so that the communication history is displayed synchronously on each of the mobile communication terminals;
the agent device includes an utterance text sending unit that generates an agent utterance text based on the detection information and sends the agent utterance text to the communication management device;
The communication control unit broadcasts synthetic voice data of the agent utterance text generated by synthetic voice processing to each of the multiple mobile communication terminals, includes the received agent utterance text in the communication history between users, and accumulates it in chronological order, and controls text distribution to each of the mobile communication terminals. the communication management device includes a user management unit in which a plurality of the mobile communication terminals are registered and which sets a communication group to be controlled by the first control unit and the second control unit;
2. The communication system according to claim 1, wherein said user management unit provides a function for registering said agent device in said communication group. the agent device further includes a control unit that determines whether the detection information satisfies a preset judgment condition,
3. The communication system according to claim 1, wherein the speech text sending unit generates the agent speech text when it is determined that the detection information satisfies the determination condition. the communication control unit transmits the speech recognition result to the agent device;
The agent device includes:
a text receiving unit for receiving the speech recognition result;
and an information providing unit that determines whether or not to provide the agent utterance text based on the speech recognition result,
4. The communication system according to claim 1, wherein the utterance text sending unit generates the agent utterance text based on a result of the determination by the information providing unit, and sends the generated utterance text to the communication management device. The communication system according to claim 4, characterized in that the information providing unit determines whether the speech recognition result includes a keyword related to the state detection device or a question about the detection information. The communication control unit is
an individual call control unit for transmitting speech voice data to only specific users in a communication group to which the speech voice data is distributed by broadcasting,
The communication system according to any one of claims 1 to 5, characterized in that the individual call control unit performs individual call control to transmit synthetic voice data of the agent utterance text generated by synthetic voice processing to a specific user. A communication method in which a user's utterance voice is broadcast to the mobile communication terminals of other users through a mobile communication terminal carried by each of a plurality of users, the mobile communication terminals being connected to a communication management device by wireless communication, and an agent device to which detection information output from a state detection device of a monitored object is input is connected to the communication management device;
A first step in which a communication management device broadcasts speech voice data received from a mobile communication terminal to each of a plurality of other mobile communication terminals;
a second step of storing a speech recognition result obtained by speech recognition processing of the received speech data in a chronological order as a communication history between users by the communication management device, and controlling text distribution so that the communication history is displayed synchronously on each of the mobile communication terminals;
a third step of the agent device generating an agent utterance text based on the detection information and transmitting the agent utterance text to the communication management device;
The first step includes broadcasting synthetic voice data of the agent utterance text generated by synthetic voice processing to each of the plurality of mobile communication terminals;
The communication method is characterized in that the second step includes storing the received agent utterance text in chronological order by including it in the communication history between users, and controlling delivery of the text to each of the mobile communication terminals. A program executed by a management device that is connected to a mobile communication terminal carried by each of a plurality of users via wireless communication and broadcasts a user's speech to the mobile communication terminals of other users,
A first function of broadcasting speech voice data received from a mobile communication terminal to each of a plurality of other mobile communication terminals;
a second function of accumulating a speech recognition result obtained by performing speech recognition processing on the received speech data in chronological order as a communication history between users, and controlling text distribution so that the communication history is displayed synchronously on each of the mobile communication terminals;
a third function of receiving detection information output from a state detection device of a monitored object, receiving an agent utterance text based on the detection information generated by an agent device connected to the management device, and generating synthetic voice data of the agent utterance text,
The first function distributes synthetic voice data of the agent utterance text to each of the plurality of mobile communication terminals by broadcasting the synthetic voice data,
The second function is a program that includes the received agent utterance text in the communication history between users, accumulates the text in chronological order, and controls delivery of the text to each of the mobile communication terminals. A communication system in which a user's speech is broadcast to other users' mobile communication terminals through a mobile communication terminal carried by each of a plurality of users,
a communication control unit including a first control unit that broadcasts speech data received from a mobile communication terminal to each of a plurality of other mobile communication terminals, and a second control unit that accumulates speech recognition results obtained by speech recognition processing of the received speech data in chronological order as a communication history between users, and controls text distribution so that the communication history is displayed synchronously on each of the mobile communication terminals;
an agent unit that receives detection information output from a state detection device of a monitored object and generates an agent utterance text based on the detection information;
The communication control unit broadcasts synthetic voice data of the agent utterance text generated by synthetic voice processing to each of the multiple mobile communication terminals, includes the received agent utterance text in the communication history between users, and accumulates it in chronological order, and controls text distribution to each of the mobile communication terminals.

Images