JP7528638B2 - Communication System - Google Patents

Description

The present invention relates to a communication system.

Patent document 1 discloses a technology that uses image information, etc. to detect the degree to which a user is participating in communication with those around them.

JP 2008-046801 A

There was a need to realize a communication system that could detect the user's level of participation in communication with those around them while suppressing a decrease in detection accuracy, even without using image information, which has a high information processing cost.

The present disclosure has been made to solve such problems, and aims to provide a communication system that can detect the user's degree of participation in communication with those around them while reducing information processing costs and suppressing a decrease in detection accuracy.

A communication system including a determination device connected to at least three wearable devices, the determination device comprising:
an utterance determination unit that determines a speaker who is speaking among the users of the wearable device based on voice data detected by the connected wearable device;
an interest action determination unit that determines whether each user of the wearable device is nodding based on acceleration data detected by the wearable device;
and an interest determination unit that calculates a time transition of nodding density, which is the number of nods in a specified period of time, during a speech section of the speaker, and calculates a correlation of the time transition of the nodding density between non-speakers.

This disclosure provides a communication system that can detect the degree of a user's participation in communication with those around them while reducing information processing costs and minimizing deterioration in detection accuracy.

1 is a block diagram showing a configuration of a communication system according to a first embodiment. 4 is a flowchart showing an operation of the determination device according to the first embodiment. 4A to 4C are diagrams illustrating a specific example of a method for estimating a similarity of interests between a speaker and non-speakers in the determination device according to the first embodiment.

Specific embodiments to which the present disclosure is applied will be described in detail below with reference to the drawings. In each drawing, the same elements are given the same reference numerals, and duplicate descriptions will be omitted as necessary to clarify the description.

First Embodiment
First, a communication system 1 according to a first embodiment will be described. Fig. 1 is a block diagram showing the configuration of the communication system 1 according to the first embodiment. As shown in Fig. 1, the communication system 1 includes at least three or more wearable devices 10 and a determination device 20 (communication determination device 20). First, the configurations of the wearable devices 10 and the determination device 20 in the communication system 1 will be described. After that, the operation of the communication system 1 will be described.

<Configuration of wearable terminal 10>
The wearable terminal 10 is worn by a user. For example, one user wears one wearable terminal 10. Thus, multiple users wear multiple wearable terminals 10. Note that, although three wearable terminals 10 are shown in FIG. 1, the number of wearable terminals 10 is not limited to this, and four or more wearable terminals 10 may be used.

The wearable device 10 is, for example, a badge. Note that the wearable device 10 is not limited to a badge, and may be a headset, earphones, glasses, necklace, pendant, etc., as long as it is worn by a user. The wearable device 10 includes a sensor 11.

The sensor 11 detects physical information of the user of the wearable device 10. For example, the sensor 11 includes a microphone that detects the user's speech, and detects the speech, i.e., the voice, of the user of the wearable device 10. The sensor 11 also includes an acceleration sensor that detects the user's movements, and detects the acceleration of the user of the wearable device 10.

Each wearable terminal 10 has a transceiver (not shown). Each wearable terminal 10 is connected to the determination device 20 via a wireless or wired communication line. For example, each wearable terminal 10 may be communicatively connected to the determination device 20 via short-range wireless communication such as Bluetooth (registered trademark). Each wearable terminal 10 may be connected to the determination device 20 via a network such as the Internet. Each wearable terminal 10 transmits detected information to the determination device 20 via the communication line. In addition, each wearable terminal 10 receives information such as a control signal from the determination device 20 via the communication line.

Furthermore, each wearable device 10 may synchronize the time. For example, each wearable device 10 may receive a Network Time Protocol (NTP) time from a determination device 20 connected to the Internet and synchronize the time.

The wearable terminals 10 may be connected to each other by wireless or wired communication lines. The wearable terminals 10 may be connected to each other so that they can communicate with each other by short-range wireless communication such as Bluetooth (registered trademark), or may be connected to each other via a network such as the Internet. The wearable terminals 10 may transmit and receive various information to each other. The determination device 20 may obtain the distance between each wearable terminal 10 by short-range wireless communication between each wearable terminal 10.

<Configuration of Determination Device 20>
The determination device 20 is connected to each wearable terminal 10 via a wireless or wired communication line. The determination device 20 is, for example, an information processing device such as a personal computer (PC), a server, or a smartphone. The determination device 20 may be provided on a cloud so that information acquired by each wearable terminal 10 can be collected via the Internet.

The determination device 20 estimates the degree of similarity in interests between users of the wearable devices 10 based on information acquired using multiple wearable devices 10. The determination device 20 includes a memory unit 21, an utterance determination unit 22, an interest action determination unit 23, and an interest determination unit 24.

The memory unit 21 stores, for example, information detected by the sensor 11 of the wearable device 10. The memory unit 21 stores voice data of speech detected by the sensor 11, acceleration data detected by the sensor 11, etc.

The speech determination unit 22 determines the speaker and speech section of the speech. The following are some examples of methods for determining the speaker. For example, the speech determination unit 22 may determine the speaker based on whether the sound pressure detected by the sensor 11 exceeds a threshold value. In addition, for example, the speech determination unit 22 may first virtually form a group of multiple users who are close to each other based on the distance between the wearable devices 10 acquired using short-range wireless communication or the like. Then, the one user in the virtual group who has the highest sound pressure and whose sound pressure is higher than the other users by a certain amount or more may be determined to be the speaker.

Furthermore, as a method for improving the accuracy of determining who is speaking, if there is a speech section in the voice data acquired by the sensor 11 of the target wearable device 10 that is to be determined as being the speaker, the speech determination unit 22 determines that the user wearing the target wearable device 10 is the speaker.

Specifically, the speech determination unit 22 determines, in the voice data acquired by the sensor 11 of the target wearable device 10, a section in which the sound pressure is smaller than the stationary noise threshold as a first non-speech section that reflects the influence of stationary noise. Stationary noise is a sound that originates from the surrounding environment, and is generated continuously within a certain range of sound pressure, such as the sound of an air conditioner running or surrounding commotion.

When the sound pressure in the voice data acquired by the sensor 11 of the target wearable device 10 is equal to or greater than the threshold for stationary noise and is similar to the sound pressure of the voice data acquired by the sensor 11 of a comparison wearable device 10 located within a predetermined distance from the target wearable device 10, the speech determination unit 22 determines that the period is a second non-speech period that reflects the influence of sudden non-stationary noise. Sudden non-stationary noise is a sound that originates other than the voice of the user wearing the wearable device 10, such as a sudden loud voice from a nearby person or a loud noise.

When the sound pressure in the voice data acquired by the sensor 11 of the target wearable device 10 is equal to or greater than the threshold for stationary noise, is not similar to the sound pressure of the comparison wearable device 10, and is smaller than a threshold that should decrease according to the distance to the comparison wearable device 10, the speech determination unit 22 determines that this is a third non-speech section in which another person's speech has been mixed in.

Then, the speech determination unit 22 determines, in the voice data acquired by the sensor 11 of the target wearable device 10, sections other than the first to third non-speech sections as speech sections in which the user wearing the target wearable device 10 is speaking. In this way, the speech determination unit 22 determines whether the user of the target wearable device 10 is the speaker. Note that the method of determining the speaker is not limited to the above method. Also, some of the above-mentioned methods of determining the speaker may be combined.

The interest action determination unit 23 determines a characteristic action that indicates interest from the data of the movement of the user of the wearable device 10 acquired by the sensor 11. Specifically, the interest action determination unit 23 determines a "nod" of the user wearing the wearable device 10 from the acceleration detected by the sensor 11. Note that the interest action is not limited to a nod, and may be a clap, etc.

The interest movement determination unit 23 determines "nodding" in the following manner. For example, the interest movement determination unit 23 extracts the vertical value from the time series data of the three axes, XYZ, of the acceleration acquired by the sensor 11 for each predetermined time interval. Then, the average value and standard deviation for that time interval are calculated. If the calculated standard deviation is smaller than a predetermined value, it is determined that a nod has occurred in that time interval. However, in this case, it is a necessary condition that no major movement such as walking or a change in posture is involved. Furthermore, the interest movement determination unit 23 may determine that a nod has occurred in that time interval if there is a point that deviates from the calculated average value by more than a predetermined deviation. However, it is a necessary condition that a single vertical movement occurs.

As another method for detecting "nodding," the interest behavior determination unit 23 extracts time series data of the three axes, XYZ, of acceleration acquired by the sensor 11 for each predetermined time interval. The value of that time interval may be applied to a deep learning Convolution Neural Network (CNN), and if the output value is equal to or greater than a predetermined value, it may be determined that a nod occurred in that interval.

The interest determination unit 24 estimates the similarity of interests between non-speakers and speakers based on the information determined by the speech determination unit 22 and the interest action determination unit 23. Specifically, it calculates the time transition of the nodding density of non-speakers within the speech section of the speaker determined by the speech determination unit 22. Here, the nodding density is, for example, the number of nods by the user of the wearable device 10 in a specified period of time. Then, the interest determination unit 24 calculates the correlation of the time transition of the nodding density between non-speakers. Therefore, the similarity of interests between non-speakers and speakers can be estimated based on the magnitude of the correlation.

<Hardware Configuration>
The wearable terminal 10 or the determination device 20 may be configured as hardware centered around a microcomputer including a CPU (Central Processing Unit) that performs calculation processing, determination processing, control processing, etc., a ROM (Read Only Memory) that stores an arithmetic program executed by the CPU, a control program, etc., a RAM (Random Access Memory) that stores various data, etc., and an interface unit (I/F) that inputs and outputs signals to and from the outside. The CPU, ROM, RAM, and interface unit are connected to each other via a data bus, etc.

<Operation of Determination Device 20>
Next, an operation of the communication system 1 according to the first embodiment will be described. Here, the operation of the determination device 20 of the communication system 1 will be mainly described. Fig. 2 is a flowchart showing the operation of the determination device 20 according to the first embodiment.

Assume a situation in which at least three or more users are having a conversation. Each user uses one wearable device 10. Thus, multiple wearable devices 10 are used by multiple users. The users of the wearable devices 10 are, for example, participants in group work.

The memory unit 21 of the determination device 20 stores information detected by the sensor 11 of the wearable device 10. The detected information includes voice data detected when the user of the wearable device 10 speaks, and acceleration data detected when the user of the wearable device 10 moves, etc.

First, as shown in FIG. 2, in step S101, the speech determination unit 22 of the determination device 20 determines a speaker from among users of the wearable device 10 based on the voice data acquired by the wearable device 10, and further determines the speech section of the speaker. The speech section corresponds to the time during which the speaker is speaking. Here, the speech determination unit 22 acquires the voice data of each user using the wearable device 10 from the memory unit 21.

Next, in step S102, the interest action determination unit 23 determines whether the user of the wearable device 10 is nodding based on the acceleration data acquired by the wearable device 10. Here, the speech determination unit 22 acquires the acceleration data of each user using the wearable device 10 from the storage unit 21.

Next, the interest determination unit 24 estimates the similarity of the non-speaker's interest in the speaker based on the information determined by the speech determination unit 22 and the interest action determination unit 23. First, in step S103, the interest determination unit 24 calculates the time transition of the nodding density of the non-speaker within the speech section of the speaker determined by the speech determination unit 22. Here, the nodding density is, for example, the number of times the user of the wearable device 10 nods in a predetermined period of time. Next, in step S104, the interest determination unit 24 calculates the correlation of the time transition of the nodding density between the non-speakers. Therefore, the interest determination unit 24 can estimate the similarity of the non-speaker's interest in the speaker based on the magnitude of the correlation.

2 and 3, a specific example of a method for estimating the similarity of interests between non-speakers and a speaker shown in steps S103 to S104 will be described. FIG. 3 is a diagram showing a specific example of a method for estimating the similarity of interests between non-speakers and a speaker in the determination device 20 according to the first embodiment.

In the example shown in FIG. 3, users A to D each use a wearable device 10. User A is a speaker. Meanwhile, user B, user C, and user D are non-speakers. User A speaks from speech start point I to speech end point II, and the time from speech start point I to speech end point II is the speech section.

First, in step S103, the interest determination unit 24 calculates the time transition of the nodding density for each non-speaker (user B to user D) within the speech section of the speaker (user A). Here, the nodding density is, for example, the number of times the user of the wearable device 10 nods in a given period of time.

Next, in step S104, the interest determination unit 24 calculates the correlation of the time transition of the nodding density between non-speakers. Therefore, the interest determination unit 24 can estimate the similarity of the interests of non-speakers in speakers based on the magnitude of the correlation. For example, the interest determination unit 24 determines that the correlation of the time transition between user B and user C is high. Then, the interest determination unit 24 estimates that the interests of user B and user C in user A are similar. On the other hand, the interest determination unit 24 determines that the correlation of the time transition between user B and user D is low. Then, the interest determination unit 24 estimates that the interests of user B and user D in user A are different.

The communication system 1 according to the first embodiment estimates the similarity of interests of a non-speaker with respect to a speaker using information such as voice data and acceleration data, which have low information processing costs, without using image information that has a large amount of information and high information processing costs. The degree of participation of a non-speaker in communication centered on the speaker can be detected from the similarity of interests. For example, the more non-speakers there are who have similar interests to a speaker, the more it can be estimated that the non-speakers are participating in communication centered on the speaker. Therefore, a communication system can be provided that can detect the degree of participation of a user in communication with those around them while reducing information processing costs and suppressing a decrease in detection accuracy.

In addition, in Patent Document 1, when the degree of participation in communication with the user's surroundings is detected using image information, the user's actions are captured by a camera or the like. This also results in the following problems (1) to (3). (1) The degree of participation in communication cannot be determined when the user is outside the range that can be captured or when the face is facing backwards. (2) Users have a psychological resistance to being photographed. (3) The cost of installing cameras is required. The communication system 1 according to the first embodiment estimates the similarity of interests of a non-speaker in a speaker without using image information. Therefore, it is possible to provide a communication system that can detect the degree of participation in communication with the user's surroundings while solving the above problems (1) to (3) and suppressing a decrease in detection accuracy.

In addition, in Patent Document 1, an information provider provides information related to an object of explanation to an information acquirer in face-to-face contact, and an interest tendency information output device is provided that acquires and outputs interest tendency information, which is information indicating the interest tendency of the information acquirer, in a situation in which the information acquirer can directly access the object of explanation. Specifically, the interest tendency information output device of Patent Document 1 includes a motion information receiving unit that receives motion information indicating the movement of the information acquirer, a voice information receiving unit that receives voice information related to the voice uttered by the information acquirer, an interest tendency information generating unit that generates interest tendency information, which is information related to the interest tendency of the information acquirer, from the motion information and the voice information, and an interest tendency information output unit that outputs the interest tendency information. With this configuration, the interest tendency information output device of Patent Document 1 attempts to meet the need to understand people with similar interests and concerns in order to deepen relationships in communication situations such as group work.

However, the interest tendency information output device in Patent Document 1 estimates the interests and concerns of a speaker by grasping the content of the speaker's speech and linking the response to that. Therefore, it is necessary to handle a large amount of information such as the speech content and video, and it is not possible to easily know interests and concerns with a simple system. In addition, speakers have a psychological resistance to having their speech content taken into account.

In contrast, the communication system 1 according to the first embodiment can reduce information processing costs because it is not necessary to grasp the content of the speaker's utterance and infer the speaker's interests and concerns from linking the responses to the content. In addition, it can suppress the speaker's psychological resistance to having the content of his or her utterance taken into account.

The present invention is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit and scope of the invention.

Each component in the above-described embodiments may be configured with hardware or software, or both, and may be configured with one piece of hardware or software, or may be configured with multiple pieces of hardware or software. The functions (processing) of each device may be realized by a computer having a CPU, memory, etc. For example, a program for performing the method in the embodiment may be stored in a storage device, and each function may be realized by executing the program stored in the storage device with a CPU.

These programs can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)). The programs may also be supplied to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the programs to a computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

1 Communication system 10 Wearable device 11 Sensor 20 Determination device (communication determination device)
21 Storage unit 22 Speech determination unit 23 Interest action determination unit 24 Interest determination unit

Claims (1)

A communication system including a determination device connected to at least three wearable devices used by at least three users, each of which has one device, the determination device comprising:
an utterance determination unit that determines a speaker of one of the at least three users and a speech period of the speaker based on voice data detected by a microphone included in each of the at least three wearable devices;
an interest gesture determination unit that determines whether the at least three users are nodding based on acceleration data detected by an acceleration sensor included in each of the at least three wearable devices;
an interest determination unit that calculates a time transition of a nod density, which is the number of nods of each of the at least two non-speakers in a predetermined time period during a speech section of the speaker, calculates a correlation of the time transition of the nod density between the at least two non-speakers, and estimates a similarity of interest between the at least two non-speakers in the speaker as a degree of participation of the at least two non-speakers in communication centered on the speaker based on the magnitude of the correlation ,
The utterance determination unit is
determine a speaker among the at least three users based on the sound pressure level of the voice data, and determine a section other than a first non-speech section, a second non-speech section, and a third non-speech section determined based on voice data detected by an acceleration sensor provided in a target wearable device among the at least three wearable devices that is to be determined as being the speaker, as a speech section spoken by the user wearing the target wearable device;
The interest action determination unit is
A vertical value of acceleration time series data included in the acceleration data of each of the at least two non-speakers is extracted for each predetermined time interval, and if a value calculated from the vertical value for the predetermined time interval satisfies a predetermined relationship with a predetermined value, it is determined that a nod of a non-speaker has occurred in the predetermined time interval.
Communication system.

Images