JP7484138B2 - Search terminal device, search system, and search method - Google Patents

Description

The present invention relates to a search terminal device, a search system, and a search method.

When meeting someone with whom one has communicated via email or phone but has never met, or when meeting someone for the first time in a long time and one whose memory has faded because it has been a long time since the last time they met, there is a desire to know personal information about that person, such as their name and affiliation.
In response to such demands, for example, Patent Document 1 discloses a communications terminal connectable to a telephone conference system in which conversations can be made simultaneously from two or more communications terminals, the communications terminal comprising: storage means for storing the voiceprints of known persons; voiceprint acquisition means for acquiring the voiceprint of a speaker speaking in the telephone conference system; voiceprint matching means for matching the voiceprint acquired by the voiceprint acquisition means with the voiceprint stored in the storage means; and output means for outputting the results of the voiceprint matching by the voiceprint matching means.

In the communication terminal of Patent Document 1 mentioned above, when performing voiceprint matching, a process is required to prompt the other party (the user to be summoned) to speak a specific sound to be used in voiceprint matching, which poses a problem that the other party may become aware that the other party is not recognized.
The present invention has been made in consideration of the above-mentioned circumstances, and has an object to make it possible to search for a contact without prompting the contact to utter a predetermined sound.

The main invention for solving the above problem is a search terminal device that is communicatively connected to an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and receives and displays personal information of a candidate searched for from the plurality of candidates from the external device, the search terminal device comprising: an acquisition means for acquiring target person voice data based on the voice of a person being searched for; a request means for extracting a candidate voice data whose voiceprint matches the target person voice data from the plurality of candidate voice data by transmitting the target person voice data to the external device, and for transmitting the personal information associated with the candidate voice data; a receiving means for receiving the personal information transmitted by the external device; and a display means for displaying the personal information received by the receiving means , the request means transmitting terminal location information indicating a current location of the search terminal device to the external device, and the external device determining a priority order when searching for the candidate voice data based on the candidate location information indicating the current location of the candidate and the terminal location information .

According to the present invention, it is possible to search for a contact without prompting the contact to say a specific sound.

1 is a system configuration diagram showing an overview of a search system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an overview of processing in a search system. FIG. 2 is a hardware configuration diagram of a remote conference server. 4A to 4C are diagrams for explaining a voice information DB and a connection information DB stored in the HD of the remote conference server. FIG. 2 is a hardware configuration diagram of an employee information server. 3A and 3B are diagrams illustrating an employee information DB stored in the HD of the employee information server. FIG. 2 is a hardware configuration diagram of a remote conference terminal. FIG. 2 is a diagram illustrating a hardware configuration of an application execution terminal. 11 is a diagram illustrating an example of a display screen of the application executing terminal before recording the voice of a person to be searched for. FIG. FIG. 11 is a diagram illustrating an example of a display screen of the application executing terminal during audio recording. 11A and 11B are diagrams illustrating an example of a display screen of the application executing terminal during a search. 13 is a diagram illustrating an example of a display screen of the application executing terminal when the other party has been identified. FIG. 13 is a diagram illustrating an example of a display screen of the application executing terminal when the other party cannot be identified. FIG. 13 is a diagram illustrating an example of a display screen of a search history in an application executing terminal. 10 is a flowchart showing a registration process of voice data in the remote conference server; 13 is a flowchart showing a search process in the application execution terminal. 13 is a flowchart showing a process performed during a search in the remote conference server. 13 is a flowchart showing a process performed during a search in an employee information server.

The present invention relates to a person search system shown in Figures 1 and 2, and is particularly characterized by: (1) recording the voice of a person to be searched for in the application executing terminal 4 to generate recorded data (target person voice data); (2) transmitting the recorded data in the application executing terminal 4 to an external device EX (remote conference server 1, employee information server 2) equipped with a voice information DB (DataBase) 104a and an employee information DB 204a; (3) searching the voice information DB 104a in the external device EX to extract one candidate voice data whose voiceprint matches the recorded data from multiple candidate voice data; (4) searching the employee information DB 204a in the external device EX to extract employee information corresponding to the one candidate voice data, and transmitting the employee information to the application executing terminal 4; and (5) displaying the received employee information in the application executing terminal 4.
The features of the present invention will be described in detail with reference to the following drawings. However, unless otherwise specified, the components, types, combinations, shapes, relative arrangements, etc. described in the embodiments are merely illustrative examples and do not intend to limit the scope of the present invention.

<Search system overview>
FIG. 1 is a system configuration diagram showing an overview of a search system according to an embodiment of the present invention.
In the search system shown in Fig. 1, a remote conference server 1, an employee information server 2, remote conference implementation terminals 3 (3A to 3D), and application execution terminals 4 (4A to 4D) are connected to each other via communication networks NW1 and NW2 so as to be able to transmit and receive data to and from each other. The communication networks NW1 and NW2 may be, for example, an in-house LAN (Local Area Network, wired or wireless) NW1 or a mobile communication network NW2. In the mobile communication network NW2, a plurality of communication terminals are connected to each other so as to be able to communicate with each other via, for example, a base station BS.
The remote conference server 1 and the remote conference terminal 3 constitute a remote conference system, and when a plurality of remote conference terminals 3A to 3D are communicably connected to the remote conference server 1, transmission and reception of video and audio becomes possible between each of the remote conference terminals 3A to 3D. To connect to the remote conference server 1, an employee ID (IDentification) and password of the employee (user) who uses the remote conference terminal 3 (3A to 3D) are used.

Figure 2 is a block diagram showing an overview of the processing in the search system. As shown in Figure 2, the remote conference server 1 has a voice information DB 104a that stores voice data of employees who have participated in a remote conference in association with an employee ID. In the remote conference system, voice data is automatically registered in the voice information DB 104a. For example, when the remote conference server 1 acquires voice data of an employee sent from the remote conference implementation terminal 3 during a remote conference, it determines whether the voice data of the employee is stored in the voice information DB 104a, and if not, it registers the voice data in the voice information DB 104a in association with the employee ID.

The employee information server 2 has an employee information DB 204a that stores employee information (personal information such as name and affiliation) of all employees in association with an employee ID. In the employee information DB 204a, the employee information of the employee is newly registered when the employee joins the company, and the employee information is updated when the employee moves to a different department or has a different position. When the employee information server 2 receives an employee ID sent by the remote conference server 1, it extracts the employee information corresponding to the received employee ID from the employee information DB 204a and sends it to the remote conference server 1.

The application executing terminals 4 (4A to 4D) are communicatively connected to the remote conference server 1 in accordance with the execution of an application program (hereinafter simply referred to as an application) installed in the application executing terminal 4. To connect to the remote conference server 1, the employee ID and password of the employee (user) who uses the application executing terminal 4 are used.
When the application executing terminal 4 records the voice of the search target person using the application, it generates recorded data (target person voice data) and transmits the recorded data to the remote conference server 1. The remote conference server 1 compares the voiceprint of the received recorded data with that of the voice data stored in the voice information DB 104a and determines whether or not there is voice data whose voiceprint matches the recorded data. When the remote conference server 1 extracts voice data whose voiceprint matches the recorded data, it transmits the employee ID of the voice data to the employee information server 2 and receives employee information corresponding to the employee ID from the employee information server 2. The remote conference server 1 transmits the received employee information to the application executing terminal 4, and the application executing terminal 4 displays the received employee information.

In this embodiment, the remote conference server 1 automatically registers voice data of employees who have participated in a remote conference in the voice information DB 104a by associating the voice data with the employee ID. Therefore, when the employee is the search target, the search can be performed without prompting the employee to make a specified sound.

<Remote conference server 1>
Next, a description will be given of the teleconference server 1. Fig. 3 is a hardware configuration diagram of the teleconference server 1. Here, the hardware configuration of the teleconference server 1 will be described.
As shown in FIG. 3, the remote conference server 1 is constructed by a computer and includes a CPU 101, a ROM 102, a RAM 103, a HD (Hard Disk) 104, a HDD (Hard Disk Drive) controller 105, a display 106, an external device connection I/F (Interface) 108, a network I/F 109, a data bus 110, a keyboard 111, a pointing device 112, a DVD-RW (Digital Versatile Disk Rewritable) drive 114, and a media I/F 116.
Among these, the CPU 101 controls the overall operation of the remote conference server 1. The ROM 102 stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is used as a work area for the CPU 101. The HD 104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data from the HD 104 under the control of the CPU 101. The display 106 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I/F 108 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 109 is an interface for data communication using the communication networks NW1 and NW2. The data bus 110 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 101.

The keyboard 111 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 112 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 114 controls the reading and writing of various data from a DVD-RW 113, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I/F 116 controls the reading and writing (storing) of data from a recording medium 115, such as a flash memory.

4A to 4C are diagrams for explaining the voice information DB 104a and the connection information DB 104b stored in the HD 104 of the remote conference server 1. FIG.
As shown in FIG. 4A, a part of the HD 104 is used as a voice information DB 104a and a connection information DB 104b. As shown in FIG. 4B, the voice information DB 104a stores voice data A to N (voice information) generated by the remote conference terminal 3 based on the voice of the employee during the remote conference in association with the employee ID. The employee whose voice data is registered in the voice information DB 104a is a candidate at the time of the search. Therefore, the voice data registered in the voice information DB 104a is candidate voice data. As shown in FIG. 4C, when the remote conference terminal 3 or the application execution terminal 4 connects to the remote conference server 1, the connection information DB 104b acquires current location information indicating the current location of each terminal 3, 4 from each terminal 3, 4, acquires schedule information of the user (employee) using each terminal 3, 4 from the employee information server 2 (employee information DB 204a), and stores each piece of information in association with the employee ID. The connection information DB 104b targets the terminals 3 and 4 currently connected to the teleconference server 1, and information about the terminals 3 and 4 whose communication with the teleconference server 1 has been cut off is deleted.
The current location information of the remote conference implementing terminal 3 is candidate location information because the user of the remote conference implementing terminal 3 corresponds to a candidate at the time of employee search and the current location information of the remote conference implementing terminal 3 indicates the current location of the candidate. The current location information of the application executing terminal 4 is terminal location information because the application executing terminal 4 is a search terminal device for searching for a search target person.

<Employee Information Server 2>
Next, a description will be given of the employee information server 2. Fig. 5 is a diagram showing the hardware configuration of the employee information server 2. Here, the hardware configuration of the employee information server 2 will be described.
As shown in FIG. 5, the employee information server 2 is constructed by a computer and includes a CPU 201, ROM 202, RAM 203, HD 204, HDD controller 205, display 206, external device connection I/F 208, network I/F 209, data bus 210, keyboard 211, pointing device 212, DVD-RW drive 214, and media I/F 216.
Note that each of these components has the same configuration as the CPU 101, ROM 102, RAM 103, HD 104, HDD controller 105, display 106, external device connection I/F 108, network I/F 109, data bus 110, keyboard 111, pointing device 112, DVD-RW drive 114, and media I/F 116 of the remote conference server 1 described in Figure 3, so detailed explanations will be omitted.

6A and 6B are diagrams for explaining the employee information DB 204a stored in the HD of the employee information server 2. FIG.
As shown in Fig. 6(a), a part of the HDD is used as an employee information DB 204a. As shown in Fig. 6(b), the employee information DB 204a stores, for example, name information indicating the name of the employee, department information indicating the department to which the employee belongs (organization information indicating the organization to which the employee belongs), schedule information indicating the employee's schedule, job title information indicating the employee's job title, and photo data of the employee's face photo, in association with the employee ID. Each employee information in the employee information DB 204a is registered when the employee joins the company, the contents are updated when the employee's affiliation or job title is changed, and the information is deleted when the employee resigns.

<Remote conference terminal 3>
Next, a description will be given of the teleconference implementing terminal 3. FIG.
As shown in FIG. 7 , the remote conference terminal 3 is constructed by a computer capable of detecting its position by GPS (Global Positioning System), and includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD (Solid State Drive) 305, a media I/F 307, an operation button 308, a power switch 309, a GPS receiver 310, a network I/F 311, a CMOS (Complementary Metal-Oxide-Semiconductor) 312, an image sensor I/F 313, a microphone 314, a speaker 315, an audio input/output I/F 316, a display I/F 317, an external device connection I/F 318, a short-range communication circuit 319, an antenna 319a of the short-range communication circuit 319, and a bus line 321.
Of these, the CPU 301 controls the overall operation of the remote conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL. The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD 305 controls the reading and writing of various data from and to the flash memory 304 under the control of the CPU 301. Note that a HD may be used instead of the SSD.

The media I/F 307 controls reading and writing (storing) data from and to a recording medium 306 such as a flash memory. The recording medium 306 is configured to be freely attached to and detached from the media I/F 307. The recording medium 306 is not limited to a flash memory, and may be an EEPROM or the like, as long as it is a non-volatile memory that reads and writes data under the control of the CPU 301.
The operation button 308 is a button that is operated when, for example, selecting a destination of the teleconference terminal 3. The power switch 309 is a switch for switching ON/OFF the power of the teleconference terminal 3. The GPS receiving unit 310 receives GPS signals from GPS satellites.

The network I/F 311 is an interface for data communication using communication networks NW1 and NW2 such as the Internet. The CMOS sensor 312 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 301 to obtain image data. Note that instead of the CMOS sensor 312, an imaging means such as a CCD (Charge Coupled Device) sensor may be used. The image sensor I/F 313 is a circuit that controls the drive of the CMOS sensor 312. The microphone 314 is a built-in circuit that converts sound into an electrical signal. The speaker 315 is a built-in circuit that converts an electrical signal into physical vibrations to generate sounds such as music and voice. The sound input/output I/F 316 is a circuit that processes the input/output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I/F 317 is a circuit that transmits image data to an external display 320 under the control of the CPU 301.

Display 320 is a type of display means configured with liquid crystal or organic EL (Electro Luminescence) for displaying an image of a subject, operation icons, etc. Display 320 is also connected to display I/F 317 by a cable. This cable may be a cable for analog RGB (VGA, Video Graphics Array) signals, a cable for component video, or a cable for HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signals.

The external device connection I/F 318 is an interface for connecting various external devices. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I/F 318 via a USB cable or the like. When an external camera is connected, the external camera is driven in priority to the built-in CMOS sensor 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone or the external speaker is driven in priority to the built-in microphone 314 or the built-in speaker 315 under the control of the CPU 301.
The short-distance communication circuit 319 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark), etc. The bus line 321 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301.

<Application Execution Terminal 4>
Next, the application executing terminal 4 will be described. Fig. 8 is a hardware configuration diagram of the application executing terminal 4. As shown in Fig. 8, the application executing terminal 4 is constructed by, for example, a smartphone, and includes a CPU 401, a ROM 402, a RAM 403, an EEPROM 404, a CMOS sensor 405, an image sensor I/F 406, an acceleration/direction sensor 407, a media I/F 409, and a GPS receiving unit 411.
Among these, the CPU 401 controls the overall operation of the application executing terminal 4. The ROM 402 stores the CPU 401 and programs such as IPL used to drive the CPU 401. The RAM 403 is used as a work area for the CPU 401. The EEPROM 404 reads or writes applications (computer programs) executed on the application executing terminal 4 and various data under the control of the CPU 401. The CMOS sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait of the user) under the control of the CPU 401 to obtain image data. Note that an imaging means such as a CCD sensor may be used instead of the CMOS sensor 405. The imaging element I/F 406 is a circuit that controls the driving of the CMOS sensor 405. The acceleration/direction sensor 407 is various sensors such as an electronic magnetic compass or gyrocompass that detects geomagnetism, and an acceleration sensor. The media I/F 409 controls the reading or writing (storing) of data from a recording medium 408 such as a flash memory. The GPS receiving unit 411 receives GPS signals from GPS satellites.

The application executing terminal 4 also includes a long-distance communication circuit 412, an antenna 412a of the long-distance communication circuit 412, a CMOS sensor 413, an image sensor I/F 414, a microphone 415, a speaker 416, an audio input/output I/F 417, a display 418, an external device connection I/F 419, a short-distance communication circuit 420, an antenna 420a of the short-distance communication circuit 420, and a touch panel 421.
Among these, the long-distance communication circuit 412 is a circuit that communicates with other devices via the antenna 412a and the communication networks NW1 and NW2. The CMOS sensor 413 is a type of built-in imaging means that captures an image of a subject (an image on the opposite side to the user) under the control of the CPU 401 to obtain image data. The image sensor I/F 414 is a circuit that controls the drive of the CMOS sensor 413. The microphone 415 is a built-in circuit that converts sound into an electric signal. The speaker 416 is a built-in circuit that converts an electric signal into a physical vibration to generate sound such as music or voice. The sound input/output I/F 417 is a circuit that processes input and output of sound signals between the microphone 415 and the speaker 416 under the control of the CPU 401. The display 418 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I/F 419 is an interface for connecting various external devices. The short-distance communication circuit 420 is a communication circuit such as NFC or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows the user to operate the application executing terminal 4 by pressing the display 418 .
The application executing terminal 4 also includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401.
The application executing terminal 4 may be a terminal device common to the above-mentioned remote conference implementing terminal 3. For example, by installing both a computer program for the application executing terminal 4 and a computer program for the remote conference implementing terminal 3 in a computer or smartphone capable of detecting its location by GPS, the computer or smartphone can be used as both the application executing terminal 4 and the remote conference implementing terminal 3.

<Example of a screen display on the application executing terminal 4>
In the application executing terminal 4, the CPU 401 reads out the application stored in the EEPROM 404, expands it in the RAM 403, and executes it to perform various processes related to person search. As various processes are executed, an image corresponding to the process being executed is displayed on the display 418, and the image prompts the user of the application executing terminal 4 to perform an input operation, informs the user of the process status, and displays the search results.

Fig. 9 to Fig. 14 are examples of display screens of the application executing terminal 4. For example, Fig. 9 is an example of a display screen of the application executing terminal 4 before recording the voice of the person to be searched for, Fig. 10 is an example of a display screen of the application executing terminal 4 during voice recording, Fig. 11 is an example of a display screen of the application executing terminal 4 during a search, Fig. 12 is an example of a display screen of the application executing terminal 4 when the other party has been identified, Fig. 13 is an example of a display screen of the application executing terminal 4 when the other party has not been identified, and Fig. 14 is a diagram for explaining an example of a display screen of the search history on the application executing terminal 4.
9, when the application is executed on the application executing terminal 4, a screen that prompts the user to record the voice of the other party (search target person) is displayed on the display 418. For example, a message 442a stating "Please record the other party's voice" is displayed in the upper half of the display area 441, and a button image 443a stating "Record" is displayed below the message 442a. Furthermore, a Home button image 444 that is operated when displaying the home screen is displayed to the lower left of the button image 443a, and a history button image 445 that is operated when displaying the search history is displayed to the lower right of the button image 443a.
When the user of the application executing terminal 4 (terminal user) operates button image 443a on the display screen of FIG. 9, touch panel 421 detects the operation and starts recording the other party's voice.

As shown in FIG. 10, while audio is being recorded, a screen indicating that recording is in progress is displayed on the display 418. For example, a message 442b stating "Recording, please wait" is displayed in the upper half of the display area 441, and a button image 443b stating "Cancel" is displayed below the message 442b. Note that a Home button image 444 and a history button image 445 are displayed to the lower left and lower right of the button image 443a. When the user of the application executing terminal 4 operates button image 443b on the display screen in FIG. 10, the touch panel 421 detects the operation and recording of the other party's voice is stopped.

As shown in FIG. 11, when the remote conference server 1 is searching for a partner, a screen indicating that the search is in progress is displayed on the display 418. For example, a message 442c stating "Searching, please wait" is displayed in the upper half of the display area 441, and a button image 443c stating "Cancel" is displayed below the message 442c. When the user of the application executing terminal 4 operates the button image 443b on the display screen in FIG. 11, the touch panel 421 detects the operation, and the search by the remote conference server 1 is stopped.

12, when the remote conference server 1 identifies the other party, the employee information (personal information) of the other party is displayed on the display 418. For example, in the employee information display area 446 provided in the display area 441, employee information 446a such as the other party's name, department, and position, a face photo image 446b of the other party, inquiry time information 446c, and a close button image 446d operated when closing the display screen are displayed. In this way, by displaying the employee information 446a such as the other party's name and the face photo image 446b, the user of the application execution terminal 4 can easily understand the relationship with the other party. In addition, a Home button image 444 and a history button image 445 are displayed in the lower left and lower right of the employee information display area 446. As shown in FIG. 13, when the remote conference server 1 cannot identify the other party, a screen indicating that the other party could not be identified is displayed on the display 418. For example, in the message display area 447 provided in the display area 441, a message 447a stating "No matching user was found" is displayed.

As shown in FIG. 14, when the history button 445 is operated, employee information of employees previously extracted by a search is displayed on the display 418. For example, the most recently extracted employee information (employee information 446a, face photo image 446b, inquiry time information 446c) is displayed in the first employee information display area 446A provided in the display area 441. In addition, the second employee information display area 446B extracted one time before is displayed below the first employee information display area 446A. This second employee information display area 446B can be displayed by a scrolling operation by the terminal user, for example, a swipe operation in which the fingertip is placed on the display (touch panel 421) and moved to the top of the screen.

<About search processing>
As described in Figures 1 and 2, when multiple remote conference implementing terminals 3 are communicatively connected to the remote conference server 1 and a remote conference is held, the remote conference server 1 associates the voice data of employees who participated in the remote conference with their employee IDs and stores them in the voice information DB 104a.
When a user of the application executing terminal 4 executes an application to search for a person to be searched for and is communicably connected to the remote conference server 1, the application executing terminal 4 records the voice of the person to be searched for. For example, the application executing terminal 4 digitally converts the voice of the person to be searched for, collected for a certain period of time by the microphone 415, to generate recorded data (person to be searched for voice data), and temporarily stores the data in the RAM 403. In addition, when the application executing terminal 4 is in a conversation with the person to be searched for, the application executing terminal 4 temporarily stores the voice data transmitted by the terminal used by the person to be searched for in the RAM 403 as recorded data.
The recorded data temporarily stored in RAM 403 is transmitted from the application executing terminal 4 to the remote conference server 1. The remote conference server 1 compares the voiceprint of the received recorded data with the voiceprint of the voice data registered in the voice information DB 104a and determines whether there is voice data with a matching voiceprint. For example, the remote conference server 1 executes a predetermined program to perform frequency analysis or the like on the recorded data and voice data to obtain voiceprints of the search target person and the voice data of employees (candidates) registered in the voice information DB 104a and determines whether the voiceprints match.

For employees who have participated in a remote conference, voice data is associated with the employee ID and registered in the voice information DB 104a, so if the employee is the target of a search, it is possible to perform a search without prompting the employee to speak a specified sound.
In a search process based on voice data, the remote conference server 1 assigns a priority to each voice data registered in the voice information DB 104a in order to improve search efficiency, and compares voice data with a higher priority before voice data with a lower priority.

For example, when the remote conference server 1 receives terminal location information indicating the current location of an application executing terminal 4 from an application executing terminal 4 (search terminal device), it compares the voice data from among multiple remote conference implementing terminals 3 connected to the remote conference server 1 at the time of reception, starting with the user (employee) of the remote conference implementing terminal 3 that is closest to the current location of the application executing terminal 4. In this way, by comparing the voice data starting with the user of the remote conference implementing terminal 3 that is closest to the application executing terminal 4, it is possible to efficiently extract the search target person when the user of the application executing terminal 4 and the search target person are participating in the same conference.
Furthermore, the remote conference server 1 may obtain the department (user organization information) of the user of the application execution terminal 4 (search terminal device) from the employee information DB 204a of the employee information server 2 based on the employee ID of the user, and compare voice data of employees in the same department (candidate organization information) as the terminal user preferentially. In this way, by comparing voice data of employees in the same department as the terminal user preferentially over voice data of employees in a different department, employees who have a close relationship with the terminal user are given priority, so that search targets can be extracted efficiently. Note that with regard to department, in addition to the current department, past departments may also be included.
Furthermore, the remote conference server 1 may refer to the schedules of candidates stored in the employee information DB 204a and determine priorities when searching for voice data based on the schedules of the candidates. For example, voice data of employees (candidates) whose schedules at the time of the search match those of the user of the application executing terminal 4 may be compared preferentially over voice data of employees whose schedules do not match. In this way, by determining priorities when searching for voice data based on the schedules of candidates, search targets can be extracted efficiently.

<Registration process of voice data by the remote conference server 1>
Next, a description will be given of the voice data registration process performed by the remote conference server 1. This registration process is realized, for example, by the CPU 101 loading a computer program stored in the HD 104 onto the RAM 103 and executing it. Fig. 15 is a flowchart showing the voice data registration process performed by the remote conference server 1.
15, in the voice data registration process, the remote conference server 1 judges whether or not it has received an employee ID and password from the remote conference terminal 3 (S1). The remote conference server 1 waits until it receives the employee ID, etc. (S1, No), and when it receives the employee ID, etc. (S1, Yes), it judges whether or not authentication has been successful based on the entered employee ID, etc. (S2). If authentication has not been successful (S2, No), it rejects the participation of the remote conference terminal 3 in the conference (S3), and ends the series of processes.
If the authentication is successful (S2, Yes), the location information of the remote conference implementing terminal 3 is obtained from the remote conference implementing terminal 3, and the schedule of the employee using the remote conference implementing terminal 3 is obtained from the employee information server 2 (employee information DB 204a) (S4). This information (location information of the remote conference implementing terminal 3, schedule information) is registered in the connection information DB 104b. Note that each piece of information registered in the connection information DB 104b is deleted from the connection information DB 104b when the communication between the remote conference implementing terminal 3 and the remote conference server 1 is disconnected.

Next, the remote conference server 1 judges whether or not it has received voice data from the remote conference terminal 3 (S5). For example, the remote conference server 1 judges that it has received voice data when an employee using the remote conference terminal 3 speaks and voice data based on this speech is received from the remote conference terminal 3. When it is judged that it has not received voice data (S5, No), the remote conference server 1 judges whether or not the conference has ended (S6), and continues the process of S5 until the conference ends (S6, No), and when the conference ends (S6, Yes), the series of processes ends.
When the remote conference server 1 receives voice data from the remote conference terminal 3 (S5, Yes), it analyzes the voiceprint of the received voice data and judges whether it can be used for voiceprint matching (S7). If it cannot be used for voiceprint matching (S8, No), the process returns to S5 to judge whether voice data has been received.
If it can be used for voiceprint matching (S8, Yes), it is determined whether the voice data is not registered in the voice information DB 104a (S9), and if it is registered voice data (S9, No), the process returns to S5 to determine whether the voice data has been received. On the other hand, if it is unregistered voice data (S9, Yes), the employee ID and the voice data are registered in the voice information DB 104a (S10), and the process ends.
By the remote conference server 1 executing the above process, the voice data of the employee who newly participates in the remote conference is automatically registered in the voice information DB 104a in association with the employee ID.

<Search Process by Application Executing Terminal 4>
Next, a description will be given of the search process performed by the application executing terminal 4. This search process is realized, for example, by the CPU 401 expanding an application (computer program) stored in the EEPROM 404 into the RAM 403 and executing it. Fig. 16 is a flowchart showing the process performed at the time of a search in the application executing terminal 4.
16, in the search process for a search target person, the application executing terminal 4 starts the application in response to a user's operation (S11). The user's operation is, for example, tapping on an icon for the application displayed on the display 418. After the application is started, the user inputs an employee ID and password into the application executing terminal 4 (S12), and these employee ID and password are transmitted to the remote conference server 1 (S13). The application executing terminal 4 waits until an authentication result is transmitted from the remote conference server 1 (S14), and if authentication is not possible (S14, No), the process returns to S12 and prompts the user to re-enter the employee ID and password.
On the other hand, if authentication is successful (S14, Yes), the recording request screen described in Fig. 9 is displayed, and the system waits for the user to operate the recording button (button image 443a) (S15, No). When the user operates the button image 443a (S15, Yes), recording of the voice of the person to be searched for is started (S16). The recording in S16 is continued until a specified recording time has elapsed (S17), and when the specified recording time has elapsed (S17, Yes), the recording data and location information (terminal location information) of the application-executing terminal 4 are transmitted to the remote conference server 1 (S18).

After transmitting the recorded data, the application executing terminal 4 waits for a reply from the remote conference server 1 (S19, S21, S22). When the application executing terminal 4 receives employee information from the remote conference server 1 (S19, Yes), the application executing terminal 4 displays the received employee information on the display 418 (display area 441) as described in Fig. 12. When the application executing terminal 4 receives no-match information from the remote conference server 1 (S21, Yes), or when no information is transmitted from the remote conference server 1 for a specified period of time (S22, Yes), the application executing terminal 4 displays a no-match message 447a on the display 418 (display area 441) as described in Fig. 13 (S23). The application executing terminal 4 performs the above process until the application is terminated (S24, Yes).
By the application executing terminal 4 executing the above process, a search for a search target person is performed in the remote conference server 1 based on the voice data recorded by the application executing terminal 4 .

<Processing of the teleconference server 1 during search processing>
Next, a description will be given of the process of the teleconference server 1 during search processing. This process is realized, for example, by the CPU 101 loading a computer program stored in the HD 104 into the RAM 103 and executing it. Fig. 17 is a flowchart showing the process of the teleconference server 1 during search.
17, in the process of searching for a search target person, the remote conference server 1 judges whether or not it has received the audio recording data and the terminal location information transmitted by the application executing terminal 4 (S31). The remote conference server 1 waits until it receives the audio recording data, etc. (S31, No), and when it receives the audio recording data, etc. (S31, Yes), it compares the voiceprint of the audio recording data with the voiceprint of the voice data stored in the voice information DB 104a (S32). In the process of S32, in order to improve the efficiency of the search, the remote conference server 1 assigns a priority to each voice data registered in the voice information DB 104a, and compares the voice data with the voice data with the higher priority before the voice data with the lower priority.

For example, when performing a search (when performing voiceprint matching), the remote conference server 1 prioritizes comparison of voice data of a user (employee) of a remote conference implementation terminal 3 located near the application execution terminal 4 over voice data of a user (employee) of a remote conference implementation terminal 3 located farther away than the remote conference implementation terminal 3. The remote conference server 1 also prioritizes comparison of voice data of an employee who belongs to the same department as the user (employee) of the application execution terminal 4 over voice data of an employee who belongs to a different department. Furthermore, the remote conference server 1 refers to the employee schedule information stored in the voice information DB 104a, and when performing a search (when performing voiceprint matching), prioritizes comparison of voice data of an employee who has the same schedule as the user (employee) of the application execution terminal 4 over voice data of an employee who has a different schedule.

The remote conference server 1 determines whether there is voice data whose voiceprint matches the recorded data (S33), and if there is no matching voice data (S33, No), it proceeds to the process of S39 described below. On the other hand, if there is matching voice data (S33, Yes), it obtains the employee ID of the voice data (S34) and sends the obtained employee ID to the employee information server 2 (S35).
After transmitting the employee ID to the employee information server 2, the remote conference server 1 waits for a reply from the employee information server 2 (S36, S38). When employee information is received from the employee information server 2 (S36, Yes), the remote conference server 1 transmits the received employee information to the application executing terminal 4 (S37) and then proceeds to the process of S31. When employee information cannot be received even after a specified time has elapsed (S38, Yes) or when it is determined in the process of S33 that there is no matching voice data (S33, No), information indicating no match is transmitted to the application executing terminal 4 (S39) and then proceeds to the process of S31.
By executing the above process, the remote conference server 1 compares the voiceprints of the recorded data and the voice data in the voice information DB 104a, and when the voiceprints match, the employee information received from the employee information server 2 is transmitted to the application executing terminal 4. On the other hand, when the voiceprints do not match, information indicating no match is transmitted to the application executing terminal 4.

<Processing of Employee Information Server 2 During Search Processing>
Next, a description will be given of the process performed by the employee information server 2 during a search process. This process is realized, for example, by the CPU 201 loading a computer program stored in the HD 204 onto the RAM 203 and executing it. Fig. 18 is a flowchart showing the process performed by the employee information server 2 during a search.
18, in the process of searching for a search target person, the employee information server 2 determines whether or not it has received the employee ID transmitted by the remote conference server 1 (S51). The employee information server 2 waits until it receives the employee ID (S51, No), and upon receiving the employee ID (S51, Yes), it acquires employee information corresponding to the employee ID from the employee information DB 204a (S52) and transmits the acquired employee information to the remote conference server 1 (S53). After transmitting the employee information, the employee information server 2 proceeds to the process of S51.
By executing the above process by the employee information server 2, the employee information corresponding to the employee ID is transmitted to the remote conference server 1.

<About Modifications>
In the above embodiment, an in-company conference system has been described as an example, but the present invention is not limited to in-company conference systems, and can be similarly applied to conference systems used outside of companies.
Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to execute each function described above.
The group of devices described in the above-mentioned embodiment (remote conference server 1, employee information server 2, remote conference implementation terminal 3, and application execution terminal 4) merely represents one of multiple computing environments for implementing the embodiments disclosed in this specification.

In one embodiment, the remote conference server 1 and the employee information server 2 include multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, a shared memory, and the like, and perform the processes disclosed herein. Similarly, the remote conference terminal 3 and the application execution terminal 4 may also include multiple computing devices configured to communicate with each other.
Furthermore, the above-mentioned device group can be configured to share the disclosed processing steps (e.g., the processing of Figs. 15 to 18) in various combinations. For example, part of the processing of the remote conference server 1 may be performed by the employee information server 2, or vice versa. Furthermore, the remote conference server 1 and the employee information server 2 may be integrated into one server, or may be distributed across three or more servers.
In addition, the above-mentioned device group is not limited to the above-mentioned configuration, as long as it is a device equipped with a communication function. The above-mentioned device group may be, for example, a PJ (Projector), an IWB (Interactive White Board: a white board having an electronic blackboard function capable of mutual communication), an output device such as a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collection device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, etc.
Machine learning may be applied to certain processes such as prioritization when the remote conference server 1 searches the voice information DB 104a.

Here, machine learning is a technology that allows computers to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data identification from training data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable.

[Summary of embodiment examples, actions and effects of the present invention]
<First embodiment>
The search terminal device (application execution terminal 4) according to this embodiment is communicatively connected to an external device EX (remote conference server 1, employee information server 2) that stores a plurality of candidate voice data (employee voice data) based on the voices of a plurality of candidates (employees registered in the voice information DB 104a) in association with personal information of each candidate (employee information registered in the employee information DB 204a), and receives and displays personal information of a candidate searched for from the plurality of candidates from the external device EX. The search terminal device (application execution terminal 4) according to this embodiment is characterized in that the search terminal device includes: an acquisition means (S16) for acquiring target person voice data (recorded data) based on the voice of a person to be searched; a request means (S18) for extracting a candidate voice data whose voiceprint matches the target person voice data from the plurality of candidate voice data by transmitting the target person voice data to the external device EX, and for transmitting personal information (employee information) associated with the candidate voice data (S37); a receiving means (S19) for receiving the personal information transmitted by the external device EX; and a display means (S20) for displaying the personal information received by the receiving means.
According to the search terminal device of this embodiment, the voiceprints of the candidate voice data pre-stored in the external device EX are compared with the voiceprints of the target person voice data obtained by recording, making it possible to perform a search without prompting the candidate to speak a specified sound.

<Second embodiment>
In the search terminal device of this embodiment, the personal information (employee information) is characterized by including name information indicating the candidate's name and candidate organization information (department information) indicating the organization to which the candidate belongs.
According to the search terminal device of this aspect, the relationship with the user of the search terminal device can be easily understood based on the name information and the candidate organization information.

<Third embodiment>
In the search terminal device of this embodiment, the request means (S18) transmits terminal location information (terminal location information of the application executing terminal 4) indicating the current location of the search terminal device to the external device EX, and the external device EX determines a priority when comparing candidate voice data based on candidate location information (location information of the remote conference implementation terminal 3) indicating the current location of the candidate and the terminal location information.
According to the search terminal device of this embodiment, candidate voice data for candidates who are close to the search terminal device is compared earlier than for candidates who are far away, thereby making the search process more efficient.

<Fourth embodiment>
In the search terminal device according to this aspect, the candidate location information is information indicating the current location of the candidate terminal device being used by the candidate.
According to the search terminal device of this aspect, candidate position information can be easily obtained.

<Fifth embodiment>
In the search terminal device of this embodiment, the personal information includes candidate organization information (department information) indicating the organization to which the candidate belongs, and the external device EX determines priorities when searching for candidate voice data based on user organization information indicating the organization to which the user of the search terminal device belongs and the candidate organization information.
According to the search terminal device of this embodiment, voice data of candidates who belong to the same organization as the user of the search terminal device can be processed prior to voice data of candidates who belong to a different organization, thereby making the search process more efficient.

<Sixth embodiment>
In the search terminal device of this embodiment, the personal information includes candidate schedule information indicating the candidate's schedule, and the external device EX determines priority when searching for candidate voice data based on the candidate schedule information.
According to the search terminal device of this aspect, the priority order for searching voice data is determined based on the schedules of candidates, thereby making it possible to improve the efficiency of search processing.

<Seventh embodiment>
The search system according to this aspect is characterized by including the search terminal device according to each of the above aspects and an external device EX.
The search system according to this aspect exerts the same effects as the search terminal device according to each of the above aspects.

Eighth embodiment
In the search system of this embodiment, when the external device EX receives user voice data based on the voice of a user of a terminal device connected to it in a communicative manner, it compares the voiceprint of the user voice data with the voiceprints of multiple candidate voice data, and if the voiceprints do not match, it associates the user voice data with the user's personal information and stores it as new candidate voice data.
According to the search system of this embodiment, the candidate voice data can be easily increased.

<Ninth embodiment>
The search method according to the present embodiment is characterized by comprising: an acquisition step (S16) of causing a search terminal device to acquire target person voice data based on the voice of the person being searched for; a first transmission step (S18) of causing the search terminal device to transmit the target person voice data to an external device EX which stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of the candidates; a search step (S32) of causing the external device EX to search for one candidate voice data whose voiceprint matches the target person voice data from among the plurality of candidate voice data; a second transmission step (S35) of causing the search terminal device to transmit personal information associated with the one candidate voice data; and a display step (S20) of displaying the personal information on the search terminal device.
According to the search method of this embodiment, it is possible to perform a search without prompting the candidate to utter a predetermined sound.

1... remote conference server, 2... employee information server, 3... remote conference implementation terminal, 4... application execution terminal, 104a... voice information DB (database), 204a... employee information DB (database)

Patent No. 5728456

Claims (9)

A search terminal device that is communicatively connected to an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and receives and displays personal information of a candidate searched for from the plurality of candidates from the external device,
An acquisition means for acquiring target person voice data based on the voice of a search target person;
a request means for extracting one candidate voice data whose voiceprint matches that of the candidate voice data from among the plurality of candidate voice data by transmitting the candidate voice data to the external device, and for transmitting the personal information associated with the one candidate voice data;
a receiving means for receiving the personal information transmitted by the external device;
a display means for displaying the personal information received by the receiving means;
Equipped with
The request means transmits terminal location information indicating a current location of the search terminal device to the external device,
The external device determines priorities for searching the candidate voice data based on candidate location information indicating the current location of the candidate and the terminal location information. 2. The search terminal device according to claim 1, wherein the candidate location information is information indicating a current location of a candidate terminal device used by the candidate. A search terminal device that is communicatively connected to an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and receives and displays personal information of a candidate searched for from the plurality of candidates from the external device,
An acquisition means for acquiring target person voice data based on the voice of a search target person;
a request means for extracting one candidate voice data whose voiceprint matches that of the candidate voice data from among the plurality of candidate voice data by transmitting the candidate voice data to the external device, and for transmitting the personal information associated with the one candidate voice data;
a receiving means for receiving the personal information transmitted by the external device;
a display means for displaying the personal information received by the receiving means;
Equipped with
The personal information includes candidate organization information indicating an organization to which the candidate belongs,
The external device is characterized in that it determines a priority when searching for the candidate voice data based on user organization information indicating the organization to which the user of the search terminal device belongs and the candidate organization information. A search terminal device that is communicatively connected to an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and receives and displays personal information of a candidate searched for from the plurality of candidates from the external device,
An acquisition means for acquiring target person voice data based on the voice of a search target person;
a request means for extracting one candidate voice data whose voiceprint matches that of the candidate voice data from among the plurality of candidate voice data by transmitting the candidate voice data to the external device, and for transmitting the personal information associated with the one candidate voice data;
a receiving means for receiving the personal information transmitted by the external device;
a display means for displaying the personal information received by the receiving means;
Equipped with
The personal information includes candidate schedule information indicating the schedule of the candidate;
The external device determines a priority order when searching for the candidate voice data based on the candidate schedule information. A search system comprising a search terminal device according to any one of claims 1 to 4 and an external device. The external device is
When receiving user voice data based on the voice of a user of the terminal device from the terminal device connected in a communicable manner, the voiceprint of the user voice data is compared with the voiceprints of the plurality of candidate voice data;
6. The search system according to claim 5, wherein when the voiceprints do not match, the user voice data is associated with personal information of the user and stored as new candidate voice data. A person search method using an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and a search terminal device that is communicatively connected to the external device and receives from the external device and displays personal information of a candidate searched for from among the plurality of candidates,
The search terminal device
An acquisition step of acquiring target person voice data based on a voice of a search target person;
a request step of extracting one candidate voice data having a voiceprint matching the candidate voice data from among the plurality of candidate voice data by transmitting the candidate voice data to the external device, and transmitting the personal information associated with the one candidate voice data;
a receiving step of receiving the personal information transmitted by the external device;
a display step of displaying the personal information received in the receiving step ;
and in the request step, transmitting terminal location information indicating a current location of the searching terminal device to the external device;
The external device is
A person search method comprising a step of determining a priority order when searching for the candidate voice data based on candidate location information indicating the current location of the candidate and the terminal location information. A person search method using an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and a search terminal device that is communicatively connected to the external device and receives from the external device and displays personal information of a candidate searched for from among the plurality of candidates,
The search terminal device
An acquisition step of acquiring target person voice data based on a voice of a search target person;
a request step of extracting one candidate voice data having a voiceprint matching the candidate voice data from among the plurality of candidate voice data by transmitting the candidate voice data to the external device, and transmitting the personal information associated with the one candidate voice data;
a receiving step of receiving the personal information transmitted by the external device;
a display step of displaying the personal information received in the receiving step ;
and the personal information includes candidate organization information indicating an organization to which the candidate belongs;
The external device is
A person search method comprising a step of determining a priority order when searching the candidate voice data based on user organization information indicating an organization to which a user of the search terminal device belongs and the candidate organization information. A person search method using an external device that stores a plurality of candidate voice data based on the voices of a plurality of candidates in association with personal information of each of the candidates, and a search terminal device that is communicatively connected to the external device and receives from the external device and displays personal information of a candidate searched for from among the plurality of candidates,
The search terminal device
An acquisition step of acquiring target person voice data based on a voice of a search target person;
a request step of extracting one candidate voice data having a voiceprint matching the target voice data from among the plurality of candidate voice data by transmitting the target voice data to the external device, and transmitting the personal information associated with the one candidate voice data;
a receiving step of receiving the personal information transmitted by the external device;
a display step of displaying the personal information received in the receiving step ;
and the personal information includes candidate schedule information indicating the schedule of the candidate;
The external device is
A person search method comprising a step of determining a priority order for searching the candidate voice data based on the candidate schedule information.

Images