JP7774279B2 - Memorandum Creation Program - Google Patents

Description

The present invention relates to a memo creation program. More particularly, the present invention relates to a memo creation program that creates a memo of a conversation that includes multiple topics.

When a conversation such as a meeting, interview, or meeting takes place, memorandums such as minutes, notes, or contracts are created to record the content of the conversation. Memorandums are usually created after the conversation has finished, while reviewing what the participants said. For this reason, creating memoranda takes a considerable amount of time after the conversation has finished. There is a demand for memoranda to be created quickly.

Patent documents 1 and 2 listed below propose devices that identify speakers based on the audio of what is said during a conversation and transcribe the audio of what is said during the conversation to create meeting minutes.

Patent Document 1 below discloses a minutes-creation system that acquires speech content from a meeting as audio information, processes the audio information, embeds speaker information in the processed information, and generates minutes based on the information with the embedded speaker information.

Patent Document 2 below discloses a minutes output device that includes an information acquisition unit, a voice acquisition unit, a voice recognition unit, a discrimination unit, and an output control unit. The information acquisition unit acquires information about the number of participants in a meeting. The voice acquisition unit acquires data related to the voice in the meeting. The voice recognition unit recognizes voice based on the voice data acquired by the voice acquisition unit and converts it into text as the speaker's remarks. The discrimination unit identifies the speaker based on the information about the number of participants acquired by the information acquisition unit and the voice data acquired by the voice acquisition unit. The output control unit outputs minutes that associate a label indicating the speaker identified by the discrimination unit with the content of the remarks converted into text by the voice recognition unit.

Japanese Patent Application Publication No. 10-294798 Japanese Patent Application Laid-Open No. 2020-95210

In conventional technology, memos contain text strings converted from speech, but the correspondence between the text strings and each topic is unclear. To clarify the correspondence between the text strings and each topic, the creator of the memo had to read the entire text string and add text indicating each topic at the appropriate starting point of each topic in the memo. This made the conventional technology less convenient.

The present invention is intended to solve the above problems, and its purpose is to provide a highly convenient memorandum creation program.

A memo creation program according to one aspect of the present invention is a memo creation program for creating a memo of a conversation including multiple topics, the memo creation program including: an acquisition step for acquiring audio of utterances made during the conversation; a conversion step for converting the audio acquired in the acquisition step into a character string; an identification step for identifying the speaker of the audio acquired in the acquisition step based on characteristics of the audio acquired in the acquisition step; a character string addition step for associating the character string converted in the conversion step with the speaker identified in the identification step and adding the character string converted in the conversion step in chronological order to a memo stored in a storage device; and a display step for displaying on a display unit a screen including a first area displaying the multiple topics and an ongoing topic among the multiple topics, and a second area displaying the memo stored in the storage device, the second area being an area different from the first area. The computer is caused to execute an input acceptance step for accepting a predetermined input during a conversation, a topic addition step for adding a symbol indicating a topic break after the latest character string added in the character string addition step in a memo stored in a storage device if the predetermined input is accepted in the input acceptance step, and an end acceptance step for accepting an input to end the conversation; if an input to end the conversation is not accepted in the end acceptance step after the symbol is added to the memo in the topic addition step, an acquisition step acquires audio of utterances made during the conversation , and the input acceptance step accepts as predetermined input a selection of the next topic after the ongoing topic from among multiple topics displayed in a first area, or the addition of a check mark in a checkbox corresponding to the next topic displayed in the first area .

In the above-mentioned memorandum creation program, preferably, in the display step, the character string converted in the conversion step is displayed on the display unit in a different manner for each speaker.

In the memo creation program, the symbol preferably includes a character string of one of a plurality of topics stored in the storage device.

In the above-mentioned memorandum creation program, the computer preferably further executes a pre-start acquisition step of acquiring the voices of the participants in the conversation before the conversation begins, and a feature registration step of registering the voice features acquired in the pre-start acquisition step in a storage device in association with the participants, and in the identification step, the speaker of the voice acquired in the acquisition step is identified based on the voice features registered in the feature registration step.

In the above memorandum creation program, preferably, the computer is further caused to execute, after the conversation ends, an approval receiving step for receiving input approving the memorandum from the speakers identified in the identification step, and a confirmation step for confirming the memorandum if approval inputs have been received from all speakers identified in the identification step in the approval receiving step.

In the above-mentioned memorandum creation program, the approval acceptance step preferably includes a first approval acceptance step of accepting, on a computer on which the memorandum creation program is installed, an input approving the memorandum from the speaker identified in the identification step, and a second approval acceptance step of accepting, via a network line, an input approving the memorandum from the speaker identified in the identification step from a computer other than the computer on which the memorandum creation program is installed.

In the above-mentioned memorandum creation program, the acquisition step preferably includes a first acquisition step of acquiring audio of utterances made during a conversation on a computer on which the memorandum creation program is installed, and a second acquisition step of acquiring audio of utterances made during a conversation via a network line from a computer other than the computer on which the memorandum creation program is installed.

In the above-mentioned memorandum creation program, the computer preferably further executes a screen transmission step of transmitting screen data displaying the ongoing topic among the multiple topics via a network line to a computer other than the computer on which the memorandum creation program is installed.

The present invention provides a highly convenient memorandum creation program.

1 is a diagram schematically illustrating a usage mode of a memo creation device 1 according to a first embodiment of the present invention. 1 is a block diagram showing a configuration of a memo creation device 1 according to a first embodiment of the present invention. 10 is a diagram schematically showing a speaker information table stored in a storage device 14. FIG. FIG. 2 is a diagram schematically illustrating a conference menu table stored in the storage device 14. FIG. 3 is a diagram showing a home screen 31 of a memo creation application according to the first embodiment of the present invention. FIG. 3 is a diagram showing a conference setting screen 32 of a memorandum creation application according to the first embodiment of the present invention. FIG. 10 is a diagram showing a first state of a minutes creation screen 33 of a memorandum creation application according to the first embodiment of the present invention. FIG. 10 is a diagram showing a second state of the minutes creation screen 33 of the memorandum creation application in the first embodiment of the present invention. FIG. 10 is a diagram showing a minutes approval screen 34 of the memorandum creation application according to the first embodiment of the present invention. 4 is a flowchart showing a process performed by the memo creating device 1 in a conference mode according to the first embodiment of the present invention. This is a subroutine of the character conversion process (S113) in FIG. FIG. 3 is a diagram showing a voice registration screen 35 of a memo creation application according to the first embodiment of the present invention. 4 is a flowchart showing a process performed by the memo creating device 1 in a voice registration mode according to the first embodiment of the present invention. FIG. 10 is a diagram showing a conference menu creation screen 37 of a memorandum creation application in the first embodiment of the present invention. 4 is a flowchart showing a process performed by the memo creation device 1 in a meeting menu creation mode according to the first embodiment of the present invention. FIG. 10 is a diagram showing a minutes selection screen 38 of the memorandum creation application according to the first embodiment of the present invention. FIG. 10 is a diagram showing a distribution destination input screen 39 of a memorandum creation application according to the first embodiment of the present invention. 10 is a flowchart showing a process performed by the memorandum creation device 1 in a minutes distribution mode according to the first embodiment of the present invention. FIG. 10 is a diagram illustrating the operation of the memo creation device 1 according to the second embodiment of the present invention. FIG. 11 is a diagram showing a state in which a check mark is added to a check box 290 on a conference setting screen 32 of a memo creation application in the second embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings.

The memorandum creation program in the following embodiments creates memoranda of conversations that include multiple topics. "Memorandums" refer to things written down to help you remember, and include things like minutes, memos, and contracts. "Conversations" refer to conversations between two or more people, and include things like meetings, interviews, and face-to-face meetings.

In the following embodiments, an example will be described in which the conversation targeted by the memorandum creation program is a meeting, the memorandum is a meeting minutes, and the topic of the conversation is the meeting agenda. The type of conversation handled by the memorandum creation program is arbitrary, and may include a meeting or interview.

[First embodiment]

(Memorandum creation device 1 usage)

Figure 1 is a diagram that schematically illustrates how the memo creation device 1 according to the first embodiment of the present invention is used.

Referring to FIG. 1, memo creation device 1 comprises a terminal such as a PC (Personal Computer) or a smartphone. In this embodiment, at least user UR possesses memo creation device 1. User UR is, for example, one of the participants in a conference who prepares the minutes. As shown in FIG. 1(a), memo creation device 1 may be placed in a position (preferably near user UR) where the voices of each participant are acquired in a conference with multiple participants in real space. As shown in FIG. 1(b), memo creation device 1 may be placed near terminal 3 used by user UR in a web conference. Terminal 3 itself may be memo creation device 1. A web conference is a conference held by multiple participants at different locations, connecting their terminals 3 to each other via network lines 7 and transmitting and receiving at least the voices of each of the multiple participants.

Network line 7 may be, for example, a dedicated line such as a wired or wireless LAN (Local Area Network), a public line, or wireless communication. Network line 7 connects various devices using the TCP/IP (Transmission Control Protocol/Internet Protocol) protocol. Devices connected to network line 7 can exchange various types of data with each other.

(Configuration of memorandum creation device 1)

Figure 2 is a block diagram showing the configuration of the memo creation device 1 in the first embodiment of the present invention. Figure 2(a) is a block diagram showing the hardware configuration of the memo creation device 1. Figure 2(b) is a block diagram showing the functional configuration of the memo creation device 1.

Referring to FIG. 2(a), the memorandum creation device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage device 14, a network interface 15, an operation unit 16, a display unit 17, and a microphone 18. The ROM 12, RAM 13, storage device 14, network interface 15, operation unit 16, display unit 17, and microphone 18 are each connected to the CPU 11 via a bus.

The CPU 11 controls the entire memo creation device 1. The CPU 11 also executes programs stored in the ROM 12 and the storage device 14.

ROM 12 is, for example, a flash ROM. ROM 12 stores various programs executed by CPU 11 and various fixed data. ROM 12 may be non-rewritable.

RAM 13 is the main memory of CPU 11. RAM 13 is used to temporarily store data required when CPU 11 executes a program.

The storage device 14 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores various data such as the memo creation program 50 and data related to the memo creation program 50 (such as minutes, speaker information tables, and conference menu tables). The memo creation program 50 is a program related to the memo creation app.

The network interface 15 communicates with other devices using communication protocols such as TCP/IP in accordance with instructions from the CPU 11.

The operation unit 16 accepts various instructions from the user.

The display unit 17 displays various information.

The microphone 18 captures sounds around the memo creation device 1 and converts them into electrical signals.

Referring to FIG. 2(b), the memorandum creation device 1 includes a voice acquisition unit 501, a conversion unit 502, an identification unit 503, a character string addition unit 504, an operation and display unit 505, a topic addition unit 506, a feature registration unit 507, an approval reception unit 508, a communication unit 509, and a translation unit 510. Each of the voice acquisition unit 501, the conversion unit 502, the identification unit 503, the character string addition unit 504, the operation and display unit 505, the topic addition unit 506, the feature registration unit 507, the approval reception unit 508, the communication unit 509, and the translation unit 510 is realized by the CPU 11 loading the memorandum creation program 50 into the RAM 13 and executing it.

The voice acquisition unit 501 acquires the voices of those speaking during a conversation using the microphone 18. Before the conversation begins, the voice acquisition unit 501 acquires the voices of the participants in the conversation using the microphone 18.

The conversion unit 502 converts the audio acquired by the audio acquisition unit 501 into a string of characters.

The identification unit 503 extracts feature data from the audio acquired by the audio acquisition unit 501. This feature data is also called voiceprint data. The identification unit 503 identifies the speaker of the audio acquired by the audio acquisition unit 501 based on the extracted feature data.

The identification unit 503 may extract feature data, for example, using the following method. The identification unit 503 digitizes the audio waveform acquired by the audio acquisition unit 501 and divides the digitized audio waveform into blocks at regular time intervals. The identification unit 503 calculates a feature vector from each block and extracts the calculated feature vector as feature data.

The character string adding unit 504 associates the character string converted by the conversion unit 502 or the character string translated by the translation unit 510 with the speaker identified by the identification unit 503 and adds them to the memo in chronological order.

The operation display unit 505 accepts various operations via the operation unit 16 and displays various screens on the display unit 17. The operation display unit 505 accepts operations such as additional input, which is an input for adding a topic to a memo during a conversation. The operation display unit 505 displays the screen of the memo creation app on the display unit 17.

When additional input is received via the operation display unit 505, the topic addition unit 506 adds a symbol indicating a topic break to the memo.

The feature registration unit 507 associates the features of the participants' voices acquired by the voice acquisition unit 501 before the conversation begins with those participants and registers them in the speaker information table.

After the conversation ends, the approval receiving unit 508 receives input approving the memorandum from the speakers identified by the identification unit 503. The approval receiving unit 508 finalizes the memorandum when it has received approval input from all speakers.

The communication unit 509 communicates with other terminals other than the memo creation device 1 regarding the creation and transmission of memoranda.

In certain cases, the translation unit 510 translates the character string converted by the conversion unit 502 into the set language.

Figure 3 is a diagram showing a speaker information table stored in the storage device 14.

Referring to Figure 3, the speaker information table is a table in which speaker information is registered. The speaker information includes the speaker's name, feature data of the speaker's voice, and the display color of the text of the speaker's utterance. In this example, the speaker information table includes four pieces of speaker information that have been previously registered. For example, the speaker information listed at the top of the four pieces of speaker information includes the speaker name "A," feature data named "Data a," and the display color "black."

In certain cases, the memo creation device 1 extracts feature data based on the acquired audio. The feature data may be, for example, a feature vector calculated based on the audio waveform. If the extracted feature data is new feature data that is not registered in the speaker information table, the memo creation device 1 registers new speaker information including the new feature data in the speaker information table.

Figure 4 is a diagram showing a schematic diagram of a conference menu table stored in the storage device 14.

Referring to Figure 4, the conference menu table is a table in which conference menus are registered. A conference menu includes a type of conference and multiple agenda items to be discussed at that conference type. Here, the conference menu table includes three conference menu items that have been previously registered. For example, the topmost of the three conference menu items includes the conference type "Regular Meeting" and the agenda items related to this conference type: "1. Confirmation of Sales," "2. Confirmation of Projected Sales," and "3. Presentation of Issues from Each Department." The memorandum creation device 1 may add new conference menu items to the conference menu table in certain cases.

Note that the order of deliberation for multiple topics in the meeting menu may or may not be set, as shown in Figure 4.

(Home screen mode)

Figure 5 shows the home screen 31 of the memo creation app in the first embodiment of the present invention.

Referring to FIG. 5, the user UR performs a predetermined input (for example, pressing the icon of the memo creation app) through the operation unit 16. When the predetermined input is received, the memo creation device 1 launches the memo creation app, enters home screen mode, and displays the home screen 31 of the memo creation app on the display unit 17.

Home screen 31 includes message 20 prompting the user to select a mode, and buttons 21, 23, 25, and 27. Button 21 is used to select conference mode. Button 23 is used to select voice registration mode. Button 25 is used to select conference menu creation mode. Button 27 is used to select minutes distribution mode.

(Conference mode)

When starting a conference, the user UR presses button 21 on the home screen 31 via the operation unit 16. When button 21 is pressed, the memo creation device 1 transitions to conference mode and first displays the conference setting screen 32 shown in Figure 4 on the display unit 17.

Figure 6 shows the meeting setting screen 32 of the memo creation app in the first embodiment of the present invention.

Referring to Figure 6, the conference setting screen 32 is a screen for setting up the conference and minutes, and includes a list box 210, a check box 211, a list box 212, buttons 213 and 214, and a check box 290, which will be described later.

List box 210 accepts the setting of the meeting type. The meeting types registered in the meeting menu table (Figure 4) are displayed in list box 210. In this example, list box 210 displays the meeting types "Regular Meeting," "Sales Contract," and "Labor-Management Agreement," and the meeting type "Regular Meeting" is set.

Checkbox 211 accepts the setting of whether or not to standardize the language of the minutes.

List box 212 accepts the language setting when a check mark is placed in check box 211. In this example, list box 212 displays the options "Japanese" and "English," with "Japanese" being set as the language. When check box 211 is unchecked, list box 212 is grayed out and cannot be set.

Button 213 is used to confirm the settings. Button 214 is used to discard the settings.

After configuring the meeting and minutes settings on the meeting settings screen 32, the user UR presses button 213 via the operation unit 16 to start the meeting. When button 213 is pressed, the memo creation device 1 confirms the settings, creates new minutes in the storage device 14, and begins capturing audio of comments made during the meeting. The memo creation device 1 also displays the minutes creation screen 33 shown in Figure 7 on the display unit 17. When button 214 is pressed, the memo creation device 1 discards the settings and returns to the home screen 31.

Figures 7 and 8 are diagrams showing the minutes creation screen 33 of the memo creation app in the first embodiment of the present invention. Note that in Figures 7, 9, and 17, differences in the display color of the text in the minutes text box are represented by different hatching applied to the text.

Referring to Figure 7, the minutes creation screen 33 is a screen that is displayed while minutes are being created (in other words, during the meeting), and includes a title 215, a check box 216, a text box 217, and a scroll bar 218.

Title 215 displays the set meeting type and the topic being discussed (in other words, the topic in progress). Here, the meeting type "Regular Meeting" and the topic "Sales Confirmation," which is the first topic for the meeting type "Regular Meeting," are displayed. By displaying the topic being discussed on the minutes-taking screen 33 in this way, anyone viewing the minutes-taking screen 33 can easily understand the topic being discussed. The topic being discussed included in title 215 is displayed in a different manner from the topic added to check box 216 (for example, in a larger size than the topic added to check box 216).

There are as many check boxes 216 as there are agenda items. Each check box 216 displays the corresponding agenda item. In this example, three check boxes 216 are provided. Each check box 216 corresponds to one of the three agenda items for the "Regular Meeting" meeting type: "Sales Confirmation," "Expected Sales Confirmation," and "Raising Issues from Each Department."

The text box 217 displays the minutes stored in the storage device 14. The portion of the minutes displayed in the text box 217 can be changed by moving the scroll bar 218.

When user UR begins discussion on the first agenda item, he or she places a check mark in the check box 216 corresponding to the first agenda item on the minutes creation screen 33 via the operation unit 16. When a check mark is placed in the check box 216, the memo creation device 1 adds a symbol 219 indicating the agenda item corresponding to that check box 216 to the minutes stored in the storage device 14, and updates the minutes displayed in the text box 217 to the latest version. In this example, user UR places a check mark in the check box 216a corresponding to the first agenda item, "1. Sales Confirmation." The memo creation device 1 adds a symbol 219 containing the agenda order "1. Sales Confirmation" and the agenda text string to the beginning of the minutes.

Note that if the order of agenda items is set in the meeting menu, the symbol 219 indicating the first agenda item may be added to the minutes stored in the storage device 14 without input by the user UR.

When the memo creation device 1 acquires audio of speech made during a conversation through the microphone 18, it converts the acquired audio into text. If the setting to standardize the language of minutes is enabled on the conference setting screen 32 and audio is acquired in a language different from the set language, the memo creation device 1 converts the acquired audio into text in the language of the audio and then translates the converted text into the set language.

Furthermore, when the memo creation device 1 acquires audio of speech during a conversation through the microphone 18, it extracts feature data of the speech from the acquired audio by the audio acquisition unit 501. The memo creation device 1 compares the extracted feature data with the speaker information table. If speaker information that matches the extracted feature data is registered in the speaker information table, the memo creation device 1 uses that speaker information to identify the speaker and display color of the converted character string. On the other hand, if speaker information that matches the extracted feature data is not registered in the speaker information table, the memo creation device 1 sets arbitrary speaker names and display colors in the speaker information, and then registers new speaker information including the extracted feature data in the speaker information table. The memo creation device 1 uses the new speaker information to identify the speaker and display color of the converted character string.

The memorandum creation device 1 associates the converted character string with the identified speaker, adds it to the minutes stored in the storage device 14 in chronological order (for example, with the character strings of the remarks arranged from oldest to newest in the minutes), and updates the minutes displayed in the text box 217 to the latest one. The character string of each speaker in the minutes displayed in the text box 217 is displayed in the display color specified in the speaker information for that speaker. In this example, black character string related to the remarks of speaker "A" and blue character string related to the remarks of speaker "B" have been added to the minutes in the text box 217. The display color may also be reflected in the character string of each speaker in the minutes stored in the storage device 14. Note that the converted character string is preferably displayed differently for each speaker, and the character string may be displayed so that the font, size, etc., in addition to the display color, differ for each speaker.

The memorandum creation device 1 may accept amendments to the minutes displayed in the text box 217 via the operation unit 16. Amendments to the minutes displayed in the text box 217 are reflected in the minutes stored in the storage device 14.

Referring to FIG. 8 , when the user UR moves on to the next agenda item during a meeting, the user UR places a check mark in the check box 216 corresponding to the next agenda item on the minutes creation screen 33 via the operation unit 16. When a check mark is placed in the check box 216, the memo creation device 1 adds a symbol 219 indicating the agenda item corresponding to the check box 216, for example, after the most recently converted character string in the minutes stored in the storage device 14, and updates the minutes displayed in the text box 217 to the latest one. In this example, the user UR places a check mark in the check box 216b corresponding to the next agenda item, "2. Confirmation of expected sales." The memo creation device 1 adds the symbol 219, which includes the order of the agenda item "2. Confirmation of expected sales" and the string of the agenda item, after the string related to the statement by the speaker "A" in the minutes, "A: Since there seems to be no other opinions, let's move on to the next agenda item."

The input that triggers adding an agenda item to the minutes is arbitrary, and may be the above-mentioned adding of a check mark, the acquisition of audio reading the next agenda item, or the selection of an agenda item displayed on the minutes creation screen 33. Also, if the order of agenda items is set in the meeting menu, the input that triggers adding an agenda item to the minutes may be the acquisition of audio indicating that the agenda item will be moved on to the next agenda item, or an operation to move on to the next minutes.

The symbol 219 added to the minutes may be any symbol that indicates a break in a topic, such as the symbol indicating the agenda mentioned above, or a page break symbol, line break symbol, etc. The symbol indicating the agenda may include the character string of the agenda mentioned above, or may be a number indicating the order of the agenda.

When the user UR ends the meeting, they perform a specific input (for example, double-clicking the title 215 via the operation unit 16 or checking the check boxes for all agenda items). When the specific input is received, the memorandum creation device 1 displays the minutes approval screen 34 shown in Figure 9 on the display unit 17.

Figure 9 shows the minutes approval screen 34 of the memorandum creation app in the first embodiment of the present invention.

Referring to Figure 9, the minutes approval screen 34 is a screen for accepting approval of the minutes from speakers listed in the minutes, and includes a text box 220, a scroll bar 221, and multiple buttons 222.

Text box 220 displays the minutes stored in storage device 14. The minutes at the end of the meeting include converted text strings of the speakers' remarks during the meeting, and symbols indicating the agenda inserted at appropriate points in the converted text strings. The portion of the minutes displayed in text box 220 can be changed by moving scroll bar 221.

The memorandum creation device 1 may accept amendments to the minutes displayed in the text box 220 via the operation unit 16. Amendments to the minutes displayed in the text box 220 are reflected in the minutes stored in the storage device 14.

Each of the multiple buttons 222 is pressed by a participant displayed in the text box 220 when they approve the minutes. In this example, there are four speakers during the meeting: "A," "B," "C," and "D," and a button 222 is displayed for each of the four participants.

Any speaker in the meeting can view the minutes displayed in the text box 220, and if they approve the minutes, they can press the button 222 corresponding to them via the operation unit 16. The pressed button 222a is grayed out and can no longer be pressed.

When all buttons 222 have been pressed, the memo creation device 1 finalizes the minutes and ends the conference mode. The memo creation device 1 may finalize the minutes after adding information to the minutes indicating that all participants have approved the minutes (such as the string "All speakers authenticated") to the minutes. The memo creation device 1 may also finalize the minutes after adding information such as characteristic data of each participant and the date and time of the conference. After finalization, the minutes are stored in the storage device 14 in an uneditable state.

The memorandum creation device 1 may also acquire information about the meeting participants through input via the operation unit 16, and finalize the minutes when approval (pressing button 22) is received from all participants.

Figure 10 is a flowchart showing the processing performed by the memo creation device 1 in conference mode in the first embodiment of the present invention.

Referring to FIG. 10, when transitioning to conference mode, the CPU 11 displays the conference setting screen 32 and accepts settings for the conference and minutes (S101). Next, the CPU 11 determines whether the settings have been confirmed (S103).

If it is determined in step S103 that the settings have not been confirmed (discarded) (NO in S103), the CPU 11 displays the home screen 31 and ends the process.

If it is determined in step S103 that the settings have been confirmed (YES in S103), the CPU 11 creates new minutes and stores them in the storage device 14 (S105). Next, the CPU 11 starts capturing audio (S107) and displays the minutes creation screen 33 (S109). The CPU 11 then determines whether audio has been captured (S111).

If it is determined in step S111 that audio has been acquired (YES in S111), the CPU 11 performs a text conversion process (S113) and proceeds to step S115.

If it is determined in step S111 that audio is not to be acquired (NO in S111), the CPU 11 proceeds to processing in step S115.

In step S115, the CPU 11 determines whether input to add an agenda item has been received (S115).

If it is determined in step S115 that input to add an agenda item has been received (YES in S115), the CPU 11 adds the necessary agenda item after the most recently converted character string in the minutes stored in the storage device 14 (S117) and updates the minutes displayed on the conference setting screen 32 (S119). Next, the CPU 11 determines whether input to end the conference has been received (S121).

If it is determined in step S115 that input to add an agenda item has not been accepted (NO in S115), or if it is determined in step S121 that input to end the meeting has not been accepted (NO in S121), the CPU 11 proceeds to processing in step S111.

If it is determined in step S121 that an input to end the meeting has been received (YES in S121), the CPU 11 displays the minutes approval screen 34 and receives approval of the minutes from the participants (S123). Next, the CPU 11 determines whether approval of the minutes has been received from all participants (S125). The CPU 11 repeats the processing of step S125 until it determines that approval of the minutes has been received from all participants.

If it is determined in step S125 that approval of the minutes has been received from all participants (YES in S125), the CPU 11 finalizes the minutes (S127) and terminates processing.

Figure 11 shows a subroutine of the characterization process (S113) in Figure 10.

Referring to FIG. 11, in the character conversion process (S113), the CPU 11 converts the acquired speech into a string of characters in the language of the acquired speech (S151). Next, the CPU 11 determines whether the setting to unify the language of the minutes is enabled (S153).

If it is determined in step S153 that the setting to unify the language of the minutes is enabled (YES in S153), the CPU 11 determines whether the converted audio is in a language different from the set language (S154).

If it is determined in step S154 that the converted speech is in a language different from the set language (YES in S154), the CPU 11 translates the converted character string (S155) and proceeds to processing in step S159.

If it is determined in step S153 that the setting to unify the language of the minutes is not enabled (NO in S153), or if it is determined in step S154 that the converted audio is not in a language different from the set language (NO in S154), the CPU 11 proceeds to processing in step S159.

In step S159, the CPU 11 extracts feature data from the acquired audio (S159) and determines whether speaker information including the feature data is registered in the speaker information table (S161).

If it is determined in step S161 that speaker information including feature data is registered in the speaker information table (YES in S161), the CPU 11 proceeds to processing in step S165.

If it is determined in step S161 that speaker information including feature data is not registered in the speaker information table (NO in S161), the CPU 11 registers new speaker information including feature data in the speaker information table (S163) and proceeds to processing in step S165.

In step S164, the CPU 11 identifies the speaker (S164). Next, the CPU 11 associates the converted or translated character string with the speaker from the identified speaker information and adds it to the minutes stored in the storage device 14 in chronological order (S165). Next, the CPU 11 updates the minutes displayed on the minutes creation screen 33 to the latest version (S167) and returns.

(Voice registration mode)

The memo creation device 1 may register voice feature data of meeting participants in the speaker information table before the meeting begins. This allows for accurate identification of speakers when there are a large number of meeting participants.

Before starting the meeting, a meeting participant presses button 23 on the home screen 31 (Figure 3) via the operation unit 16. When button 23 is pressed, the memo creation device 1 transitions to voice registration mode and first displays voice registration screen 35 shown in Figure 12(a) on the display unit 17.

Figure 12 shows the voice registration screen 35 of the memo creation app in the first embodiment of the present invention.

Referring to FIG. 12(a), the voice registration screen 35 is a screen for registering the voice characteristics data of participants in the speaker information table, and includes a message 231 encouraging speech, and buttons 232 and 233. Button 232 is a button for starting acquisition of the spoken voice. Button 233 is a button for ending the voice registration mode.

A participant in a meeting presses button 232 on the voice registration screen 35 via the operation unit 16 and then makes a predetermined speech. The memo creation device 1 acquires the voice of the speech and extracts feature data from the acquired voice. If the memo creation device 1 successfully extracts the feature data, it displays the speaker name registration screen 36 shown in Figure 12 (b) on the display unit 17.

Referring to FIG. 12(b), the speaker name registration screen 36 is a screen for registering a speaker name and display color in the speaker information table, and includes a message 234 prompting the user to set the speaker name and display color, a text box 235, a combo box 236, and buttons 237 and 238. The text box 235 accepts input of the speaker name. The combo box 236 accepts the setting of the display color from among multiple color candidates. The button 237 is a button for confirming the set content. The button 238 is a button for discarding the content if no setting is made.

Here, text box 235 is left blank (i.e., an arbitrary speaker name is set), and the display color is set to "black."

When a meeting participant wants to set their own participant name and display color in the minutes, they make the necessary settings in text box 235 and combo box 236 via operation unit 16 and then press button 237. When button 237 is pressed, memo creation device 1 registers the set participant name and display color, as well as the extracted feature data, in the speaker information table as new speaker information. On the other hand, when a meeting participant does not want to set their own participant name and display color in the minutes, they press button 238 via operation unit 16. When button 238 is pressed, memo creation device 1 registers an arbitrary participant name and display color different from the participant name and display color already registered in the speaker information table, as well as the extracted feature data, in the speaker information table as new speaker information.

Figure 13 is a flowchart showing the processing performed by the memo creation device 1 in the voice registration mode of the first embodiment of the present invention.

Referring to FIG. 13, when the voice registration mode is entered, the CPU 11 displays the voice registration screen 35 (S201) and determines whether an instruction to start capturing spoken voice has been received (S203).

If it is determined in step S203 that an instruction to start capturing the spoken audio has not been received (NO in S203), the CPU 11 terminates the processing.

If it is determined in step S203 that an instruction to start capturing the spoken voice has been received (YES in S203), the CPU 11 starts capturing the voice and extracting feature data (S205). Next, the CPU 11 determines whether the extraction of the feature data was successful (S207). The CPU 11 repeats the processing of step S207 until it determines that the extraction of the feature data was successful.

If it is determined in step S207 that the extraction of feature data was successful (YES in S207), the CPU 11 displays the speaker name registration screen 36 (S209) and determines whether an operation to set the speaker name and display color has been accepted (S211).

If it is determined in step S211 that an operation to set the speaker name and display color has been accepted (YES in S211), the CPU 11 registers speaker information including the set speaker name and display color and feature data in the speaker information table (S213), and then ends the process.

If it is determined in step S211 that the operation to set the speaker name and display color has not been accepted (NO in S211), the CPU 11 registers speaker information including the optional speaker name, display color, and feature data in the speaker information table (S215), and terminates processing.

(Meeting menu creation mode)

The memo creation device 1 may accept the registration of a new meeting menu in the meeting menu table through the meeting menu creation mode. This allows the user UR to create minutes that match the content of the meeting if the desired meeting menu is not available in the meeting menu table.

Before starting the meeting, a meeting participant presses button 25 on the home screen 31 (Figure 3) via the operation unit 16. When button 25 is pressed, the memo creation device 1 transitions to the meeting menu creation mode, and displays the meeting menu creation screen 37 shown in Figure 14 on the display unit 17.

Figure 14 shows the meeting menu creation screen 37 of the memo creation app in the first embodiment of the present invention.

Referring to FIG. 14, the conference menu creation screen 37 is a screen for registering a conference menu in the conference menu table, and includes text boxes 251 and 252 and buttons 253, 254, 255, and 256. Text box 251 is an input field for the type of conference. Multiple text boxes 252 are provided. Each of the multiple text boxes 252 is an input field for an agenda and accepts input of each of multiple agenda items related to the type of conference entered in text box 251. Button 253 is a button for decreasing the number of text boxes 252 by one. Button 254 is a button for increasing the number of text boxes 252 by one. Button 255 is a button for registering the entered content. Button 256 is a button for discarding the entered content.

Here, the word "interview" has been entered as the type of meeting, and three topics have been entered: "1. Self-introduction," "2. Work history," and "3. Reasons for applying." The order of the topics can be set arbitrarily.

On the meeting menu creation screen 37, the user UR inputs the type of meeting in text box 251 via the operation unit 16, inputs the meeting agenda in text box 252, and then presses button 255. When button 255 is pressed, the memo creation device 1 registers the input meeting agenda and topic as a meeting menu in the meeting menu table. When button 256 is pressed, the memo creation device 1 discards the content input on the meeting menu creation screen 37 and returns to the home screen 31.

Figure 15 is a flowchart showing the processing performed by the memo creation device 1 in the meeting menu creation mode of the first embodiment of the present invention.

Referring to FIG. 15, when transitioning to the meeting menu creation mode, the CPU 11 displays the meeting menu creation screen 37 (S301) and determines whether an operation to increase or decrease the number of agenda input fields has been accepted (S302).

If it is determined in step S302 that an operation to increase or decrease the number of input fields for the agenda has been received (YES in S302), the CPU 11 increases or decreases the number of input fields for the agenda in accordance with the operation (S303) and proceeds to processing in step S305.

If it is determined in step S302 that an operation to increase or decrease the number of agenda input fields is not accepted (NO in S302), the CPU 11 proceeds to processing in step S305.

In step S305, the CPU 11 determines whether an operation to register a conference menu has been accepted (S305).

If it is determined in step S305 that an operation to register a conference menu has been accepted (YES in S305), the CPU 11 registers the input conference menu in the conference menu table (S307) and terminates processing.

If it is determined in step S305 that the operation to register the conference menu has not been accepted (NO in S305), the CPU 11 discards the input conference menu and terminates processing.

(Minutes distribution mode)

The memorandum creation device 1 may distribute the finalized minutes stored in the storage device 14 to the necessary recipients in accordance with operations by the user UR. This allows the user UR to distribute the created minutes to the desired recipients (such as meeting participants).

After the meeting ends, the user UR presses button 27 on the home screen 31 (Figure 3) via the operation unit 16. When button 27 is pressed, the memo creation device 1 transitions to minutes distribution mode and first displays the minutes selection screen 38 shown in Figure 16.

Figure 16 shows the minutes selection screen 38 of the memo creation app in the first embodiment of the present invention.

Referring to FIG. 16, the minutes selection screen 38 is a screen for selecting the minutes to be distributed, and includes a message 271 prompting the user to select the minutes to be distributed, a radio button 272, and buttons 273 and 274. The radio button 272 accepts the selection of the minutes to be distributed from the minutes stored in the storage device 14. The same number of radio buttons 272 as the number of minutes stored in the storage device 14 are displayed, and each radio button 272 corresponds to the file name of one minutes. Here, the file name of the minutes is a combination of the type of meeting related to the minutes and the date and time of the meeting. Button 273 is a button for confirming the selected minutes. Button 274 is a button for discarding the selection of minutes.

Instead of accepting the selection of minutes to be distributed from among the minutes stored in the storage device 14, the memorandum creation device 1 may also accept input of the destination where the minutes to be distributed will be saved (such as the URL of an external server).

On the minutes selection screen 38, the user UR selects the file name of the minutes to be distributed via the operation unit 16 and then presses button 273. When button 273 is pressed, the memo creation device 1 displays the distribution destination input screen 39 on the display unit 17. When button 274 is pressed, the memo creation device 1 discards the content entered on the minutes selection screen 38 and returns to the home screen 31.

Figure 17 shows the distribution destination input screen 39 of the memo creation app in the first embodiment of the present invention.

Referring to Figure 17, the distribution destination input screen 39 is a screen that accepts input of distribution destinations for the minutes, and includes a message 275 prompting input of distribution destinations, a text box 276, a scroll bar 277, a text box 278, and buttons 279 and 280.

Text box 276 displays the selected minutes in an uneditable state. The portion of the minutes displayed in text box 276 can be changed by moving scroll bar 277. Text box 278 is an input field for distribution destinations (email addresses, etc.). The same number of text boxes 278 as the number of speakers included in the selected minutes may be displayed. The number of text boxes 278 may be increased or decreased by input from operation unit 16. Button 279 is a button for confirming the entered distribution destinations. Button 280 is a button for discarding the entered distribution destinations.

The user UR enters the recipients of the minutes on the recipient input screen 39 via the operation unit 16, and then presses button 279. When button 279 is pressed, the memo creation device 1 distributes (sends) the selected minutes to the entered recipients. When button 280 is pressed, the memo creation device 1 discards the content entered on the recipient input screen 39 and returns to the home screen 31.

Figure 18 is a flowchart showing the processing performed by the memorandum creation device 1 in the minutes distribution mode of the first embodiment of the present invention.

Referring to FIG. 18, when transitioning to minutes distribution mode, the CPU 11 displays the minutes selection screen 38 (S401) and determines whether the selected agenda has been confirmed (S403).

If it is determined in step S403 that the selected agenda item has been confirmed (YES in S403), the CPU 11 displays the distribution destination input screen 39 (S405). Next, the CPU 11 determines whether the entered distribution destination has been confirmed (S407).

If it is determined in step S407 that the input distribution destinations have been confirmed (YES in S407), the CPU 11 sends the selected minutes to the input distribution destinations (S409) and terminates the process.

If it is determined in step S403 that the selected agenda item has not been finalized (discarded) (NO in S403), or if it is determined in step S407 that the input distribution destination has not been finalized (discarded) (NO in S407), the CPU 11 terminates processing.

[Second embodiment]

Figure 19 is a diagram illustrating the operation of the memo creation device 1 in the second embodiment of the present invention. Figure 20 is a diagram illustrating the state in which a check mark has been added to the checkbox 290 on the meeting setting screen 32 of the memo creation app in the second embodiment of the present invention.

Referring to Figures 19 and 20, in this embodiment, the memo creation device 1 accepts input to the memo creation app from at least one terminal 3 via a network line 7 during a web conference. The memo creation device 1 is a terminal used by the user UR, and the terminal 3 is a terminal used by a conference participant PT other than the user UR. The terminal 3 includes a CPU, ROM, RAM, a storage device, a network interface, an operation unit, a display unit, a microphone, etc. In addition to the memo creation device 1, the terminal 3 may also store the memo creation program 50. In this case, the terminal 3 may instead execute some of the processing executed by the memo creation device 1 (for example, processing performed by the conversion unit 502 or the identification unit 503). The memo creation program 50 itself may have the ability to send and receive images and audio necessary for a web conference.

Referring to FIG. 20, when accepting input from terminal 3 while creating minutes, user UR checks check box 290 when making settings on the conference setting screen 32. When check box 290 is checked, memo creation device 1 generates connection information 291 (such as an IP address or URL) and displays it on the conference setting screen 32. Connection information 291 is information necessary to connect to memo creation device 1 via network line 7.

The user UR notifies the participant PT of the connection information 291 by, for example, sending an email from the memorandum creation device 1 to the terminal 3 (process PR1).

The participant PT who has acquired the connection information 291 connects the terminal 3 to the memo creation device 1 based on the connection information 291 at the start of the conference (process PR2).

When the memorandum creation device 1 displays the minutes creation screen 33 on the display unit 17, it transmits data of the displayed minutes creation screen 33 to the terminal 3 (process PR3). Based on the received data, the terminal 3 displays the minutes creation screen 33 on the display unit of the terminal 3.

Terminal 3 transmits the audio of participant PT's speech during the conference, acquired through the microphone of terminal 3, to memo creation device 1 (process PR4). Memo creation device 1 processes the audio acquired from terminal 3 in the same way as it processes the audio acquired through microphone 18 of memo creation device 1 (converting it to text, identifying the speaker, etc.).

When the memorandum creation device 1 displays the minutes approval screen 34 on the display unit 17, it transmits the data of the displayed minutes approval screen 34 to the terminal 3 (process PR5). Based on the received data, the terminal 3 displays the minutes approval screen 34 on the display unit of the terminal 3.

The terminal 3 transmits the input approving the minutes received through the operation unit of the terminal 3 to the memo creation device 1 (process PR6). When the memo creation device 1 receives the input approving the minutes from the terminal 3, it reflects it on the minutes approval screen 34 displayed on the display unit 17. Furthermore, when the memo creation device 1 receives the input approving the minutes through the operation unit 16, it reflects it on the minutes approval screen 34 displayed on the display unit 17.

The memo creation device 1 will disconnect from the terminal 3 when the minutes are finalized and the conference mode is terminated.

Note that the configuration and operation of the memorandum creation program 50 and memorandum creation device 1 other than those described above are the same as in the first embodiment, so their description will not be repeated.

[Effects of the embodiment]

According to the above-described embodiment, speakers are identified based on the audio of what is said during a conversation, and the audio of what is said during the conversation is converted into text and added to a memo, making it easy to create a memo of the conversation. In addition, by making a specified input during the conversation, symbols indicating topic breaks are added to the memo, making it possible to clarify the correspondence between the text in the memo and each topic. As a result, convenience can be improved.

[others]

The various screens of the memo creation app in the above-described embodiment are examples, and other display formats may also be used.

The above-mentioned embodiments can be combined as appropriate.

The above-described embodiments and variations should be considered in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the claims, not the above description, and is intended to include all modifications that are equivalent in meaning to and within the scope of the claims.

1 Memorandum creation device 3 Terminal 7 Network line 11 CPU (Central Processing Unit)
12 ROM (Read Only Memory)
13 RAM (Random Access Memory)
14 Storage device 15 Network interface 16 Operation unit 17 Display unit 18 Microphone 20, 231, 234, 271, 275 Messages 21, 23, 25, 27, 213, 214, 222, 222a, 232, 233, 237, 238, 253, 254, 255, 256, 273, 274, 279, 280 Buttons 31 Home screen 32 Conference setting screen 33 Minutes creation screen 34 Minutes approval screen 35 Voice registration screen 36 Speaker name registration screen 37 Conference menu creation screen 38 Minutes selection screen 39 Distribution destination input screen 50 Memorandum creation program 210, 212 List box 211, 216, 216a, 216b, 290 Check box 215 Title 217, 220, 235, 251, 252, 276, 278 Text boxes 218, 221, 277 Scroll bar 219 Symbol 236 Combo box 272 Radio button 291 Connection information 501 Speech acquisition unit 502 Conversion unit 503 Identification unit 504 Character string addition unit 505 Operation display unit 506 Topic addition unit 507 Feature registration unit 508 Approval reception unit 509 Communication unit 510 Translation unit PR1, PR2, PR3, PR4, PR5, PR6 Processing

Claims (8)

A memo creation program for creating a memo of a conversation including multiple topics,
an acquisition step of acquiring audio of utterances during the conversation;
a conversion step of converting the speech acquired in the acquisition step into a character string;
an identifying step of identifying a speaker of the voice acquired in the acquiring step based on characteristics of the voice acquired in the acquiring step;
a character string adding step of associating the character string converted in the converting step with the speaker identified in the identifying step and adding the character string to the memorandum stored in a storage device in chronological order;
a display step of displaying on a display unit a screen including a first area displaying the plurality of topics and an ongoing topic among the plurality of topics, and a second area displaying a memorandum stored in the storage device, the second area being an area different from the first area;
an input receiving step of receiving a predetermined input during the conversation;
a topic adding step of adding a symbol indicating a topic break to the memo stored in the storage device after the latest character string added in the character string adding step when the predetermined input is received in the input receiving step;
an end accepting step of accepting an input to end the conversation;
After the symbol is added to the memorandum in the topic adding step, if an input to end the conversation is not accepted in the end accepting step, audio of utterances made during the conversation is acquired in the acquiring step ;
A memorandum creation program that, in the input acceptance step, accepts as the specified input the selection of the next topic after the ongoing topic from among the multiple topics displayed in the first area, or the addition of a check mark to a check box corresponding to the next topic displayed in the first area . 2. The memorandum creation program according to claim 1 , wherein in said displaying step, the character string converted in said converting step is displayed on said display unit in a different manner for each speaker. 3. The memorandum creation program according to claim 1 , wherein the symbol includes a character string of one of the plurality of topics stored in the storage device. a pre-start acquisition step of acquiring voices of participants of the conversation before the conversation starts;
a feature registration step of registering the voice features acquired in the pre-start acquisition step in the storage device in association with the participants;
4. The memorandum creation program according to claim 1 , wherein in the identifying step, the speaker of the voice acquired in the acquiring step is identified based on the voice characteristics registered in the characteristics registering step. an approval receiving step of receiving an input approving the memorandum from the speaker identified in the identification step after the conversation ends;
The memorandum creation program according to any one of claims 1 to 4, further comprising causing a computer to execute a confirmation step of finalizing the memorandum when the approval inputs from all speakers identified in the identification step are received in the approval receiving step. The approval receiving step
a first approval receiving step of receiving, in a computer having the memorandum creation program installed therein, an input approving the memorandum from the speaker identified in the identifying step;
The memorandum creation program of claim 5, further comprising a second approval receiving step of receiving an input approving the memorandum from the speaker identified in the identification step via a network line from a computer other than the computer on which the memorandum creation program is installed. The obtaining step includes:
a first acquisition step of acquiring, in a computer having the memorandum creation program installed thereon, a voice of a utterance made during the conversation;
The memorandum creation program according to any one of claims 1 to 6 , further comprising a second acquisition step of acquiring audio of utterances made during the conversation via a network line from a computer other than the computer on which the memorandum creation program is installed. A memorandum creation program according to any one of claims 1 to 7, further comprising a screen transmission step of transmitting screen data displaying an ongoing topic among the plurality of topics to a computer other than the computer on which the memorandum creation program is installed via a network line.