JP7822008B2 - Program and information processing method - Google Patents

Description

Embodiments of the present invention relate to a program and an information processing method.

In recent years, medical treatment methods at hospitals have shifted from "face-to-face treatment," in which patients visit the hospital and meet with a doctor face-to-face, to "telemedicine," in which doctors and patients use communication devices to receive treatment without actually visiting a medical institution. In telemedicine, medical equipment is installed in the patient's home or a designated location, and vital data obtained by the medical equipment is transmitted via communication to a doctor in a remote location, who then makes a diagnosis based on the data.

One type of remote medical consultation is an information processing system known as a remote medical consultation booth, box, or kiosk, which is located in a designated location and equipped with equipment capable of communicating with medical equipment, allowing for real-time consultations between patients and doctors via video calls, etc. This allows for not only consultations via communication devices, but also consultations using medical equipment such as blood pressure monitors and stethoscopes, making it possible to examine symptoms in more detail.

In remote medical care, conversations conducted via communication devices are sometimes converted into text data, creating conversation logs that doctors, patients, and the patient's family can review after the conversation has ended.

However, regular conversation logs also record conversations unrelated to the main topic, such as communication problems, which can lead to misunderstandings when the person in question or their family members review the conversation log at a later date.

Therefore, there is a need for a call processing system that can create conversation logs that avoid misunderstandings by inserting annotations, etc.

Japanese Patent Application Laid-Open No. 2020-149072

The problem that embodiments of the present invention aim to solve is to provide a program and information processing method that can create a conversation log with inserted annotations.

In one embodiment, the program can cause a computer to perform the following functions: display a conversation log based on calls between multiple terminals; acquire information about annotations to be set for utterances included in the conversation log based on user operations; and display an updated conversation log based on the information about the annotations.

FIG. 1 is an external view illustrating a telemedicine booth included in an information processing system according to an embodiment. FIG. 2 is a block diagram illustrating an information processing system according to the embodiment. FIG. 3 is a diagram illustrating an example of a data structure of conversation log information according to the embodiment. FIG. 4 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. FIG. 5 is a schematic diagram illustrating an example of an editing instruction image according to the embodiment. FIG. 6 is a schematic diagram illustrating an example of a notification image according to the embodiment. FIG. 7 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. FIG. 8 is a schematic diagram illustrating an example of an edited input image according to the embodiment. FIG. 9 is a schematic diagram illustrating an example of a notification image according to the embodiment. FIG. 10 is a schematic diagram illustrating an example of a notification image according to the embodiment. FIG. 11 is a schematic diagram illustrating an example of a conversation log image according to the embodiment. FIG. 12 is a sequence diagram illustrating an example of a procedure for information processing by the information processing system according to the embodiment. FIG. 13 is a flowchart illustrating an example of a procedure for information processing by the server according to the embodiment. FIG. 14 is a flowchart illustrating an example of a procedure for information processing by the second communication device according to the embodiment. FIG. 15 is a flowchart illustrating another example of the procedure of information processing by the second communication device according to the embodiment. FIG. 16 is a flowchart illustrating an example of a procedure for information processing by the first communication device according to the embodiment.

(Embodiment)
(Configuration example)
Hereinafter, an embodiment will be described with reference to the drawings.

In the drawings, the same components are denoted by the same reference numerals whenever possible, and redundant explanations will be omitted.
FIG. 1 is an external view illustrating a telemedicine booth included in an information processing system 100 according to an embodiment.
A remote consultation booth is a facility surrounded by walls, preventing conversations from being overheard and ensuring privacy during consultations. A remote consultation booth is also called a consultation booth or simply a booth. Inside the booth, chairs, desks, and other equipment are installed, along with medical equipment for measurements and a communication device for communicating with a doctor outside the booth. For example, multiple consultation booths are installed in each area, such as a city, ward, town, or village. A patient makes a reservation for a remote consultation at a consultation booth in advance and enters the reserved booth at the scheduled time. The patient communicates with the doctor via a video call or other means via the communication device in the consultation booth, and uses medical equipment based on the doctor's instructions to measure biometric data and other data. Measurement data from the medical equipment can be viewed in real time on the doctor's equipment or other devices used by the doctor. The doctor can then perform a consultation based on the measurement data. Each consultation booth may be installed for each hospital, or may be used by multiple doctors from multiple hospitals.

FIG. 2 is a block diagram illustrating an information processing system 100 according to the embodiment.
The information processing system 100 includes a server 1, a second communication device 4, and at least one remote medical examination booth. The remote medical examination booth includes a remote medical examination support device 2 and a first communication device 3. The server 1, the remote medical examination support device 2, the first communication device 3, and the second communication device 4 are communicably connected to each other via a network. For example, the network may be one or more of various networks such as the Internet, a mobile communication network, and a local area network (LAN). The one or more networks may include a wireless network or a wired network. The remote medical examination support device 2 and the first communication device 3 are communicably connected to each other via a network. The network may be a LAN or the like. The LAN may be a wireless LAN or a wired LAN. The remote medical examination support device 2 is communicably connected to at least one medical device. The remote medical examination support device 2 and at least one medical device 251-1 to 251-m are directly communicably connected via a wired or wireless connection. The remote medical care support device 2 and at least one of the medical devices 251-1 to 251-m may be connected via, for example, a LAN, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc. Note that the information processing system 100 may also refer to a system including the server 1, the remote medical care support device 2, the first communication device 3, the second communication device 4, and at least two of the medical devices 251-1 to 251-m.

Server 1 is an electronic device that collects and processes data. The electronic device includes a computer. Server 1 is communicatively connected to remote medical consultation support device 2, first communication device 3, and second communication device 4 via a network. Server 1 receives various data from remote medical consultation support device 2, first communication device 3, and second communication device 4, and outputs various data to remote medical consultation support device 2, first communication device 3, and second communication device 4. Server 1 may also be a server used in a cloud service.

The server 1 may provide a call service for video communication or the like between the first communication device 3 and the second communication device 4. The call service may be a call service based on voice communication. The call service may not involve communication using video images. The server 1 may provide a remote medical consultation service between the remote medical consultation support device 2, the first communication device 3, and the second communication device 4. An example configuration of the server 1 will be described later.

The remote medical care support device 2 is an electronic device capable of communicating with other electronic devices. The remote medical care support device 2 is a device installed in a medical examination booth. For example, the remote medical care support device 2 is a PC (Personal Computer), a smartphone, or a tablet terminal. The term "participant" may also be interpreted as "user" or "person." An example configuration of the remote medical care support device 2 will be described later.

The first communication device 3 is an electronic device capable of communicating with other electronic devices. The first communication device 3 is a device installed in a medical examination booth. The first communication device 3 is, for example, a device used by a patient receiving remote medical treatment. For example, the first communication device 3 is a PC, smartphone, tablet terminal, or the like. The term "patient" may also be interpreted as "user" or "person." An example configuration of the first communication device 3 will be described later. The first communication device 3 is an example of a terminal.

The second communication device 4 is an electronic device capable of communicating with other electronic devices. The second communication device 4 is, for example, a device used by a doctor who performs remote medical treatment. For example, the second communication device 4 is a PC, smartphone, tablet terminal, etc. The term "doctor" may also be interpreted as a medical professional, user, or person. The second communication device 4 is, for example, installed in a location different from the examination booth. The location where the examination booth is installed is an example of a first base. The location where the second communication device 4 is installed is an example of a second base. The second base is, for example, a medical institution such as a hospital. An example configuration of the second communication device 4 will be described later. The second communication device 4 is an example of a terminal.

An example of the configuration of the server 1 will be described.
The server 1 is an electronic device that includes a processor 11, a main memory 12, an auxiliary storage device 13, and a communication interface 14. The components that make up the server 1 are connected to each other so that signals can be input and output. In Fig. 1, the interface is indicated as "I/F".

Processor 11 corresponds to the central part of server 1. Processor 11 is an element that constitutes the computer of server 1. For example, processor 11 is a CPU (Central Processing Unit), but is not limited to this. Processor 11 may be composed of various circuits. Processor 11 loads programs that are pre-stored in main memory 12 or auxiliary storage device 13 into main memory 12. The programs are programs that cause processor 11 of server 1 to realize or execute each unit described below. Processor 11 performs various operations by executing programs loaded into main memory 12.

Main memory 12 corresponds to the main storage portion of server 1. Main memory 12 is an element that constitutes the computer of server 1. Main memory 12 includes a non-volatile memory area and a volatile memory area. Main memory 12 stores an operating system or programs in the non-volatile memory area. Main memory 12 uses the volatile memory area as a work area where data is rewritten as appropriate by processor 11. For example, main memory 12 includes ROM (Read Only Memory) as a non-volatile memory area. For example, main memory 12 includes RAM (Random Access Memory) as a volatile memory area. Main memory 12 stores programs.

The auxiliary storage device 13 corresponds to the auxiliary storage portion of the server 1. The auxiliary storage device 13 is an element that constitutes the computer of the server 1. The auxiliary storage device 13 is an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), HDD (Hard Disc Drive), SSD (Solid State Drive), etc. The auxiliary storage device 13 stores the above-mentioned programs, data used by the processor 11 in performing various processes, and data generated by the processes in the processor 11. The auxiliary storage device 13 stores the above-mentioned programs. The auxiliary storage device 13 is an example of a storage unit.

The auxiliary memory device 13 stores conversation log information. The conversation log information is information that records a conversation between a user of a first communication device and a user of a second communication device. The conversation log information includes at least speaker information and conversation information. The speaker information is, for example, speaker identification information that can identify the speaker. The speakers include the user of the first communication device and the user of the second communication device. The speaker identification information includes a user ID, a speaker name, etc. The user ID is information that can individually identify each speaker. The speaker name may be the speaker's name or username, etc. The speaker identification information may be information that indicates the speaker's attributes, such as "doctor" or "patient." The conversation information is associated with the speaker identification information. The conversation information indicates the content of the utterance spoken by the speaker. The conversation information is information obtained by performing voice recognition processing based on audio information related to the utterance. Speech recognition includes, for example, converting speech information related to speech into text data and segmenting it using known techniques. The conversation information includes at least one spoken sentence. The spoken sentence is, for example, segmented based on divisions in the user's speech. The spoken sentence may include identification information that identifies each spoken sentence. The spoken sentence includes at least one word. A word is the smallest meaningful unit of language. A word includes parts of speech such as nouns, verbs, and exclamations.

The conversation log information will be described with reference to FIG.
FIG. 3 is a diagram illustrating an example of a data structure of conversation log information according to the embodiment.
FIG. 3 shows conversation log information when a conversation takes place between a doctor and a patient. The conversation log information has records linked to a conversation log ID, speaker identification information, date and time, annotation flag, conversation information, etc. The conversation log shows data in which spoken sentences are arranged in chronological order. The conversation log ID is information that can individually identify each spoken sentence. Spoken sentences are also called conversation sentences. The conversation log ID may be information that can individually identify each segment. The speaker identification information may include a user ID and a speaker name. In the example of FIG. 3, the speaker name is "doctor" or "patient." The speaker identification information is linked to each spoken sentence. The speaker identification information may be information based on information that can identify the communication device that output the audio information. The speaker identification information may be information based on information that can identify the user of the communication device that output the audio information. The date and time indicate speech time information. The speech time information may include time information indicating the time when the speech was made, or elapsed time information indicating the time elapsed since the conversation started. In the example of Fig. 3, the date and time indicates time information indicating the time when the speech was made. The time when the speech was made may correspond to the time when the server 1 obtained the call information from the first communication device 3 or the second communication device 4. The conversation log information may also be simply referred to as log information.

The annotation flag is information indicating whether an annotation has been set based on a user operation. An annotation indicates additional information to be recorded for a selected utterance or range of utterances when the user selects an utterance or range of utterances from the conversation log. A range of utterances may include multiple utterances. A range of utterances may be part of a single utterance. Additional information includes text information entered based on a user operation. Additional information may include information such as images, audio, and video. Annotations include information about the utterance. Information about the utterance includes information such as descriptions, references, and explanations to be added to the utterance. Information about the utterance includes information explaining the content, etc., of the utterance. Information about the utterance includes information explaining the context, etc., of the utterance. For example, when a conversation takes place about a topic different from the topic of the conversation, annotations are information that explains the different topic. For example, annotations can be set for an utterance or range of utterances selected by the user.

Let us consider a case where the topic of a conversation between a user of a first communication device 3 and a user of a second communication device 4 is "about treatment." Topics different from the conversation topic "about treatment" include, for example, "about a different conversation from the topic," "about communication problems," "about different symptoms," etc. For example, assume that a communication problem occurs during a conversation between a user of a first communication device 3 and a user of a second communication device 4. In this case, the conversation about the communication problem is a conversation on a different topic. A different topic could be "about a different conversation from the topic," or "about communication problems." It is conceivable that the user will set an annotation to indicate that the conversation about the communication problem is a conversation on a different topic from the topic of the conversation.

The annotation may include the reason for the conversation. Reasons for the conversation may include "due to communication problems," "due to a conversation about a different symptom," etc. The reason for the conversation may be the same as or different from the topic of the conversation.

An annotation flag can be set for each spoken sentence or for each range of spoken sentences. The annotation flag includes "YES", which indicates that an annotation has been set, and "NO", which indicates that an annotation has not been set. The annotation flag may be one-bit information. The annotation flag can be set by the user selecting an utterance sentence or a range of spoken sentences. The user selects an utterance sentence or a range of spoken sentences and sets an annotation. Setting an annotation flag may be interpreted as adding an annotation.

Conversation information is, for example, text information indicating the content of the utterance. Conversation information is linked to each spoken sentence. Conversation information may include a main text and annotation information. The main text indicates information obtained by performing speech recognition processing based on audio information related to the utterance. Annotation information is information related to annotations input based on user operations. Information related to annotations includes at least one of information indicating the content of the annotation and information indicating the reason for the annotation. User operations include operations to edit utterances on a conversation log screen on which the conversation log is displayed. Editing operations include operations to select an edit button or the like by touch input or the like on the screen on which the conversation log is displayed. The edit button is a button operated by the user. The edit button is a software key. The edit button is an operator for instructing editing. The edit button may include an annotation setting button and an annotation deletion button. Editing operations include operations to set an annotation and an annotation deletion operation. Operations to set annotations include operations to select a spoken sentence or a range of a spoken sentence, operations to input text information, operations to rewrite part of an annotation, operations to copy part or all of an annotation, operations to move part or all of an annotation, etc. Operations to delete annotations include operations to select a spoken sentence or a range of a spoken sentence, operations to delete text information, operations to delete part or all of an annotation, etc.

In the example of Figure 3, conversation log ID "1001" is a sentence spoken by a doctor with user ID "A." The time when the sentence with conversation log ID "1001" was spoken was "17:00:35." The annotation flag for conversation log ID "1001" indicates that an annotation has been set for the sentence with conversation log ID "1001." The conversation content of the sentence with conversation log ID "1001" is "Are you OK?" When an annotation is set by the user for the sentence with conversation log ID "1001," information about the annotation is saved in the annotation information field. For example, the example of Figure 3 indicates that the user set annotation information for the sentences with conversation log ID "1001" and conversation log ID "1002" as "Content: conversation other than the topic, Reason: communication trouble." Information about the annotation is not limited to the above example and can be entered by the user as desired.

The user who sets the annotation may include the user of the first communication device 3 and the user of the second communication device 4. The user who sets the annotation may be a pre-set user. For example, the user who sets the annotation may be only the user of the second communication device 4. The user who sets the annotation may be a doctor in remote medical care. The user who sets the annotation may be set as appropriate by the speaker, administrator, setter, etc. In the following example, the user who sets the annotation will be described as being the user (doctor) of the second communication device 4.

Conversation log information is information that links at least the conversation log ID, speaker identification information, date and time, annotation flag, and conversation information.

The auxiliary storage device 13 may store information about examination booths. The information about examination booths includes information that can identify the examination booths. The information about examination booths may also include information about medical equipment installed in the examination booths.

The auxiliary storage device 13 may store user information. User information is information about users (patients) who use the examination booth. User information includes user identification information. User identification information is unique identification information assigned to each user to identify each user. User information may include information such as the user's location information and medical history information. User information may also include measurement data from medical equipment.

The communication interface 14 includes various interfaces that connect the server 1 to other electronic devices via a network in accordance with a specific communication protocol.

Note that the hardware configuration of Server 1 is not limited to the configuration described above. Server 1 allows for the omission or modification of the above components and the addition of new components as appropriate.

Each unit implemented in the processor 11 will now be described.
The processor 11 realizes a call information acquisition unit 110, a log creation unit 111, a storage control unit 112, a log update unit 113, and an output unit 114. Each unit realized in the processor 11 can also be referred to as a function. Each unit realized in the processor 11 can also be referred to as being realized in a control unit including the processor 11 and the main memory 12. The processor 11 is an example of a processing circuit.

The call information acquisition unit 110 acquires call information from the first communication device 3 and the second communication device 4 via the communication interface 14. The call information includes voice information. The voice information includes voice information related to the user's speech. The voice information may include information indicating voice characteristics. Voice characteristics may include voice pitch, timbre, etc. The call information includes speaker information and speaking time information. The call information acquisition unit 110 may also acquire user biometric data from the first communication device 3 and the second communication device 4. The biometric data may be measurement data obtained by medical devices 251-1 to 251-m. The biometric data may also be data measured by sensors connected to the first communication device 3 and the second communication device 4.

The log creation unit 111 creates a conversation log based on the call information. The log creation unit 111 creates the conversation log in chronological order based on audio information related to the user's speech, speaker information, and speech time information. Information related to the conversation log created by the log creation unit 111 is also referred to as conversation log information. The conversation log information is also simply referred to as the conversation log. The log creation unit 111 may stop creating the conversation log based on an annotation setting instruction to set an annotation. The log creation unit 111 may resume creating the conversation log based on an annotation confirmation instruction to confirm the annotation.

The memory control unit 112 stores the information acquired by the call information acquisition unit 110 in the auxiliary memory device 13. The memory control unit 112 stores the conversation log information. The memory control unit 112 updates the conversation log information. The memory control unit 112 sets or updates annotation flags based on annotation flags set by the log update unit 113 (described below). The memory control unit 112 stores the conversation log information updated by the log update unit 113 in the auxiliary memory device 13. The memory control unit 112 updates the conversation log information.

The log update unit 113 acquires an annotation setting instruction from the second communication device 4 via the communication interface 14. The annotation setting instruction includes an instruction to set an annotation for an utterance sentence or a range of an utterance sentence included in the conversation log, and an instruction to delete an annotation set for an utterance sentence. Setting an annotation is also referred to as inserting an annotation. The log update unit 113 acquires information about the annotation from the second communication device 4 via the communication interface 14. The information about the annotation includes information indicating the range in which the annotation is to be set. The range in which the annotation is to be set includes an utterance sentence or a range of an utterance sentence. The range in which the annotation is to be set is also referred to as the annotation target. The information about the annotation includes information indicating the content of the annotation. The content of the annotation may include a subject of the utterance sentence that is the annotation target. The information about the annotation may include the reason for the annotation. The information about the annotation may include information about the time the annotation was set. The information about the annotation may include information about the user who set the annotation. The log update unit 113 sets or updates an annotation flag based on acquiring the annotation setting instruction. The log update unit 113 may receive an edit instruction to edit the annotation from the second communication device 4 via the communication interface 14. The log update unit 113 updates the conversation log information based on information about the annotation. The log update unit 113 may also update the conversation log information based on an annotation confirmation instruction.

The output unit 114 outputs conversation log information to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 may output conversation log information each time conversation log information is created. The output unit 114 may output conversation log information based on a conversation log display request from the first communication device 3 and the second communication device 4. The output unit 114 outputs updated conversation log information to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 may output updated conversation log information each time the conversation log information is updated. The output unit 114 may output updated conversation log information based on a display request from the first communication device 3 and the second communication device 4 for the updated conversation log.

The output unit 114 outputs information indicating various notifications to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 outputs various requests to the first communication device 3 and the second communication device 4 via the communication interface 14. The output unit 114 outputs information regarding annotations to the first communication device 3 via the communication interface 14.

An example of the configuration of the remote medical care support device 2 will be described.
The remote medical consultation support device 2 is an electronic device including a processor 21, a main memory 22, an auxiliary storage device 23, a communication interface 24, an input/output interface 25, a display device 26, a speaker 27, and an input device 28. The components constituting the remote medical consultation support device 2 are connected to each other so that signals can be input and output.

The processor 21 corresponds to the central part of the remote medical consultation support device 2. The processor 21 is an element that constitutes the computer of the remote medical consultation support device 2. For example, the processor 21 has the same hardware configuration as the processor 11 described above. The processor 21 performs various operations by executing programs deployed in the main memory 22. The processor 21 is an example of a processing circuit.

The main memory 22 corresponds to the main storage portion of the remote medical consultation support device 2. The main memory 22 is an element that constitutes the computer of the remote medical consultation support device 2. The main memory 22 has the same hardware configuration as the main memory 12 described above. The main memory 22 stores programs.

The auxiliary storage device 23 corresponds to the auxiliary storage portion of the remote medical consultation support device 2. The auxiliary storage device 23 is an element that constitutes the computer of the remote medical consultation support device 2. The auxiliary storage device 23 has the same hardware configuration as the auxiliary storage device 13 described above. The auxiliary storage device 23 stores the above-mentioned programs.

The communication interface 24 includes various interfaces that connect the remote medical assistance device 2 to other devices via a network in accordance with a predetermined communication protocol.

The input/output interface 25 is an interface for connecting the remote medical consultation support device 2 to external devices. The external devices include at least one medical device 251-1 to 251-m (m is an integer greater than or equal to 1). The medical devices 251-1 to 251-m include, for example, an electrocardiograph, a blood pressure monitor, a digital stethoscope, a pulse oximeter, a slit lamp, and an external otoscope. The medical devices 251-1 to 251-m have communication capabilities and output measurement data to the remote medical consultation support device 2.

The display device 26 is a device capable of displaying various screens under the control of the processor 21. For example, the display device 26 is a liquid crystal display or an EL display.

The speaker 27 is a device that can output sound under the control of the processor 21. The speaker 27 is an example of an output device.

The input device 28 is a device that can input data or instructions to the remote medical consultation support device 2. For example, the input device 28 includes a built-in microphone that can input voice and a built-in camera that can acquire image data of the imaging range. The input device 28 may also include a keyboard, touch panel, etc.

Note that the hardware configuration of the remote medical consultation support device 2 is not limited to the configuration described above. The remote medical consultation support device 2 allows for the omission or modification of the above-described components and the addition of new components as appropriate.

An example of the configuration of the first communication device 3 will be described.
The first communication device 3 is an electronic device including a processor 31, a main memory 32, an auxiliary storage device 33, a communication interface 34, an input/output interface 35, and an input device 38. The components constituting the first communication device 3 are connected to each other so as to be able to input and output signals to and from each other.

Processor 31 corresponds to the central part of first communication device 3. Processor 31 is an element that constitutes the computer of first communication device 3. Processor 31 has the same hardware configuration as processor 11 described above. Processor 31 performs various operations by executing programs that are pre-stored in main memory 32 or auxiliary storage device 33. Processor 31 is an example of a processing circuit.

The main memory 32 corresponds to the main storage portion of the first communication device 3. The main memory 32 is an element that constitutes the computer of the first communication device 3. The main memory 32 has the same hardware configuration as the main memory 12 described above. The main memory 32 stores programs.

The auxiliary storage device 33 corresponds to the auxiliary storage portion of the first communication device 3. The auxiliary storage device 33 is an element that constitutes the computer of the first communication device 3. The auxiliary storage device 33 has the same hardware configuration as the auxiliary storage device 13 described above. The auxiliary storage device 33 stores the above-mentioned programs.

The communication interface 34 includes various interfaces that connect the first communication device 3 to other devices via a network in accordance with a predetermined communication protocol.

The input/output interface 35 is an interface for connecting the first communication device 3 with external devices. The external devices include a display device 351-1, a camera 351-2, a microphone 351-3, and a speaker 351-4. The display device 351-1 is a device capable of displaying various screens under the control of the processor 31. For example, the display device 351-1 is an LCD display or an EL display. The camera 351-2 is a device capable of acquiring image data of the image capture range under the control of the processor 31. The microphone 351-3 is a device capable of inputting audio under the control of the processor 31. The speaker 351-4 is a device capable of outputting audio under the control of the processor 31.

The input device 38 is a device that can input data or instructions to the first communication device 3. For example, the input device 38 includes a keyboard, a touch panel, etc.

Note that the hardware configuration of the first communication device 3 is not limited to the configuration described above. The first communication device 3 may omit or modify the above-described components and add new components as appropriate.

An example of the configuration of the second communication device 4 will be described.
The second communication device 4 is an electronic device including a processor 41, a main memory 42, an auxiliary storage device 43, a communication interface 44, an input/output interface 45, and an input device 48. The components constituting the second communication device 4 are connected to each other so as to be able to input and output signals to and from each other.

Processor 41 corresponds to the central part of second communication device 4. Processor 41 is an element that constitutes the computer of second communication device 4. Processor 41 has the same hardware configuration as processor 11 described above. Processor 41 performs various operations by executing programs that are pre-stored in main memory 42 or auxiliary storage device 43. Processor 41 is an example of a processing circuit.

The main memory 42 corresponds to the main storage portion of the second communication device 4. The main memory 42 is an element that constitutes the computer of the second communication device 4. The main memory 42 has the same hardware configuration as the main memory 12 described above. The main memory 42 stores programs.

The auxiliary storage device 43 corresponds to the auxiliary storage portion of the second communication device 4. The auxiliary storage device 43 is an element that constitutes the computer of the second communication device 4. The auxiliary storage device 43 has the same hardware configuration as the auxiliary storage device 13 described above. The auxiliary storage device 43 stores the above-mentioned programs.

The communication interface 44 includes various interfaces that connect the second communication device 4 to other devices via a network in accordance with a predetermined communication protocol.

The input/output interface 45 is an interface for connecting the second communication device 4 with external devices. The external devices include a display device 451-1, a camera 451-2, a microphone 451-3, and a speaker 451-4. The display device 451-1 is a device capable of displaying various screens under the control of the processor 41. For example, the display device 451-1 is an LCD display or an EL display. The camera 451-2 is a device capable of acquiring image capture data of the capture range under the control of the processor 41. The microphone 451-3 is a device capable of inputting audio under the control of the processor 41. The speaker 451-4 is a device capable of outputting audio under the control of the processor 41.

The input device 48 is a device that can input data or instructions to the second communication device 4. For example, the input device 48 includes a keyboard, a touch panel, etc.

Note that the hardware configuration of the second communication device 4 is not limited to the configuration described above. The second communication device 4 may omit or modify the above-described components and add new components as appropriate.

Each unit implemented in the above-mentioned processor 41 will now be described.
The processor 41 realizes a log processing unit 410 and a display processing unit 411. Each unit realized by the processor 41 can also be referred to as each function. Each unit realized by the processor 41 can also be referred to as being realized by a control unit including the processor 41 and the main memory 42.

The log processing unit 410 outputs call information to the server 1 via the communication interface 44. The log processing unit 410 acquires conversation log information from the server 1 via the communication interface 44. The log processing unit 410 acquires updated conversation log information from the server 1 via the communication interface 44.

The log processing unit 410 performs processing related to editing the conversation log. The log processing unit 410 acquires an annotation setting instruction based on a user operation. The user operation includes a touch input operation such as an edit button. The log processing unit 410 outputs an annotation setting instruction to the server 1 via the communication interface 44. The log processing unit 410 acquires information related to the annotation based on the user operation. The information related to the annotation includes information indicating the range in which the annotation is to be set. The range in which the annotation is to be set can be specified by the user selecting an utterance sentence using a touch operation or the like, or selecting a range of an utterance sentence using a slide operation or swipe operation or the like on the conversation log screen on which the conversation log is displayed. The information related to the annotation can be input arbitrarily by the user operation. The user operation includes inputting information related to the annotation by text input or the like. The log processing unit 410 outputs the information related to the annotation to the server 1 via the communication interface 44. The log processing unit 410 outputs an annotation confirmation instruction to confirm the annotation via the communication interface 44.

The log processing unit 410 outputs various requests to the server 1 via the communication interface 44. The requests include a stop request to request a restart. The requests include a resume request to resume recording the conversation log.

The log processing unit 410 obtains information regarding various notifications from the server 1 via the communication interface 44. Notifications include notifications regarding the recording of conversation logs. The log processing unit 410 obtains various requests from the server 1 via the communication interface 44. Requests include confirmation requests to confirm that annotations are reflected.

(Display example)
Display examples of the conversation log and the annotation log will be described.
FIG. 4 is a schematic diagram illustrating an example of a conversation log image according to the embodiment.

Figure 4 shows the conversation log image Ima. The conversation log image Ima can be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4. The conversation log image Ima is a conversation log image displayed on a communication device used by a user who can set annotations in the conversation log. A user who can set annotations is a user who has annotation setting authority. Annotation setting authority can be set in advance by, for example, an administrator, a host of a call between the communication devices, a setter, etc. Annotation setting authority can be updated as appropriate by, for example, an administrator, a host of a call between the communication devices, a setter, etc. Annotation setting authority can be granted to at least one of the speakers, for example. Annotation setting authority may also be granted to all speakers. In the following example, annotation setting authority is granted to the user of the second communication device 4.

The conversation log image Ima is displayed in real time each time a conversation log is created. Note that the conversation log image Ima may also be displayed based on a conversation log display request from the user of the second communication device 4.

The conversation log image Ima includes at least speaker identification marks Ma, Mb, spoken sentence La, and an edit button Ba. The speaker identification marks Ma, Mb indicate marks that can identify the speaker. The speaker identification marks Ma, Mb may include the speaker's name. In the example of Figure 4, the speaker identification mark Ma indicates "doctor" or "patient." The speaker identification mark Ma may have a different shape, color, or size for each speaker. The spoken sentence La is text information that indicates the content of the utterance. The spoken sentence La may include a speech bubble-shaped frame, for example, as shown in Figure 4. The spoken sentence La is arranged in chronological order based on the speech time information. The edit button Ba is a button that is selected by the user by touch input, etc. The edit button Ba is, for example, an operator that can input an annotation setting instruction. In response to the user selecting the edit button Ba, the display processing unit 411 displays the edit instruction image shown in FIG. 5 on the display device 451-1. In response to the user selecting the edit button Ba, the log processing unit 410 may output an annotation setting instruction to the server 1. In response to the user selecting the edit button Ba, the processor 41 may transition to edit mode.

Note that the conversation log image Ima, excluding the edit button Ba, may be displayed on the first communication device 3. The conversation log image may be displayed on the first communication device 3 in real time each time a conversation log is created. The conversation log image may be displayed based on a conversation log display request from the user of the first communication device 3.

Figure 5 is a schematic diagram illustrating an example of an editing instruction image according to an embodiment.

Figure 5 shows the editing instruction image Imb. The editing instruction image Imb is displayed on the display device 451-1 of the second communication device 4. The editing instruction image Imb is displayed on the display device 451-1 in response to the user selecting the edit button Ba on the conversation log image Ima. The editing instruction image Imb may be superimposed on or combined with the conversation log image Ima. The editing instruction image Imb may be displayed on the display device 451-1 as a pop-up display. In the following description, "in response" may be read as "based on."

The editing instruction image Imb includes an annotation setting button Bb and an annotation deletion button Bc. The annotation setting button Bb is an operator that can be used to input an instruction to set an annotation. The annotation deletion button Bc is an operator that can be used to input an instruction to delete an annotation. To set an annotation, the user selects the annotation setting button Bb by touch input, etc. To delete an annotation, the user selects the annotation deletion button Bc by touch input, etc.

FIG. 6 is a schematic diagram illustrating an example of a notification image according to the embodiment.
6 shows a notification image Imc. The notification image Imc is displayed on the display device 451-1 of the second communication device 4. For example, the notification image Imc is displayed on the display device 451-1 in response to the user selecting the annotation setting button Bb or the annotation deletion button Bc in the edit instruction image Imb. The notification image Imc may also be displayed on the display device 451-1 in response to the user selecting the edit button Ba in the conversation log image Ima. The notification image Imc may be superimposed on or composited with the conversation log image Ima. The notification image Imc may also be displayed on the display device 451-1 as a pop-up display.

The notification image Imc includes text information indicating that recording of the conversation log will be stopped while the annotation process is being performed. The text information includes, for example, information such as "The conversation during this time will not be recorded." The annotation process includes, for example, the processing from when the user inputs an annotation setting instruction to when information related to the annotation is recorded in or updated in the conversation log information. The annotation process includes processing in edit mode. The notification image Imc can be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4.

This allows the user of the first communication device 3 and the user of the second communication device 4 to recognize that conversations during annotation processing will not be recorded.

FIG. 7 is a schematic diagram illustrating an example of a conversation log image according to the embodiment.
7 shows a conversation log image Imd. The conversation log image Imd is displayed on the display device 451-1 of the second communication device 4. The conversation log image Imd can be displayed on a communication device used by a user who has annotation setting authority.

Like the conversation log image Ima, the conversation log image Imd includes at least a speaker identification mark and a spoken sentence. The conversation log image Imd also includes an annotation target area Ara. The annotation target area Ara indicates the spoken sentence or range of the spoken sentence to which the user selects an annotation. In this example, the user selects multiple spoken sentences as the annotation target through a slide operation. The range of the annotation target is not limited to selection through a slide operation, and may be selected for each spoken sentence through a touch operation or the like. The range of the annotation target may be displayed in a manner that makes it recognizable from other spoken sentences. The display processing unit 411 may display the range of the annotation target in a different display manner from other spoken sentences. Display manners include character decoration such as character color, size, font, brightness or luminance, background color, background brightness or luminance, background pattern, frame color, or blinking display. In this example, the display processing unit 411 displays the range of the annotation target by enclosing it in a dashed line. The range of the annotation target corresponds to information indicating the range to be annotated.

FIG. 8 is a schematic diagram illustrating an example of an edited input image according to the embodiment.
8 shows an edited input image Ime. The edited input image Ime is displayed on the display device 451-1 of the second communication device 4. The edited input image Ime can be displayed on a communication device used by a user authorized to set annotations. The edited input image Ime can be displayed on the display device 451-1 in response to the user selecting a range to be annotated.

The edited input image Ime includes an area where information about the annotation can be entered. The area where information about the annotation can be entered can include an annotation content input area Arb where the annotation content can be entered, and an annotation purpose input area Arc where the annotation purpose can be entered. The annotation content input area Arb is an area where the annotation content can be entered as desired. The annotation purpose input area Arc is an area where the annotation purpose can be entered as desired. In this example, the edited input image Ime shows that the user has entered "Conversation other than the main topic" as the annotation content in the annotation content input area Arb, and "Communication trouble" as the annotation purpose in the annotation purpose input area Arc.

The edit input image Ime includes an OK button Bd and a cancel button Be. The OK button Bd is an operator that allows the user to input information indicating that the user has confirmed the input content. The cancel button Be is an operator that allows the user to input information indicating that the user wishes to cancel the input content. In response to the user selecting the OK button Bd by touch input or the like, the log processing unit 410 outputs information regarding the annotation to the server 1. In response to the user selecting the cancel button Be by touch input or the like, the display processing unit 411 may transition the displayed image from the edit input image Ime to a conversation log image Imd. Note that in response to the user selecting the cancel button Be by touch input or the like, the display processing unit 411 may transition the displayed image to a conversation log image Ima or an edit instruction image Imb.

FIG. 9 is a schematic diagram illustrating an example of a notification image according to the embodiment.
FIG. 9 shows a notification image Imf. The notification image Imf is displayed on the display device 351-1 of the first communication device 3. The notification image Imf may be displayed on a communication device used by a user other than the user with annotation setting authority. A user other than the user with annotation setting authority is, for example, a user who can accept annotations. A user who can accept annotations indicates a user with annotation acceptance authority. The annotation acceptance authority may be updated as appropriate by an administrator, a host of a call between communication devices, a setter, etc. The annotation acceptance authority may be granted to at least one of the speakers, for example. The annotation acceptance authority may also be granted to a user other than the speaker. In the following example, it is assumed that the annotation acceptance authority is granted to the user of the first communication device 3.

The notification image Imf includes information notifying that the server 1 has acquired information related to the annotation. The notification image Imf may be superimposed or combined with the conversation log image. The notification image Imf may be displayed on the display device 351-1 as a pop-up display.

The notification image Imf includes a notification area Ard, an annotation area Are, an OK button Bf, and a reject button Bg. The notification area Ard includes information indicating that an annotation has been set. The notification area Ard may include information related to the annotation. The notification area Ard includes text information such as "An annotation has been added to the following conversation record. Content: Off-topic conversation, Reason: Communication trouble." The annotation area Are includes information indicating the range of the annotation target. The annotation area Are indicates the range of the annotation target in the conversation log in a recognizable manner. In this example, the spoken sentences included in the range of the annotation target are indicated by dashed speech bubbles.

The OK button Bf is an operator that allows the user to input an acceptance instruction indicating that the user accepts the annotation. Accepting the annotation includes at least one of accepting the annotation to be set and accepting the content of the annotation. The reject button Bg is an operator that allows the user to input a rejection instruction indicating that the user rejects the annotation. Rejecting the annotation includes at least one of refusing to set the annotation and rejecting the content of the annotation. The acceptance instruction and the rejection instruction are examples of information related to acceptance. In response to the user selecting the OK button Bf by touch input or the like, the processor 31 outputs information indicating that the annotation is accepted to the server 1. Information related to the annotation is output to the server 1. In response to the user selecting the reject button Bg by touch input or the like, the processor 31 outputs information indicating that the annotation is rejected to the server 1.

In this example, the information processing system 100 can confirm whether users other than the user who set the annotation have permission to set the annotation and the content of the annotation. Users other than the user who set the annotation can express their permission to set the annotation and the content of the annotation. This allows the information processing system 100 to reach an agreement regarding the annotation between the person who set the annotation and other users. In this way, the information processing system 100 can create a conversation log with annotations inserted based on the intentions of the users.

FIG. 10 is a schematic diagram illustrating an example of a notification image according to the embodiment.
10 shows a notification image Img. The notification image Img is displayed on the display device 451-1 of the second communication device 4. The notification image Img can be displayed on a communication device used by a user who has the authority to set an annotation. The notification image Img includes information notifying that the server 1 has acquired information indicating that the annotation is accepted. The notification image Img may be superimposed or combined with the conversation log image. The notification image Img may be displayed on the display device 451-1 as a pop-up display.

The notification image Img includes a notification area Arf and an OK button Bh. The notification area Arf includes information indicating that the annotation has been accepted. The notification area Arf includes text information such as, "The annotation has been accepted. The annotation will be reflected." The OK button Bh is an operator that allows the user to input a confirmation instruction indicating that the annotation will be reflected. In response to the user selecting the OK button Bh by touch input or the like, the log processing unit 410 outputs information indicating that the annotation will be reflected to the server 1. The notification image Img is an example of a confirmation request.

In this example, a user can allow annotations to be set by users other than the user who set the annotation, and can confirm consent to the content of the annotation. Based on consent from users other than the user who set the annotation, the user can input an instruction to confirm the annotation. This allows the information processing system 100 to create a conversation log with annotations inserted based on the intentions of the users.

FIG. 11 is a schematic diagram illustrating an example of a conversation log image according to the embodiment.
FIG. 11 shows a conversation log image Imh. The conversation log image Imh can be displayed on the display device 351-1 of the first communication device 3 and the display device 451-1 of the second communication device 4. The conversation log image Imh includes an annotation area Arg and an annotation target area Arh. The annotation area Arg can include information related to the annotation. For example, the annotation area Arg includes text information such as "Conversation other than the main topic (reason: communication trouble)." The text information is not limited to the display format shown in FIG. 11. The annotation target area Arh indicates the range of the annotation target in a recognizable manner. In this example, spoken sentences included in the range of the annotation target are indicated by dashed line speech bubbles.

The display processing unit 411 displays the annotation in a recognizable manner based on the annotation area Arg and the annotation target area Arh. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

In this example, the information processing system 100 can display the presence or absence of annotations and the content of the annotations on the conversation log display screen in a manner that makes it easy for the user to recognize the presence or absence of annotations and the content of the annotations.

(Example of operation)
The processing procedure performed by the information processing system 100 will be described.
In the following description focusing on the server 1, the server 1 may be read as the processor 11. In the following description focusing on the first communication device 3, the first communication device 3 may be read as the processor 31. In the following description focusing on the second communication device 4, the second communication device 4 may be read as the processor 41.
The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

Figure 12 is a sequence diagram showing an example of the information processing procedure by the information processing system 100 according to the embodiment.

The following process is assumed to start when a call between the first communication device 3 and the second communication device 4 is initiated. In the following example, it is assumed that annotation setting authority has been granted for the user of the second communication device 4. It is assumed that annotation acceptance authority has been granted for the user of the first communication device 3.

The first communication device 3 and the second communication device 4 output the call information to the server 1 .
The server 1 acquires the call information, creates conversation log information based on the call information, and outputs the conversation log information to the first communication device 3 and the second communication device 4.

The second communication device 4 outputs an annotation setting instruction to the server 1 based on the user's operation. The second communication device 4 outputs a stop request to the server 1 to stop recording the conversation log.

The server 1 receives an annotation setting instruction from the second communication device 4. The server 1 receives a stop request from the second communication device 4. Based on receiving the stop request, the server 1 outputs notification information to the first communication device 3 and the second communication device 4. The notification indicates that recording of the conversation log will be stopped. The server 1 stops recording the conversation log. The server 1 may also stop creating conversation log information.

The first communication device 3 and the second communication device 4 obtain information about the notification from the server 1. The first communication device 3 and the second communication device 4 display a notification image on a display device based on the information about the notification.

The second communication device 4 outputs information about the annotation to the server 1 based on a user operation. Note that the second communication device 4 may output a stop request after outputting the information about the annotation to the server 1. Upon receiving the stop request, the server 1 may output information about the notification to the first communication device 3 and the second communication device 4.

The server 1 obtains information about the annotation from the second communication device 4. The server 1 outputs the information about the annotation to the first communication device 3. The first communication device 3 obtains the information about the annotation from the server 1. The first communication device 3 displays a notification image on the display device 351-1 based on the information about the annotation. The notification image includes information indicating that an annotation has been set.

Based on the user's operation, the first communication device 3 outputs an acceptance instruction to the server 1 indicating that the content of the annotation is accepted. The first communication device 3 may also output a rejection instruction to the server 1 indicating that the content of the annotation is rejected based on the user's operation.

The server 1 receives an acceptance instruction from the first communication device 3. Based on the acceptance instruction, the server 1 outputs a confirmation request to the second communication device 4. The confirmation request is a request to confirm that the annotation will be reflected. If the server 1 receives a rejection instruction from the first communication device 3, it may also output a confirmation request to the second communication device 4 based on the rejection instruction. In this case, the confirmation request is, for example, a request to confirm that the annotation content will not be reflected. The confirmation request may include text information such as, "The annotation content has been rejected. The annotation content will not be reflected."

The second communication device 4 outputs a confirmation instruction to the server 1 based on the user's operation. The second communication device 4 outputs a resume request to the server 1 to resume recording the conversation log.

The server 1 receives a confirmation instruction from the second communication device 4 to confirm the annotation setting. The server 1 updates the conversation log information based on the confirmation instruction. The server 1 adds information about the annotation to the conversation log information. The server 1 resumes recording the conversation log based on a resume request. The server 1 may also resume creating the conversation log information.

The server 1 outputs the updated conversation log information to the first communication device 3 and the second communication device 4. The first communication device 3 and the second communication device 4 display a conversation log image on a display device based on the updated conversation log information.

The above process can be repeated until the call between the first communication device 3 and the second communication device 4 is terminated.

According to this example, the information processing system 100 can set annotations for spoken sentences included in a conversation log based on a user's operation. The user can add information to the conversation log that cannot be recognized from the spoken sentences alone. In this way, the information processing system 100 can create a conversation log with an annotation inserted. The information processing system 100 can support the creation of a conversation log that will not cause misunderstandings to viewers of the conversation log, for example, when viewing the conversation log after the conversation has ended.
Furthermore, the information processing system 100 can display the presence or absence of annotations and the content of the annotations in a recognizable manner on the display screen of the conversation log. This allows the user to easily recognize the presence or absence of annotations and the content of the annotations. Furthermore, the information processing system 100 can reflect the annotation setting in the conversation log information when an instruction of consent is received from a user other than the user who set the annotation. Therefore, the information processing system 100 can set annotations in the conversation log based on the consent of the other users.

Figure 13 is a flowchart showing an example of the information processing procedure by server 1 according to the embodiment.

The following processing is assumed to begin when a call between the first communication device 3 and the second communication device 4 is initiated.

The call information acquisition unit 110 acquires call information from the first communication device 3 and the second communication device 4. (ACT 1). In ACT 1, for example, the call information acquisition unit 110 acquires voice information from the first communication device 3 and the second communication device 4. The voice information may include speaker identification information for identifying the user of the first communication device 3 or the second communication device 4. The memory control unit 112 may store the acquired call information in the auxiliary memory device 13.

The log creation unit 111 creates conversation log information based on the call information (ACT 2). In ACT 2, for example, the log creation unit 111 performs speech recognition processing based on audio information related to the user's utterance. The log creation unit 111 segments text information based on the audio information related to the user's utterance into spoken sentences. The log creation unit 111 links the segmented text information to speaker identification information. The log creation unit 111 creates conversation log information including the spoken sentences linked to the speaker identification information. The log creation unit 111 may also create conversation log information including speech time information related to each spoken sentence.

The memory control unit 112 stores the conversation log information in the auxiliary memory device 13 (ACT 3).

The output unit 114 outputs the conversation log information to at least one of the first communication device 3 and the second communication device 4 (ACT 4). At least one of the first communication device 3 and the second communication device 4 that has acquired the conversation log information displays the conversation log on a display device based on the conversation log information.

The log update unit 113 determines whether or not an annotation setting instruction has been received from the second communication device 4 (ACT 5). If the log update unit 113 has received an annotation setting instruction (ACT 5: YES), the process transitions from ACT 5 to ACT 6. If the log update unit 113 has not received an annotation setting instruction (ACT 5: NO), the process transitions from ACT 5 to ACT 9.

The log update unit 113 acquires information about the annotation from the second communication device 4 (ACT 6). In ACT 6, for example, the log update unit 113 acquires information indicating the range in which the annotation is to be set and information indicating the content of the annotation. The log update unit 113 saves the information about the annotation. The output unit 114 outputs the information about the annotation to the first communication device 3. Note that the log creation unit 111 stops creating the conversation log information based on having acquired a stop request to stop recording the conversation log from the second communication device 4.

The log update unit 113 updates the conversation log information based on the information about the annotation (ACT 7). In this example, it is assumed that an acceptance instruction indicating acceptance of the information about the annotation has been output from the first communication device 3. In ACT 7, for example, the log update unit 113 acquires the acceptance instruction from the first communication device 3. Based on the acquisition of the acceptance instruction, the output unit 114 outputs a confirmation request to the second communication device 4. The log update unit 113 acquires a confirmation instruction from the second communication device 4. Based on the confirmation instruction, the log update unit 113 adds the information about the annotation to the conversation log information.

The output unit 114 outputs the updated conversation log information to the first communication device 3 and the second communication device 4 (ACT 8). Note that the log creation unit 111 resumes creating the conversation log information upon receiving a resume request from the second communication device 4 to resume recording the conversation log.

The processor 11 determines whether the call between the first communication device 3 and the second communication device 4 has ended (ACT 9). If the processor 11 determines that the call has ended (ACT 9: YES), the processing ends. If the processor 11 determines that the call has not ended (ACT 9: NO), the processing transitions from ACT 9 to ACT 1. The processor 11 repeats the processing of ACT 1 to ACT 8 until the call between the first communication device 3 and the second communication device 4 has ended.

The processor 11 may perform the processes of ACT 2 to ACT 8 after the call between the first communication device 3 and the second communication device 4 has ended. For example, the call information acquisition unit 110 may acquire call information while the call between the first communication device 3 and the second communication device 4 is in progress. The log creation unit 111 may create the conversation log information after the call between the first communication device 3 and the second communication device 4 has ended.

In addition, the output unit 114 may output the conversation log information to at least one of the first communication device 3 and the second communication device 4 based on the conversation log display request. In this example, the processor 11 acquires the conversation log display request from at least one of the first communication device 3 and the second communication device 4. For example, the processor 11 acquires the conversation log display request based on a user operation by the user of the first communication device 3 or the second communication device 4. The user operation may include, for example, a click operation or a touch operation on a display button to display the conversation log displayed on the display device. Based on the conversation log display request, the output unit 114 may output the conversation log information to at least one of the first communication device 3 and the second communication device 4, which are the source of the conversation log display request.

The conversation log display request may be output from an electronic device other than the first communication device 3 and the second communication device 4. For example, the conversation log display request may be output by a user terminal (not shown). In this case, the output unit 114 may output the conversation log information to the user terminal that output the conversation log display request.

Figure 14 is a flowchart showing an example of the information processing procedure by the second communication device 4 according to the embodiment.

The following process is assumed to be initiated when a call between the first communication device 3 and the second communication device 4 is initiated. The following process may also be initiated when a conversation log is displayed on at least one of the first communication device 3 and the second communication device 4. In the following example, it is assumed that the user of the second communication device 4 has been granted annotation setting authority. The following process illustrates the process when an annotation is set by the user of the second communication device 4.

The display processing unit 411 displays a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 451-1.

The log processing unit 410 acquires an instruction to set an annotation based on a user operation (ACT 101). In ACT 101, for example, the log processing unit 410 acquires an instruction to set an annotation based on a user operation.

The log processing unit 410 outputs an instruction to set an annotation to server 1 (ACT 102). The log processing unit 410 outputs a stop request to server 1.

The display processing unit 411 displays the notification on the display device 451-1 (ACT 103). In ACT 103, for example, the log processing unit 410 acquires information about a notification indicating that recording of the conversation log will be stopped from the server 1. Based on the information about the notification, the display processing unit 411 displays a notification image including information indicating that recording of the conversation log will be stopped on the display device 451-1.

The log processing unit 410 acquires information about the utterance sentence included in the conversation log or the annotation to be set for the range of the utterance sentence based on the user's operation (ACT 104). In ACT 104, for example, the log processing unit 410 acquires information indicating the range in which the annotation is to be set based on the user's operation. The log processing unit 410 acquires information indicating the content of the annotation based on the user's operation.

The log processing unit 410 outputs information about the annotation to the server 1 (ACT 105). Note that the log processing unit 410 may also output a stop request to the server 1 in ACT 105. In this case, after ACT 105, the display processing unit 411 may display a notification image on the display device 451-1 based on the information about the notification.

The log processing unit 410 determines whether a confirmation request has been received from the server 1 to confirm that the annotation content is to be reflected (ACT 106). If the log processing unit 410 determines that a confirmation request has been received (ACT 106: YES), the process transitions from ACT 106 to ACT 107. If the log processing unit 410 determines that a confirmation request has not been received (ACT 106: NO), the process ends. Note that in this case, the log processing unit 410 may also receive a confirmation request to confirm that the annotation content is not to be reflected.

The display processing unit 411 displays a notification on the display device 451-1 based on the confirmation request to confirm whether the annotation content should be reflected (ACT 107). In ACT 107, for example, the display processing unit 411 displays a notification image on the display device 451-1 based on the confirmation request. The notification image includes information confirming that the annotation content should be reflected. Note that if the log processing unit 410 acquires a confirmation request to confirm that the annotation content should not be reflected, the display processing unit 411 may display a notification image on the display device 451-1 that includes information confirming that the annotation content should not be reflected.

The log processing unit 410 outputs a confirmation instruction to the server 1 (ACT 108). In ACT 108, for example, the log processing unit 410 acquires a confirmation instruction for confirming that the annotation content will be reflected in the conversation log information based on a user operation. The log processing unit 410 outputs the confirmation instruction to the server 1. The log processing unit 410 outputs a resume request to the server 1 to resume recording of the conversation log. The server 1 resumes recording of the conversation log. The server 1 may output information indicating that recording of the conversation log will be resumed to the second communication device 4. In this case, the display processing unit 411 may display a notification image including information indicating that recording of the conversation log will be resumed on the display device 451-1.

The log processing unit 410 obtains updated conversation log information based on the information about the annotation from the server 1 (ACT 109).

The display processing unit 411 displays the updated conversation log on the display device 451-1 based on the updated conversation log information (ACT110). The display processing unit 411 displays the annotation on the display device 451-1 in a recognizable manner. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

The processor 41 may perform the above-mentioned processing after the call between the first communication device 3 and the second communication device 4 has ended.

According to this example, the second communication device 4 can set annotations for spoken sentences included in the conversation log based on a user's operation. The user can add information to the conversation log that cannot be recognized from the spoken sentences alone. In this way, the second communication device 4 can create a conversation log with inserted annotations. The second communication device 4 can assist in creating a conversation log that will not cause misunderstandings to viewers of the conversation log, for example, when viewing the conversation log after the conversation has ended.
Furthermore, the second communication device 4 can display the presence or absence of annotations and the content of the annotations in a recognizable manner on the display screen of the conversation log. This allows the user of the second communication device 4 to easily recognize the presence or absence of annotations and the content of the annotations. Furthermore, the second communication device 4 can reflect the annotation settings in the conversation log information when it receives an instruction of consent from another user regarding the setting of the annotations. This allows the user of the second communication device 4 to obtain consent from another user regarding the setting of the annotations.

Figure 15 is a flowchart showing another example of the information processing procedure by the second communication device 4 according to the embodiment.

The following process is assumed to be initiated when a call between the first communication device 3 and the second communication device 4 is initiated. The following process may also be initiated when a conversation log is displayed on at least one of the first communication device 3 and the second communication device 4. In the following example, it is assumed that annotation setting authority has been granted to the user of the second communication device 4. The following process illustrates the process when an annotation is deleted by the user of the second communication device 4.

The display processing unit 411 displays a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 451-1.

The log processing unit 410 acquires an instruction to set an annotation based on a user operation (ACT 201). In ACT 201, for example, the log processing unit 410 acquires an instruction to delete an annotation based on a user operation.

The log processing unit 410 outputs an annotation setting instruction to the server 1 (ACT 202). The log processing unit 410 may also output a stop request to the server 1. Note that when the log processing unit 410 outputs a stop request, the display processing unit 411 may display a notification on the display device 451-1, as in ACT 103.

The log processing unit 410 acquires information regarding the deletion of an utterance sentence included in the conversation log or an annotation set in a range of an utterance sentence based on a user operation (ACT 203). In ACT 203, for example, the log processing unit 410 acquires information indicating the range of annotations to be deleted based on a user operation. The log processing unit 410 acquires information indicating the content of the deletion based on a user operation. The content of the deletion may include information indicating the deletion of some or all of the annotations.

The log processing unit 410 outputs information regarding the deletion to the server 1 (ACT 204). Note that the server 1 may output the information regarding the deletion to the first communication device 3. The server 1 may also obtain an instruction of consent regarding the information regarding the deletion from the first communication device 3.

The display processing unit 411 displays a notification on the display device 451-1 to confirm that the deletion details are to be reflected (ACT 205). In ACT 205, for example, the log processing unit 410 acquires a confirmation request from the server 1. The display processing unit 411 displays a notification on the display device 451-1 based on the confirmation request.

The log processing unit 410 outputs a confirmation instruction to the server 1 (ACT 206). In ACT 206, for example, the log processing unit 410 acquires a confirmation instruction to confirm that the deletion of the annotation will be reflected in the conversation log information based on a user operation. The log processing unit 410 outputs the confirmation instruction to the server 1. The log processing unit 410 may also output a resume request to the server 1 to resume recording of the conversation log. The server 1 resumes recording of the conversation log. The server 1 may output information indicating that recording of the conversation log will be resumed to the second communication device 4. In this case, the display processing unit 411 may display a notification image including information indicating that recording of the conversation log will be resumed on the display device 451-1.

The log processing unit 410 obtains updated conversation log information from the server 1 based on information regarding the deletion of the annotation (ACT 207).

The display processing unit 411 displays the updated conversation log on the display device 451-1 based on the updated conversation log information (ACT 208).

The processor 41 may perform the above-mentioned processing after the call between the first communication device 3 and the second communication device 4 has ended.

Figure 16 is a flowchart showing an example of the information processing procedure by the first communication device 3 according to the embodiment.

In the following example, it is assumed that annotation setting authority has been set for the user of the second communication device 4. It is assumed that annotation acceptance authority has been set for the user of the first communication device 3. The following processing is assumed to be initiated when an instruction to set an annotation is output from the second communication device 4 to the server 1. It is assumed that the processor 31 is displaying a conversation log based on a call between the first communication device 3 and the second communication device 4 on the display device 351-1.

The following process shows the process when an annotation is set by the user of the second communication device 4. Note that the following process can also be applied when an annotation is deleted by the user of the second communication device 4.

The processor 31 displays the notification (ACT 301). In ACT 301, for example, the processor 31 acquires information about the notification from the server 1. The processor 31 acquires information about the notification indicating that recording of the conversation log will be stopped from the server 1. Based on the information about the notification, the processor 31 displays a notification image including information indicating that recording of the conversation log will be stopped on the display device 351-1.

The processor 31 obtains information about the annotation from the server 1 (ACT 302). In ACT 302, for example, the processor 31 obtains from the server 1 information indicating the range in which the annotation is to be set and information indicating the content of the annotation.

The processor 31 displays a notification (ACT 303). In ACT 303, for example, the processor 31 displays a notification image including information indicating that an annotation has been set on the display device 351-1 based on information about the annotation. The notification image may include a button or the like that can indicate whether or not to accept the annotation content.

Based on the user's operation, the processor 31 determines whether an acceptance instruction to accept the annotation content has been acquired (ACT 304). If the processor 31 determines that an acceptance instruction has been acquired (ACT 304: YES), the process transitions from ACT 304 to ACT 305. If the processor 31 determines that an acceptance instruction has not been acquired (ACT 304: NO), the process transitions from ACT 304 to ACT 308. The fact that an acceptance instruction has not been acquired may also be interpreted as the fact that a rejection instruction indicating that the annotation content is rejected has been acquired.

The processor 31 outputs an acceptance instruction to the server 1 (ACT 305). Based on the acceptance instruction, the server 1 outputs a confirmation request to the second communication device 4. The server 1 acquires a confirmation instruction from the second communication device 4 to confirm the annotation setting. Based on the confirmation instruction, the server 1 updates the conversation log information.

The processor 31 obtains updated conversation log information from the server 1 (ACT 306).

The processor 31 displays the updated conversation log on the display device 351-1 based on the updated conversation log information (ACT 307). The processor 31 displays the annotation on the display device 351-1 in a recognizable manner. Displaying the annotation in a recognizable manner includes displaying at least one of the range in which the annotation is set and the content of the annotation in a recognizable manner.

The server 1 may also output information indicating that recording of the conversation log will resume to the first communication device 3. In this case, the processor 31 may display a notification image on the display device 351-1 that includes information indicating that recording of the conversation log will resume.

The processor 31 may perform the above-mentioned processing after the call between the first communication device 3 and the second communication device 4 has ended.

In this example, the first communication device 3 can display the presence or absence of annotations and their contents on the conversation log display screen in a recognizable manner. This allows the user of the first communication device 3 to easily recognize the presence or absence of annotations and their contents. Furthermore, the first communication device 3 can output an acceptance instruction to the server 1 for annotations set by other users based on the user's operation. The user can indicate whether or not they accept the annotation.

(Other embodiments)
In the above-described embodiment, an example has been described in which the examination booth includes the remote medical consultation support device 2 and the first communication device 3, but this is not limited to this. The examination booth may include either the remote medical consultation support device 2 or the first communication device 3. When the examination booth does not include the first communication device 3 but includes the remote medical consultation support device 2, the remote medical consultation support device 2 may realize the functions of the first communication device 3. When the examination booth does not include the remote medical consultation support device 2 but includes the first communication device 3, the first communication device 3 may realize the functions of the remote medical consultation support device 2.

In the above-described embodiment, a call between a doctor and a patient during remote medical care was used as an example, but this is not limited to this. The above-described embodiment can also be applied to calls between users of multiple communication devices, such as web conferencing, video conferencing, and distance learning.

The information processing device may be realized as a single device such as server 1, or as multiple devices with distributed functions.

The above-described embodiments may be applied not only to devices but also to methods executed by the devices. The above-described embodiments may be applied to programs that enable the computer of the devices to execute various functions. The above-described embodiments may be applied to recording media that store programs. The above-described embodiments may be applied not only to systems but also to methods executed by multiple elements included in the systems.

The processing circuit includes one or more circuits that perform multiple processes with multiple functions. For example, the circuit may be, but is not limited to, a processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

Each of the one or more circuits that make up the processing circuit performs one or more of the multiple processes. When the processing circuit is composed of a single circuit, the single circuit performs all of the multiple processes. When the processing circuit is composed of multiple circuits, each of the multiple circuits performs a part of the multiple processes. The part of the multiple processes may be one of the multiple processes, or two or more of the multiple processes. When the processing circuit is composed of multiple circuits, the multiple circuits may be included in a single device or may be distributed across multiple devices.

The program may be transferred in a state where it is stored on a device, or in a state where it is not stored on a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory, tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important, as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or memory card.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations corresponding to the embodiments may be adopted as appropriate.

In short, this invention is not limited to the above-described embodiment, and can be embodied by modifying the components in the implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be omitted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be combined as appropriate.

(Additional Note)
The above embodiment may be expressed as follows:
(1) To the computer,
The ability to view conversation logs based on calls between multiple devices,
a function of acquiring information about annotations to be set on utterances included in the conversation log based on user operations;
displaying an updated conversation log based on information about the annotations;
A program that can be executed.
(2) To the computer:
The program according to (1), further capable of executing a function of outputting a stop request for stopping recording of the conversation log based on an operation of the user.
(3) To the computer:
The program according to (1), further capable of executing a function of outputting a resume request for resuming recording of the conversation log based on an operation by the user.
(4) The program described in (1) is further capable of executing a function of outputting a confirmation request to confirm that the annotation is to be reflected based on obtaining consent information for the annotation from another terminal.
(5) The program according to (1), wherein the function of displaying the updated conversation log includes a function of displaying the annotation in a recognizable manner.
(6) displaying a conversation log based on calls between multiple devices;
acquiring information about annotations to be set on utterances included in the conversation log based on a user operation;
displaying an updated conversation log based on information about the annotations; and
An information processing method comprising:

1...Server, 2...Remote medical care support device, 3...First communication device, 4...Second communication device, 11...Processor, 12...Main memory, 13...Auxiliary storage device, 14...Communication interface, 21...Processor, 22...Main memory, 23...Auxiliary storage device, 24...Communication interface, 25...Input/output interface, 26...Display device, 27...Speaker, 28...Input device, 31...Processor, 32...Main memory, 33...Auxiliary storage device, 34...Communication interface, 35...Input/output interface, 38...Input device, 41...Processor, 42...Main memory, 43...Auxiliary storage device, 44...Communication interface, 45...Input/output interface, 48...Input device, 100...Information processing system, 110...Call information acquisition unit, 111...Log creation unit, 112...Memory control unit, 113...Log update unit, 114...Output unit, 251-1 to 251-m...Medical device, 351- 1...display device, 351-2...camera, 351-3...microphone, 351-4...speaker, 410...log processing unit, 411...display processing unit, 451-1...display device, 451-2...camera, 451-3...microphone, 451-4...speaker, Ara...annotation target area, Arb...annotation content input area, Arc...annotation purpose input area, Ard...notification area, Are...annotation area, Arf...notification area, Arg...annotation area, Arh...annotation target area A, Ba...Edit button, Bb...Annotation setting button, Bc...Annotation deletion button, Bd...OK button, Be...Cancel button, Bf...OK button, Bg...Reject button, Bh...OK button, Ima...Conversation log image, Imb...Edit instruction image, Imc...Notification image, Imd...Conversation log image, Ime...Edit input image, Imf...Notification image, Img...Notification image, Imh...Conversation log image, La...Spoken sentence, Ma...Speaker identification mark, Mb...Speaker identification mark.

Claims (5)

On the computer,
The ability to view conversation logs based on calls between multiple devices,
a function of acquiring information about annotations to be set on utterances included in the conversation log based on user operations;
displaying an updated conversation log based on information about the annotations;
a function of outputting a confirmation request for confirming that the annotation is to be reflected based on information regarding consent for the annotation obtained from another terminal;
A program that can be executed. On the computer,
2. The program according to claim 1, further capable of executing a function of outputting a stop request for stopping recording of the conversation log based on an operation by the user. On the computer,
2. The program according to claim 1, further capable of executing a function of outputting a resume request for resuming recording of the conversation log based on an operation by the user. The program described in claim 1, wherein the function for displaying the updated conversation log includes a function for displaying the annotations in a recognizable manner. Displaying a conversation log based on calls between multiple devices;
acquiring information about annotations to be set on utterances included in the conversation log based on a user operation;
displaying an updated conversation log based on information about the annotations; and
outputting a confirmation request for confirming that the annotation is to be reflected based on the information regarding the consent for the annotation obtained from the other terminal;
An information processing method comprising: