JP7757351B2 - Server device, display control method and program - Google Patents

Description

This disclosure relates to a server device, a display control method, and a program.

Recent advances in information and communications technology have promoted the use of telemedicine. For example, in telemedicine, images of the patient are shared between the hospital to which the patient is taken and a hospital located far away from the hospital, and conversations between medical staff at both hospitals are transmitted. Through this type of telemedicine, doctors at the local hospital can provide more appropriate treatment for the patient with the advice of the patient's primary care physician and specialists at the remote hospital.

Known telemedicine tools for realizing this type of telemedicine include those described in the following literature.

U.S. Pat. No. 10,483,007

On the other hand, in emergency medical settings such as emergency medical care and neonatal care, oral communication is the primary method of communication. When remote medical systems are used in such settings, it can become more difficult to ensure the accuracy of information and to share information among medical staff. For example, treatment is often based primarily on conversations between doctors and nurses on-site with the patient, making it difficult for doctors located remotely to confirm the patient's condition and make judgments. Furthermore, if the audio becomes unclear due to the speaker's position or microphone angle, it can become even more difficult for doctors located remotely to understand the conversation. Furthermore, when multiple locations participate remotely, doctors who join late or who are away for a short time find it difficult to understand the content of the conversation.

Furthermore, when a nurse visits a patient's home alone to receive a diagnosis from a remote doctor, such as in a home nursing program, the nurse must converse with the doctor while providing care, which often makes it difficult for the nurse to input information using a keyboard, and treatment and advice may be provided solely through audio communication. As a result, there is a concern that the remote doctor may not be able to accurately grasp the information due to mispronunciations or poor audio quality.

In light of the above problems, one objective of this disclosure is to provide technology for sharing conversational information exchanged in a remote medical system.

One aspect of the present disclosure relates to a server device having a communication control unit that controls communication between a first information processing terminal placed at a medical site and a second information processing terminal placed remotely from the medical site; a keyword detection unit that determines whether a predetermined keyword is included in conversation information exchanged between the first information processing terminal and the second information processing terminal and extracts related information related to the detected keyword from the conversation information ; and a display control unit that displays the detected keyword and the related information as provisional information on the first information processing terminal and the second information processing terminal, and when a confirmation operation is received from the first information processing terminal or the second information processing terminal, displays the provisional information as confirmed information on the first information processing terminal and the second information processing terminal.

This disclosure makes it possible to share conversation information exchanged in a remote medical system.

FIG. 1 is a schematic diagram illustrating a telemedicine system according to one embodiment of the present disclosure. FIG. 2 is a diagram illustrating patient information shared in a telemedicine system according to one embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a hardware configuration of a server device according to an embodiment of the present disclosure. FIG. 4 is a block diagram illustrating a functional configuration of a server device according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating registered keyword information according to an embodiment of the present disclosure. FIG. 6 is a diagram showing an example of a screen in a telemedicine system according to an embodiment of the present disclosure. FIG. 7 is a diagram showing an example of a screen in a telemedicine system according to an embodiment of the present disclosure. FIG. 8 is a flowchart illustrating a display control process according to an embodiment of the present disclosure.

Embodiments of the present disclosure will be described below with reference to the drawings.

In the following embodiment, a telemedicine system for sharing conversation information between information processing terminals is disclosed.

[Telemedicine System]
A telemedicine system according to one embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a schematic diagram illustrating a telemedicine system according to one embodiment of the present disclosure.

As shown in FIG. 1, the telemedicine system 10 includes an information processing terminal 20, a user terminal 30, and a server device 100. The information processing terminal 20, the user terminal 30, and the server device 100 are communicatively connected, for example, via a network (not shown).

The information processing terminal 20 is placed, for example, at a medical facility where a patient is present, and acquires images of the patient and medical staff, as well as conversations uttered by the patient and medical staff, and transmits the acquired images, conversations, etc. to the server device 100. The information processing terminal 20 also receives data acquired from the server device 100 and displays the received data, etc. on a monitor.

The user terminal 30 is used, for example, by a doctor located remotely from the patient's medical site, and provides the remote doctor with images, conversations, etc. received from the information processing terminal 20 via the server device 100 via a monitor and speaker. The user terminal 30 also transmits speech, messages, etc. sent by the remote doctor to the server device 100.

The server device 100 exchanges information between the information processing terminal 20 and the user terminal 30, as described in detail below. In the following example, the server device 100 acquires audio information of a conversation between a patient, doctor, nurse, etc., using the information processing terminal 20 and a doctor (e.g., a primary care physician, specialist, etc.), etc., using the user terminal 30. For example, the server device 100 may perform speech recognition on the acquired audio information and detect whether the acquired conversation information contains a predetermined keyword as a result of the recognition using text data. If the server device 100 detects a predetermined keyword in the acquired conversation information, it extracts related information related to the detected keyword and displays the extracted related information as provisional information. For example, if "body temperature" is detected in the conversation information as a predetermined keyword, the server device 100 detects a numerical value corresponding to body temperature from the context before and after "body temperature" in the conversation information and displays the detected numerical value together with "body temperature" as related information on the information processing terminal 20 and the user terminal 30 as provisional information. Then, when a confirmation operation is received for the displayed provisional information, the server device 100 displays the keyword and related information as confirmed information on the information processing terminal 20 and the user terminal 30.

For example, in the example shown in Figure 2, keywords such as "body temperature," "weight," "blood pressure," "breathing," "tracheal intubation," and "age in weeks" can be detected. Then, as related information for "body temperature" and "tracheal intubation," "36.0°C" and "yes" are displayed as confirmed information. Furthermore, as related information for "weight" and "age in weeks," "2,200g" and "30 weeks" are displayed as provisional information, waiting for the user to confirm.

As a result, conversational information exchanged orally between medical staff at the site where the information processing terminal 20 is located and remote advisors, specialists, etc. is displayed as text data on the screen, allowing for the sharing of conversational information. Furthermore, keywords and related information automatically extracted by the server device 100 are initially displayed as provisional information, and are then displayed as confirmed information after confirmation by the user. This allows keywords and related information to be shared as accurate information.

Here, the server device 100 may be realized by any type of computing device, information processing device, etc., and may have a hardware configuration such as that shown in FIG. 3. That is, the server device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

Programs or instructions that implement the various functions and processes described below in the server device 100 may be stored on removable storage media such as CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is inserted into the drive device 101, the programs or instructions are installed from the storage medium via the drive device 101 into the storage device 102 or memory device 103. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may be downloaded from an external device via a network, etc. The storage device 102 is implemented by a hard disk drive, etc., and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions. The memory device 103 is implemented by random access memory, static memory, etc., and when a program or instruction is activated, it reads and stores the program, instruction, data, etc. from the storage device 102. The storage device 102, memory device 103, and removable storage media may be collectively referred to as non-transitory storage media.

The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., each of which may be composed of one or more processor cores, and executes various functions and processes of the server device 100 (described below) in accordance with programs, instructions, and data such as parameters required to execute the programs or instructions stored in the memory device 103. The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speakers, headset, and printer, and input/output devices such as a touch panel, and implements an interface between the user and the server device 100. For example, a user operates the server device 100 by operating a keyboard, mouse, etc. to use a GUI (Graphical User Interface) displayed on a display or touch panel. The communication device 106 is realized by various communication circuits that perform communication processing with communication networks such as external devices, the Internet, and LANs (Local Area Networks).

However, the above-described hardware configuration is merely an example, and the server device 100 according to the present disclosure may be realized using any other appropriate hardware configuration.

[Server device]
Next, a server device 100 according to an embodiment of the present disclosure will be described. Fig. 4 is a block diagram showing a functional configuration of the server device 100 according to an embodiment of the present disclosure. As shown in Fig. 4, the server device 100 includes a communication control unit 110, a keyword detection unit 120, and a display control unit 130. For example, one or more functional units of the communication control unit 110, the keyword detection unit 120, and the display control unit 130 may be realized by one or more processors 104 executing one or more programs or instructions.

The communication control unit 110 controls communication between the information processing terminal 20 located at the medical site and the user terminal 30 located remotely from the medical site. For example, when the communication control unit 110 receives video information, audio information, text information, etc. from the information processing terminal 20 and the user terminal 30, it transmits the received video information, audio information, text information, etc. to the information processing terminal 20 and the user terminal 30 in one format, such as a video conference, online conference, or chat, or in a combination of two or more formats.

Here, the communication control unit 110 may perform speech recognition on voice information received from the information processing terminal 20 and the user terminal 30, for example, conversation information exchanged orally between the user of the information processing terminal 20 and the user of the user terminal 30, and convert the conversation information into text information. The communication control unit 110 may also convert text information received from the information processing terminal 20 and the user terminal 30, for example, text information such as advice sent by the user of the user terminal 30, into voice information, and provide this to the information processing terminal 20 as voice information as well as text information.

The keyword detection unit 120 determines whether predetermined keywords are included in the conversation information exchanged between the information processing terminal 20 and the user terminal 30, and extracts related information related to the detected keywords. For example, in the case of perinatal care, keywords may be terms related to the patient's vital signs, medical history, treatment, etc., such as "age after birth," "weight," "blood pressure," "breathing," "endotracheal intubation," and "infectious disease," and may be pre-registered by the user, etc. In addition, related information may be a numerical value for each keyword (e.g., "30 weeks" for "age after birth," "36.0°C" for "body temperature"), an answer (e.g., "yes" or "no" for "endotracheal intubation"), etc.

Registered keywords may be registered in registered keyword information such as that shown in FIG. 5. In the illustrated example, "weeks after birth" is registered as the keyword number "#001," and it is registered that the value of "weeks after birth" is expressed in units of "weeks." Furthermore, "weight" is registered as the keyword number "#002," and it is registered that the value of "weight" is expressed in units of "g." Similarly, keywords such as "body temperature," "blood pressure," "respiration," "tracheal intubation," and "infectious disease," along with the numerical units of each keyword or answer candidates such as "yes" or "no," may be registered in the registered keyword information. Furthermore, a priority may be assigned to each keyword, as shown in the figure.

For example, if the conversation information exchanged between the information processing terminal 20 and the user terminal 30 includes the content "The baby is 30 weeks old," the keyword detection unit 120 may detect "weeks old" as a keyword and extract "30 weeks" as related information, using the "weeks" corresponding to the "weeks old" registered in the registered keyword information as a clue. Similarly, the keyword detection unit 120 may extract related information such as numerical values or answers corresponding to each keyword included in the conversation information.

The display control unit 130 displays the detected keywords and related information as provisional information, and upon receiving a confirmation operation, displays the provisional information as final information. Specifically, upon acquiring keywords included in conversation information and related information corresponding to those keywords, the display control unit 130 may first display the acquired keywords and related information as provisional information in text format on the information processing terminal 20 and the user terminal 30. In other words, the keywords and related information are acquired through voice recognition and word detection by the server device 100, and there is a possibility of misrecognition and/or misdetection. For this reason, the acquired keywords and related information may be adopted as final information only after being approved by the user.

After the keywords and related information are displayed as provisional information on the monitors of the information processing terminal 20 and the user terminal 30, if a confirmation operation is received that the provisional information is correct, the display control unit 130 may display the displayed provisional information as confirmed information. For example, in the example screen shown in FIG. 2, "body temperature" and "tracheal intubation" are displayed as confirmed information upon confirmation. On the other hand, "weight" and "age in weeks" have not received a confirmation operation and are displayed as provisional information. At this time, the display control unit 130 may highlight (e.g., blink) the provisional information to prompt the user to confirm the provisional information.

Here, the confirmation operation may be a touch operation on the monitor screen of the information processing terminal 20 or the user terminal 30. For example, when temporary information displayed on the monitor screen of the information processing terminal 20 or the user terminal 30 is touched, the display control unit 130 may determine that the temporary information has been approved. On the other hand, when temporary information displayed on the monitor screen of the information processing terminal 20 or the user terminal 30 is subjected to a slide operation or the like, the display control unit 130 may determine that the temporary information has been rejected. In this case, the display control unit 130 may request re-input from the information processing terminal 20 and the user terminal 30.

Alternatively, the confirmation operation may be a voice operation on the information processing terminal 20 or the user terminal 30. For example, if "30 weeks old" is displayed as provisional information, and a positive voice response such as "30 weeks old is fine" is received from the user, the display control unit 130 may determine that the provisional information has been approved. On the other hand, if a negative voice response such as "The baby is not 30 weeks old" is received, the display control unit 130 may determine that the provisional information has been rejected and request re-entry into the information processing terminal 20 and the user terminal 30. Furthermore, when confirming "30 weeks old," the display control unit 130 may highlight "30 weeks old" on the monitor screen of the information processing terminal 20 and/or the user terminal 30.

Here, if the detected keyword was sent from the information processing terminal 20, the display control unit 130 may be configured to accept a confirmation operation from the information processing terminal 20. In other words, the display control unit 130 may be configured to accept a confirmation operation for the provisional information of the keyword and related information from the information processing terminal 20 or user terminal 30 of the user who spoke the keyword and related information, or may not accept a confirmation operation from a terminal other than the one from which the speech was made.

In one embodiment, the display control unit 130 may associate the detected keywords with the related information and display them on the information processing terminal 20 and the user terminal 30. For example, the display control unit 130 may display each detected keyword and related information together with an image of the patient and an image of the participating doctor so that the correspondence between each keyword and its related information can be understood. As shown in the figure, provisional information may be displayed with "(provisional)" and final information may be displayed with "(confirmed)". Provisional information may also be highlighted, such as by flashing, to prompt the user to confirm.

For example, the display control unit 130 may display keywords and related information according to the set display mode. For example, if the conversation information includes "age 30 weeks, tracheal intubation completed, weight 2,200 g, body temperature 36.0°C," the display control unit 130 may display the keywords and related information in chronological order of the utterances. For example, the display control unit 130 may display keywords that were spoken more recently at the top (live mode). In this case, for the above-mentioned conversation information, the display control unit 130 may display "Body temperature 36.0°C (confirmed)," "Weight 2,200 g (tentative)," "Tracheal intubation yes (confirmed)," and "Age 30 weeks (confirmed)" from the top of the keyword screen on a keyword screen showing keywords and related information, as shown in FIG. 6.

Alternatively, the display control unit 130 may display keywords and related information in order of priority. For example, the display control unit 130 may display keywords with higher priority at the top (priority mode). Here, the priority of each keyword may be set in the registered keyword information. In the example shown in FIG. 5, "age at birth" has the highest priority A, "weight" and "tracheal intubation" have the next highest priority B, and "body temperature" has the next highest priority C. In accordance with this priority, the display control unit 130 may display "age at birth 30 weeks (confirmed)," "weight 2,200 g (tentative)," "tracheal intubation yes (confirmed)," and "body temperature 36.0°C (confirmed)" on the keyword screen from the top, as shown in FIG. 7. Furthermore, keywords with high priority and related information may be highlighted.

The display control unit 130 may further determine the display order of each keyword and related information according to a frequency mode and a history mode. The frequency mode displays keywords detected more frequently in the conversation information at the top, while the history mode may display keywords in chronological order of the conversation's progress. The display control unit 130 may also display only specific keywords and their related information (registration mode), or may display general medical terms that appear in the conversation information in addition to the specific keywords and their related information (general medical terminology mode). The user of the information processing terminal 20 and the user terminal 30 may set which display mode to use.

The displayed keywords and related information may also be highlighted by user operation of the information processing terminal 20 and the user terminal 30. The information processing terminal 20 and the user terminal 30 do not necessarily need to display the same screen. For example, the information processing terminal 20 may display a patient image in a relatively small size, and the keyword screen in a relatively large size. On the other hand, the user terminal 30 may display a patient image in a relatively large size, and the keyword screen in a relatively small size.

Furthermore, keywords may be changed depending on the patient's symptoms and the type of disease. Furthermore, conversation information is not necessarily limited to audio information, but may also be input as text information. For example, the information processing terminal 20 or the user terminal 30 may be muted and conversation information may be input via a keyboard. Furthermore, the speaker may be identified in the conversation information and the identified speaker may be displayed. For example, in Figures 6 and 7, an image of the speaking doctor is displayed with a black frame. This makes it possible to clearly identify the questioner and the answerer. Furthermore, the communication control unit 110 may perform control so that a confirmation operation is not performed on the content of a question asked by a questioner based on the questioner's statement.

[Display control processing]
Next, a display control process according to an embodiment of the present disclosure will be described. Fig. 8 is a flowchart showing the display control process according to an embodiment of the present disclosure. The display control process is executed by the server device 100 described above, and more specifically, may be realized by one or more processors 104 of the server device 100 executing one or more programs or instructions stored in one or more memory devices 103.

As shown in FIG. 8, in step S101, the server device 100 acquires conversation information. For example, the conversation information may be speech-format conversation information indicating a conversation exchanged between the information processing terminal 20 and the user terminal 30. For example, the server device 100 may perform speech recognition on the conversation information and acquire text data of the utterances included in the conversation information.

In step S102, the server device 100 determines whether the conversation information contains a predetermined keyword. Specifically, the server device 100 references the registered keyword information in which the keyword to be extracted is registered, and determines whether the conversation information contains a keyword registered in the registered keyword information. If a keyword is detected in the conversation information (S102: YES), the display control process proceeds to step S103. If a keyword is not detected in the conversation information (S102: NO), the display control process returns to step S101.

In step S103, the server device 100 displays the detected keywords and related information as provisional information. Specifically, the server device 100 extracts numerical values or answers related to the detected keywords from the conversation information, associates the keywords with the related information, and displays them as provisional information on the information processing terminal 20 and the user terminal 30. The server device 100 then waits for a confirmation operation on the keywords and related information.

In step S104, the server device 100 determines whether a confirmation operation has been received. If a confirmation operation has been received (S104: YES), the display control process proceeds to step S105. If a confirmation operation has not been received (S104: NO), the display control process continues to wait for a confirmation operation.

In step S105, the server device 100 displays the keywords and related information as confirmed information. Once the keywords and related information are determined as confirmed information, medical staff using the information processing terminal 20 and the user terminal 30 can provide medical treatment, advice, and treatment based on the confirmed confirmed information.

The server device 100 and display control process described above enable more accurate information to be shared between on-site medical staff and remote doctors by displaying oral utterances made in the medical field as text data. While the above-described embodiment focuses on on-site medical staff and remote doctors, the present disclosure is not limited to this and may also be operated by patients, their family members, etc.

In addition, the following supplementary notes are provided in relation to the above description.
(Appendix 1)
a communication control unit that controls communication between a first information processing terminal placed at a medical site and a second information processing terminal placed remotely from the medical site;
a keyword detection unit that determines whether a predetermined keyword is included in conversation information exchanged between the first information processing terminal and the second information processing terminal, and extracts information related to the detected keyword;
a display control unit that displays the detected keyword and the related information as provisional information, and, upon receiving a confirmation operation, displays the provisional information as confirmed information;
A server device having the above configuration.
(Appendix 2)
2. The server device according to claim 1, wherein the display control unit associates the detected keyword with the related information and displays the associated information on the first information processing terminal and the second information processing terminal.
(Appendix 3)
3. The server device according to claim 1, wherein the related information indicates a numerical value or an answer related to the keyword.
(Appendix 4)
A server device described in any one of appendices 1 to 3, wherein if the detected keyword is sent from the first information processing terminal, the display control unit accepts the confirmation operation from the first information processing terminal.
(Appendix 5)
The server device according to any one of appendices 1 to 4, wherein the confirmation operation includes at least one of a touch operation on a monitor screen of the first information processing terminal or the second information processing terminal, and a voice operation on the first information processing terminal or the second information processing terminal.
(Appendix 6)
6. The server device according to any one of appendices 1 to 5, wherein the display control unit displays the keyword and the related information in accordance with a set display mode.
(Appendix 7)
7. The server device of claim 6, wherein the display mode is selected from a live mode, a priority mode, a frequency mode, a history mode, a registration mode, and a medical terminology mode.
(Appendix 8)
Acquiring conversation information communicated between a first information processing terminal located at a medical site and a second information processing terminal located remotely from the medical site;
determining whether a predetermined keyword is included in the conversation information and extracting information related to the detected keyword;
displaying the detected keyword and the related information as provisional information, and when a confirmation operation is received, displaying the provisional information as confirmed information;
A display control method performed by a computer.
(Appendix 9)
Acquiring conversation information communicated between a first information processing terminal located at a medical site and a second information processing terminal located remotely from the medical site;
determining whether a predetermined keyword is included in the conversation information and extracting information related to the detected keyword;
displaying the detected keyword and the related information as provisional information, and when a confirmation operation is received, displaying the provisional information as confirmed information;
A program that causes a computer to execute the following.

The above describes in detail examples of the present disclosure, but the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims.

This disclosure is useful in telemedicine systems.

REFERENCE SIGNS LIST 10 Telemedicine system 20 Information processing terminal 30 User terminal 100 Server device 110 Communication control unit 120 Keyword detection unit 130 Display control unit

Claims (9)

a communication control unit that controls communication between a first information processing terminal placed at a medical site and a second information processing terminal placed remotely from the medical site;
a keyword detection unit that determines whether a predetermined keyword is included in conversation information exchanged between the first information processing terminal and the second information processing terminal, and extracts information related to the detected keyword from the conversation information ;
a display control unit that displays the detected keyword and the related information as provisional information on the first information processing terminal and the second information processing terminal , and, when a confirmation operation is received from the first information processing terminal or the second information processing terminal , displays the provisional information as confirmed information on the first information processing terminal and the second information processing terminal;
A server device having: The server device according to claim 1, wherein the display control unit associates the detected keywords with the related information and displays them on the first information processing terminal and the second information processing terminal. The server device according to claim 1, wherein the related information indicates a numerical value or an answer related to the keyword. The server device according to claim 1, wherein if the detected keyword was transmitted from the first information processing terminal, the display control unit accepts the confirmation operation from the first information processing terminal. The server device according to claim 1, wherein the confirmation operation includes at least one of a touch operation on a monitor screen of the first information processing terminal or the second information processing terminal, and a voice operation on the first information processing terminal or the second information processing terminal. The server device according to claim 1, wherein the display control unit displays the keywords and the related information according to a set display mode. The server device of claim 6, wherein the display mode is selected from a live mode, a priority mode, a frequency mode, a history mode, a registration mode, and a medical terminology mode. Acquiring conversation information communicated between a first information processing terminal located at a medical site and a second information processing terminal located remotely from the medical site;
determining whether the conversation information includes a predetermined keyword, and extracting information related to the detected keyword from the conversation information ;
displaying the detected keyword and the related information as provisional information on the first information processing terminal and the second information processing terminal, and when a confirmation operation is received from the first information processing terminal or the second information processing terminal , displaying the provisional information as confirmed information on the first information processing terminal and the second information processing terminal;
A display control method performed by a computer. Acquiring conversation information communicated between a first information processing terminal located at a medical site and a second information processing terminal located remotely from the medical site;
determining whether the conversation information includes a predetermined keyword, and extracting information related to the detected keyword from the conversation information ;
displaying the detected keyword and the related information as provisional information on the first information processing terminal and the second information processing terminal, and when a confirmation operation is received from the first information processing terminal or the second information processing terminal , displaying the provisional information as confirmed information on the first information processing terminal and the second information processing terminal;
A program that causes a computer to execute the following.