JP7382760B2 - Medical support equipment and programs - Google Patents

Description

Embodiments of the present invention relate to a medical support device and a program.

2. Description of the Related Art Telemedicine systems have conventionally existed in which medical procedures such as examinations and diagnoses are performed remotely. For example, a technique has been proposed to remotely convey instructions and advice to a user via a character such as an avatar representing a doctor or the like.

On the other hand, automatic conversation technology and automatic diagnosis technology using AI (artificial intelligence) etc. are also beginning to become popular.

By the way, when medical professionals such as doctors inform users, such as patients, of medical matters such as test results and diagnosis results, they take into account the feelings of the users and adjust the way they speak and interact depending on the content of the announcement. It is common practice to take this into account. This also applies to remote treatment, but if the above-mentioned automatic conversation technology is used to mechanically notify the user, the user may feel uneasy depending on the way the user speaks or interacts with the patient.

Japanese Patent Application Publication No. 2017-143992

The problem to be solved by the present invention is to reduce the anxiety of users who receive telemedicine.

The medical support device according to the embodiment includes a first generation section, a second generation section, and a control section. The first generation unit generates first data for operating an object representing a speaker who converses with a user. The second generation unit generates second data that expresses in voice the content of the utterance that the object is to utter. The control unit controls the movement of the object and /or the method of expressing the voice according to the emotional state of the user estimated based on the medical notice included in the content of the speaker's utterance or the reaction of the user. Control to change. Further, the control unit prioritizes the announcement item and the emotional state, and controls to change the motion of the object and/or the method of expressing the voice according to the priority.

FIG. 1 is a diagram showing an example of the configuration of a remote medical system according to the first embodiment. FIG. 2 is a diagram illustrating a configuration example of a user terminal according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of the medical support device according to the first embodiment. FIG. 4 is a diagram showing an example of the data structure of speaker setting information according to the first embodiment. FIG. 5 is a diagram illustrating an example of a data structure of expression setting information according to the first embodiment. FIG. 6 is a flowchart illustrating an example of medical support processing performed by the medical support device of the first embodiment. FIG. 7 is a diagram illustrating a configuration example of a medical support device according to the second embodiment.

Hereinafter, embodiments of a medical support device and a program will be described with reference to the drawings.

[First embodiment]
FIG. 1 is a diagram showing an example of the configuration of a remote medical system according to the first embodiment. As shown in FIG. 1, the telemedicine system 1 includes a medical support device 10 and a user terminal 20. The medical support device 10 and the user terminal 20 are communicably connected via a network N such as the Internet. Note that the number of user terminals 20 connected to network N is not limited to the number shown in FIG. 1.

The medical support device 10 supports remote medical care for users such as subjects or patients. The medical support device 10 is installed, for example, in a medical facility such as a hospital, and carries out a conversation with a user who operates a user terminal 20 to notify (notify) medical notification matters. Note that the medical information processing device 100 is realized by, for example, computer equipment such as a workstation and a server device.

The user terminal 20 is a terminal device used by a user of the telemedicine system 1. A user can enjoy telemedicine by accessing the medical support device 10 via the user terminal 20. Note that the user terminal 20 is realized by, for example, a computer device such as a PC (Personal Computer), a smartphone, or a tablet terminal.

In the telemedicine system 1 of this embodiment, the medical support device 10 is used, for example, to notify the user of the results of an examination or diagnosis performed on the user at a medical facility. Each device included in the telemedicine system 1 will be described below.

FIG. 2 is a diagram showing an example of the configuration of the user terminal 20. As shown in FIG. As shown in FIG. 2, the user terminal 20 includes a communication interface 210, a storage circuit 220, an input interface 230, an audio input interface 240, an imaging device 250, a display 260, an audio output interface 270, and a processing circuit. 280.

The communication interface 210 is connected to the processing circuit 280 and controls transmission and communication of various data via the network N. For example, the communication interface 210 is realized by a network card, network adapter, NIC (Network Interface Controller), or the like.

The storage circuit 220 is connected to the processing circuit 280 and stores various data. For example, the storage circuit 220 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, or the like.

The input interface 230 is connected to the processing circuit 280 , converts an input operation received from an operator into an electrical signal, and outputs the electrical signal to the processing circuit 280 . For example, the input interface 230 is realized by a trackball, a switch button, a mouse, a keyboard, a touch panel, or the like.

The audio input interface 240 is an example of a sound collection device. The audio input interface 240 is connected to the processing circuit 280 , collects the audio uttered by the user, converts the audio into an electrical signal, and outputs the electrical signal to the processing circuit 280 . For example, the voice input interface 240 is realized by a microphone or the like.

Imaging device 250 is an example of an imaging apparatus. The imaging device 250 is connected to the processing circuit 280 and outputs a captured image (video) of the user obtained by imaging to the processing circuit 280. For example, the imaging device 250 is realized by an imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

Note that the input interface 230 and the voice input interface 240 are not limited to those that include physical components such as a keyboard and a microphone. For example, an electrical signal processing circuit that receives an electrical signal from an external input device provided separately from the user terminal 20 and outputs this electrical signal to the processing circuit 280 is also an example of the input interface 230 and the audio input interface 240. included. Furthermore, the input interface 230 may be configured to receive a captured image from an imaging device 250 that is an external input device.

The display 260 is connected to the processing circuit 280 and displays various information and various image data output from the processing circuit 280. For example, the display 260 is realized by a liquid crystal monitor, a CRT (cathode ray tube) monitor, a touch panel, or the like.

The audio output interface 270 is connected to the processing circuit 280 and outputs the sound (voice, etc.) output from the processing circuit 280. For example, the audio output interface 270 is realized by a speaker or the like. Note that the audio output interface 270 is not limited to one that includes a physical component such as a speaker. For example, a processing circuit that outputs an audio signal to an external audio output device (such as headphones) provided separately from the user terminal 20 is also included in the example of the audio output interface 270.

The processing circuit 280 comprehensively controls the operation of the user terminal 20. For example, processing circuit 280 is implemented by a processor. Further, the processing circuit 280 has a reception function 281 , a display function 282 , an audio output function 283 , a collection function 284 , and a transmission function 285 .

Here, for example, each processing function executed by the reception function 281, display function 282, audio output function 283, collection function 284, and transmission function 285, which are the components of the processing circuit 280, is in the form of a program executable by a computer. It is stored in the memory circuit 220. The processing circuit 280 reads each program from the storage circuit 220 and executes it, thereby realizing a function corresponding to each program. In other words, the processing circuit 280 in a state where each program is read has each function shown in the processing circuit 280.

The reception function 281 cooperates with the communication interface 210 to receive various data transmitted from an external device (medical support device 10) to its own device. For example, the reception function 281 receives display data and audio output data provided from the medical support device 10.

The display function 282 cooperates with the display 260 to display various information on the display 260. For example, the display function 282 causes the display 260 to display an image (a speaker image to be described later) based on first data provided from the medical support device 10.

The audio output function 283 cooperates with the audio output interface 270 to output various sounds from the audio output interface 270. For example, the audio output function 283 causes the audio output interface 270 to output audio (speaker audio to be described later) based on second data provided from the medical support device 10 .

The collection function 284 collects various data related to the user of the user terminal 20. For example, the collection function 284 collects the user's voice collected by the voice input interface 240 as voice data. Further, for example, the collection function 284 collects captured images of the user captured by the imaging device 250 as captured data. Furthermore, the collection function 284 acquires attribute data indicating the user's attributes from the storage circuit 220 of the user terminal 20 or the like. For example, the attribute data includes the user's age, gender, nationality, language used, and the like.

Note that the data collected by the collection function 284 is not limited to the above example. For example, the collection function 284 may collect history data that records behavior history such as purchase history of various products and viewing history of websites, video sites, etc. from the storage circuit 220 of the user terminal 20 and the like. Further, for example, the collection function 284 may collect history data that records selection history (behavior history) of character data and voice data, which will be described later.

The transmission function 285 cooperates with the communication interface 210 to transmit various data to an external device (medical support device 10). For example, the transmission function 285 transmits the audio data, imaging data, attribute data, history data, etc. collected by the collection function 284 to the medical support device 10.

Next, with reference to FIG. 3, the medical support device 10 will be described. FIG. 3 is a diagram showing an example of the configuration of the medical support device 10. As shown in FIG. 3, the medical support device 10 includes a communication interface 110, a storage circuit 120, an input interface 130, a display 140, and a processing circuit 150.

The communication interface 110 is connected to the processing circuit 150 and controls transmission and communication of various data via the network N. For example, the communication interface 110 is realized by a network card, network adapter, NIC, or the like.

The storage circuit 120 is connected to the processing circuit 150 and stores various data, various setting information, and the like. For example, the storage circuit 120 is realized by a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like.

The input interface 130 is connected to the processing circuit 150 , converts an input operation received from an operator into an electrical signal, and outputs the electrical signal to the processing circuit 150 . For example, the input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touch panel, or the like. Note that in this specification, the input interface 130 is not limited to one that includes a physical component such as a keyboard. For example, an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the medical support device 10 and outputs this electrical signal to the processing circuit 150 is also an example of the input interface 130. include.

The display 140 is connected to the processing circuit 150 and displays various information and various image data output from the processing circuit 150. For example, the display 140 is realized by a liquid crystal monitor, a CRT monitor, a touch panel, or the like.

The processing circuit 150 centrally controls the operation of the medical support device 10. For example, processing circuit 150 is implemented by a processor. Furthermore, the processing circuit 150 has a reception function 151 , an image generation function 152 , an audio generation function 153 , a control function 154 , and a transmission function 155 .

Here, for example, each processing function executed by the reception function 151, image generation function 152, sound generation function 153, control function 154, and transmission function 155, which are the components of the processing circuit 150, is in the form of a program executable by a computer. is stored in the memory circuit 120. The processing circuit 150 reads each program from the storage circuit 120 and executes it, thereby realizing a function corresponding to each program. In other words, the processing circuit 150 in a state where each program is read has each function shown in the processing circuit 150.

Note that the image generation function 152 is an example of a first generation unit and a first generation function in the claims. Further, the audio generation function 153 is an example of a second generation unit and a second generation function in the claims. Further, the control function 154 is an example of a control unit and a control function in the claims.

The reception function 151 cooperates with the communication interface 110 to receive various data sent from an external device (user terminal 20) to the own device. For example, the reception function 151 receives user audio data, image data, attribute data, history data, etc. provided from the user terminal 20.

The image generation function 152 generates a speaker image representing a speaker having a conversation with the user of the user terminal 20 as first data. Specifically, the image generation function 152 generates a dynamic speaker image based on the character data that is the source of the speaker image under the control of the control function 154. Character data is an example of an object representing a speaker, and is, for example, a character image representing a character or avatar such as an animal, a person, or a robot in two or three dimensions. Note that the character data may be created by processing an image of an actual doctor or the like.

FIG. 4 is a diagram showing an example of the data structure of character setting information. As shown in FIG. 4, character setting information stores a plurality of types of character data in association with voice data that is the source of the speaker's voice. Such character setting information is stored in advance in the storage circuit 120, for example.

For example, when character data of "dog" is specified by the control function 154, which will be described later, the image generation function 152 generates a speaker image by using this character data. More specifically, the image generation function 152 displays the character's facial expressions, gestures, opening/closing movements of the mouth, etc. in animation in accordance with the utterance timing of the speaker's voice and the expression method instructed by the control function 154, which will be described later. generate a speaker image. In other words, the speaker image becomes data for operating the character image (object).

Note that, in this embodiment, the speaker image generated by the image generation function 152 is image data such as a character image and animation representing its movement, but it is not limited to this form.

For example, the image generation function 152 may generate instruction data indicating parameters and the like for displaying (generating) a speaker image as the first data. In this case, for example, by storing the above-mentioned character setting information in the user terminal 20 and transmitting the first data to the user terminal 20, the display function 282 can display the character data specified by the first data. It is also possible to generate a personal image. Thereby, the speaker image can be presented to the user of the user terminal 20 in the same way as when transmitting the image data itself.

The voice generation function 153 generates, as second data, a speaker's voice representing the voice to be uttered by the character data (speaker image). Specifically, under the control of the control function 154, the voice generation function 153 generates the speaker's voice representing the content of the utterance by using voice data corresponding to the character data of the speaker image. Further, the voice generation function 153 controls the tone, speech speed, intonation, etc. of the speaker's voice under the control of the control function 154 . For example, the voice generation function 153 can generate the speaker's voice from the voice data by using a known voice synthesis technique or the like.

In this embodiment, it is assumed that the speaker's voice generated by the voice generation function 153 is voice data representing the voice itself, but the present invention is not limited to this form.

For example, the voice generation function 153 may generate instruction data indicating utterance content, parameters, etc. for outputting the speaker's voice as the second data. In this case, for example, by providing the user terminal 20 with the character setting information and the voice synthesis device described above, the voice output function 283 can generate the speaker's voice based on the instruction content of the second data. This allows the user of the user terminal 20 to hear the speaker's voice, similar to when transmitting the voice data itself.

Note that in this embodiment, the content of the utterance on which the speaker's voice is based is automatically generated from the content of the user's utterance by using automatic conversation technology such as AI (artificial intelligence). Further, medical notification items such as test and diagnosis results may be automatically generated from information indicating the user's test results and diagnosis results, or may be stored in advance in the storage circuit 120 or the like in the form of a text file or the like. It may also be in the form shown below.

Further, in this embodiment, an example has been shown in which character data and voice data are associated on a one-to-one basis, but the present invention is not limited to this, and a configuration in which character data and voice data can be individually designated may be used.

The control function 154 cooperates with the image generation function 152 and the voice generation function 153 to execute various processes for facilitating conversation with the user of the user terminal 20. Various processes performed by the control function 154 will be described below.

(Object selection process)
The control function 154 selects character data and voice data to be used for generating a speaker image and speaker voice from among the character data and voice data registered in the character setting information. Then, the control function 154 instructs the image generation function 152 and the voice generation function 153 to use the selected character data and voice data to generate a speaker image and speaker voice using the character data and voice data.

Specifically, the control function 154 determines the type of character data and voice data used to generate the speaker image and speaker voice based on information such as age and gender included in the attribute data received by the reception function 151. Select (change).

For example, if the user is a child (eg, under 12 years old), control function 154 selects character data and voice data representing friendly characters such as animals. Further, for example, if the user is an adult (eg, 20 years old or older), the control function 154 selects character data and voice data representing a realistic character such as a doctor.

Furthermore, if the content to be made to speak by the object is known in advance, the control function 154 selects (changes) the types of character data and voice data used to generate the speaker image and speaker voice based on the content. .

For example, if a urinary-related disease is suspected and a conversation (announcement) regarding the disease is scheduled, the control function 154 selects character data and voice data representing a character of the same gender as the user. Further, for example, if the user is an adult and the interview is mainly scheduled, the control function 154 selects character data and voice data representing characters in a similar age group so that the user can talk easily. Further, for example, if a medical announcement is mainly scheduled, the control function 154 selects character data and voice data representing a character in an older age group who seems to have a lot of experience.

As a result, the medical support device 10 can generate a speaker image and speaker voice suitable for the user's attributes and the content of the conversation with the user, thereby reducing the user's feelings of anxiety and discomfort toward the speaker. It can be reduced.

Note that the method for implementing the selection process based on attribute data and conversation content is not particularly limited, and various methods can be adopted. For example, by registering applicable attributes (gender, age group, etc.) for each character data (and voice data) registered in the character setting information, it is possible to Selection processing can be performed efficiently. In this case, the control function 154 compares the age, gender, etc. included in the attribute data with the applicable attributes associated with each character data, thereby controlling the character data and voice data according to the user's attributes. can be selected efficiently.

Furthermore, the control function 154 may be configured to select (change) character data and voice data to be used for generating the speaker image and speaker voice based on the history data received by the reception function 151.

For example, the control function 154 extracts animals, people, anime characters, etc. related to products purchased by the user or videos viewed by the user from the purchase history and viewing history included in the history data, and extracts characters similar to those characters. A configuration may be adopted in which character data and voice data are selected. For example, the control function 154 may select character data and voice data from among character data and voice data applied to other users based on other users who have similar tendencies to the purchase history and viewing history of the user. It may be configured to select. As a result, the medical support device 10 can generate a speaker image and speaker voice similar to a character that the user is interested in or a favorite character, so that the user may feel anxious or uncomfortable about the speaker. can be reduced.

Further, for example, the control function 154 may select character data and voice data based on past selection history of character data and voice data included in the history data. The selection method is not particularly limited; for example, the control function 154 may select the most recently selected character data and voice data, or may select the most frequently occurring character data and voice data. As a result, the medical support device 10 can generate a speaker image and a speaker's voice using a character that the user is familiar with or a favorite character. It can be reduced.

Note that although an example has been described above in which the control function 154 changes both character data and voice data, it may also change only one of the data.

Further, although the above description has been given of a mode in which the control function 154 automatically selects the character data and voice data, it may also be a mode in which the user is allowed to select the character data and voice data.

In this case, for example, the control function 154 transmits a list of character data and voice data to the user terminal 20, and cooperates with the display function 282 and the audio output function 283 to create an environment in which character data and voice data can be selected. is realized in the user terminal 20. When the control function 154 receives information indicating the character data and voice data selected by the user from the user terminal 20, it selects the specified character data and voice data. As a result, the medical support device 10 can generate a speaker image and a speaker's voice using a character that the user is interested in or a favorite character, so that the user may feel anxious or uncomfortable about the speaker. can be reduced.

(Expression method change process according to user reaction)
The control function 154 estimates the state of the user from the user's voice data and imaging data received by the reception function 151, and changes the method of expressing the speaker's voice based on the estimation result.

Specifically, the control function 154 estimates the user's emotional state from the user's voice data or imaged data, and changes the method of expressing the speaker image and the speaker's voice according to the estimated emotional state. Here, the emotion estimation method is not particularly limited, and any known technique may be used. For example, the control function 154 may estimate the user's emotional state from a specific wording included in the content of the user's utterance, tone (prosody, intonation), and the like. Furthermore, the control function 154 may estimate the user's emotion based on the size of the pupils of the user's face, the upward angle of the corners of the mouth, the presence or absence of sweating, gestures, etc. of the user's face included in the imaging data.

When the control function 154 estimates the user's emotional state, it changes the expression method, such as the facial expression and gesture of the speaker image, and the tone and intonation of the speaker's voice, according to the state. As an example, the control function 154 uses reaction setting information that associates each emotional state classified based on Russell's circular diagram, Plutchik's circle of emotions, etc. with a speaker image and a method of expressing conversational audio. Based on this, the expression method is determined according to the user's condition. It is assumed that the reaction setting information includes an expression method for alleviating negative emotions such as fear and anxiety.

For example, if the estimated user's emotion indicates a negative emotion such as fear or anxiety, the control function 154 changes the expression of the speaker image generated by the image generation function 152 to a smile or suppresses the amount of movement. By doing so, the expression changes to a gentle or calm expression. In addition, when the estimated user's emotions indicate negative emotions such as fear and anxiety, the control function 154 controls the speech speed and intonation of the speaker's voice generated by the voice generation function 153 to maintain a gentle tone and calm tone. change the tone. Thereby, the medical support device 10 can alleviate the user's negative emotions, thereby facilitating conversation with the user.

Further, for example, when the estimated user's emotion indicates doubt, the control function 154 performs control such as reducing the speaking speed of the speaker's voice. In addition, if the control function 154 detects from the user's voice data or imaged data that a specific phrase (for example, a technical term, etc.) included in the utterance content of the speaker's voice raises a question, the control function 154 displays the corresponding phrase. The voice generation function 153 may generate the speaker's voice for explanation. Thereby, when the user has a question, the medical support device 10 can assist in resolving the question, thereby facilitating conversation.

Furthermore, the control function 154 performs control such as making the speaker's image or voice perform a cooperative action while having a conversation with the user, or making the speaker's image look the user in the eye. conduct. Furthermore, when the control function 154 detects that the user is asking a question to the speaker image based on the content of the user's utterances, the control function 154 increases the frequency of performing the interrogation motion and the amount of time the user looks into the user's eyes. Thereby, the medical support device 10 can appeal to the user that it is listening to questions, and can therefore encourage the user to ask questions.

Note that the method of expressing the speaker image or the speaker's voice may be changed depending on the user's attribute data. For example, if the user is an elderly person (for example, over 60 years old), the control function 154 may set the speaking speed of the speaker's voice lower than that of other age groups, or set the pitch of the voice higher. . As a result, the medical support device 10 can improve the ease of audibility of the speaker's voice, thereby facilitating conversation.

(Processing for changing the expression method according to medical notification matters)
The control function 154 changes the method of expressing the speaker image and the speaker's voice in accordance with the medical notice included in the utterance content of the speaker's voice.

Specifically, prior to the generation of the speaker's voice by the voice generation function 153, the control function 154 analyzes the utterance content of the generated speaker's voice and makes sure that medical announcements such as test and diagnosis results are uttered. Determine whether it is included in the content. If a medical notice is included, the control function 154 changes the method of expressing the speaker image and the speaker's voice depending on the content of the notice.

For example, when announcing the results of a user's test or diagnosis using the speaker's voice generated by the voice generation function 153, the utterance may include medical announcements such as the user's test results, health condition, treatment policy, etc. is included. When such a medical notification item is included in the utterance content, the control function 154 is configured to notify the speaker according to the wording expressing a predetermined event related to the notification item and the degree of influence that the event has on the user. Change the way images and speaker voices are expressed. Here, the event related to the notification item means, for example, the name of a disease such as "cancer" or the name of a treatment method. In addition, "the degree of influence that an event has on the user" refers to the seriousness and seriousness of a disease (disease name), the degree of invasion to the body due to treatment, and the like.

Note that the method of changing the expression according to the medical notification matter is not particularly limited, and various methods can be adopted. In the present embodiment, as an example, a change method using setting information (expression setting information) that associates medical notification matters with expression methods of speaker images and speaker voices will be described.

FIG. 5 is a diagram showing an example of the data structure of expression setting information. As shown in FIG. 5, the expression setting information includes a character string (keyword) that represents an event related to the announcement matter, a character string that represents the degree (level) of influence that the keyword event has on the user, and a speaker image and The expression method of the speaker's voice is associated with and stored. Note that FIG. 4 shows an example in which "stomach cancer" is used as a keyword, and the values of "stage" and "3-year survival rate" correspond to character strings representing levels.

Here, the method of expressing the speaker image and the speaker's voice is set in association with each value of "stage" or "3-year survival rate." Specifically, for each value of "stage" or "3-year survival rate", an expression method is set according to its level (severity). For example, for stages 1 and 2, which are relatively low in severity, an expression method is set to achieve a cheerful expression, a gentle tone, etc. so as not to discourage the user. On the other hand, in stages 3 and 4, which have a high degree of severity, an expression method is set to achieve a calm expression, a calm tone, etc. so as to give the user a sense of security.

Note that the words expressing keywords and levels are not limited to the example shown in FIG. 4. For example, the keyword may be a name representing the type of treatment method, and the level may be a word representing the degree of invasiveness of the treatment method.

The control function 154 determines whether or not the utterance content of the speaker's voice generated by the voice generation function 153 includes the wording representing the above-mentioned keyword and level. For example, when the words "stomach cancer" and "stage 3" are included in the utterance content, the control function 154 reads the expression method corresponding to the set of words from the expression setting information. Then, the control function 154 causes the image generation function 152 and the voice generation function 153 to generate a speaker image and speaker voice for announcement based on the read expression method.

In this way, the medical support device 10 generates a speaker image and a speaker's voice for announcements with different expression methods depending on the medical announcement item. As a result, even if the content to be notified to the user indicates a highly serious disease, the medical support device 10 can generate the speaker image and the speaker's voice in an expression method according to the severity. can do. Therefore, the medical support device 10 can announce the results of examinations and diagnoses, etc. in a way of speaking and interacting with the user that takes into account the user's feelings, thereby reducing the user's anxiety.

Note that the method of changing the representation of the speaker image and the speaker's voice is not limited to the example shown in FIG. 5. For example, when the level of severity is high, such as in stages 3 and 4, the control function 154 may change the content of the utterance so that the name of the disease is not repeated, or express it in a euphemism so that the severity is not emphasized. May be controlled.

In addition, when the user of the user terminal 20 accesses the medical support device 10, if the medical notification matter to be notified to the user is known, the control function 154 controls the character It is also possible to select the type of data and voice data. In this case, for example, when making a highly serious announcement, it is preferable that the control function 154 selects character data and voice data that imitate a realistic character such as a doctor. As a result, the medical support device 10 can notify the user of test results, diagnosis results, etc. using a speaker image and speaker voice suitable for the notification content, so that the user may feel anxious or uncomfortable about the speaker. can be reduced.

In addition, as in the embodiment described later, when the user's diagnosis result becomes known during a conversation with the user, the control function 154 controls the character data and voice according to the content of the medical notice. Control may be performed to switch data types. For example, if an animal character such as a cat is selected at the beginning of the conversation, the control function 154 may perform control to switch to a realistic character such as a doctor at the stage of making a medical announcement to the user. good. Further, for example, if the user attribute is female and the disease related to the notification is a gynecological disease or a urinary disease, the control function 154 may perform control to switch to a character imitating a female doctor. good. As a result, the medical support device 10 can notify the user of test or diagnosis results, etc. using a speaker image and speaker voice suitable for the notification content, so that the user may feel anxious or uncomfortable about the speaker. can be reduced.

The control function 154 is assumed to perform the various processes described above in a complex manner during the conversation with the user, but the control function 154 should give top priority to the process of changing the expression method according to the medical notice. is preferred. In addition, the control function 154 is configured to keep the difference between the expression methods before and after the change within a predetermined range so that the expression methods of the speaker image and the speaker's voice do not change suddenly during a conversation with the user. It is preferable that the image generation function 152 and the sound generation function 153 be controlled so that the image generation function 152 and the sound generation function 153 are within the range of the above.

The transmission function 155 transmits the speaker image (first data) generated by the image generation function 152 and the speaker voice (second data) generated by the voice generation function 153 to the user terminal 20. Here, the transmission function 155 may transmit the first data and the second data in the form of video data so that the movement of the speaker image and the timing of the utterance of the speaker's voice are synchronized.

The first data and second data transmitted from the medical support device 10 are received by the reception function 281 included in the user terminal 20. Then, in the user terminal 20, the display function 282 displays the speaker image represented by the first data on the display 260, and the audio output function 283 outputs the speaker's voice represented by the second data from the audio output interface 270. let Thereby, the user of the user terminal 20 can view the speaker image and the speaker's voice via the user terminal 20.

Furthermore, when the user of the user terminal 20 speaks a question or the like while viewing the speaker image and the speaker's voice, the voice is transmitted to the medical support device 10 along with the captured image of the user. Then, the medical support device 10 changes the representation method of the speaker image and the speaker's voice based on various data transmitted from the user terminal 20 and the contents of the utterance to the user (medical announcements, etc.). Executes data exchange with 20.

Next, an example of the operation of the medical support device 10 will be described with reference to FIG. Here, FIG. 6 is a flowchart showing an example of medical support processing performed by the medical support device 10. Note that, as a premise of this process, it is assumed that the medical support device 10 is in a state where communication is established with the user terminal 20. Further, it is assumed that various data are constantly exchanged between the medical support device 10 and the user terminal 20.

First, the control function 154 selects the speaker's model data and voice data based on the attribute data of the user of the user terminal 20 received by the reception function 151, the contents of the scheduled conversation, etc. (step S11). Next, the image generation function 152 and the voice generation function 153 start generating the speaker image and the speaker voice using the model data and voice data selected by the control function 154 (step S12).

The speaker image and the speaker's voice generated by the image generation function 152 and the voice generation function 153 are provided to the user terminal 20 by being transmitted from the transmission function 155 as first data and second data. The user terminal 20 provides the user with an environment in which the speaker image and the speaker's voice can be viewed by outputting the first data and the second data via the display 260 and the audio output interface 270.

Subsequently, the control function 154 estimates the user's emotional state based on the user's imaging data and audio data received by the receiving function 151 (step S13). The control function 154 changes the method of expressing the speaker image and the speaker's voice generated by the image generation function 152 and the voice generation function 153 according to the estimated emotional state (step S14).

Subsequently, the control function 154 determines whether the user has uttered a utterance based on the user's voice data received by the reception function 151 (step S15). If it is determined that the user has uttered a utterance (step S15; Yes), the control function 154 generates a response speaker image and speaker voice using an expression method according to the content of the utterance, and generates the image generation function 152 and the voice generation function 153. (step S16), and the process moves to step S20.

If it is determined that the user is not speaking (step S15; No), the control function 154 determines whether it is time to make a medical announcement such as a test result or diagnosis result (step S17). The method for determining whether such timing is appropriate is not particularly limited, and various methods can be employed. For example, the control function 154 may determine that it is the timing to make the notification when it detects a phrase requesting test results, diagnosis results, etc. from the content of the user's utterance. Further, for example, when the conversation with the user is interrupted for a predetermined period of time, it may be determined that it is the timing to make the notification.

If it is determined that it is not the timing to make the announcement (step S17; No), the control function 154 returns the process to step S13. On the other hand, if it is determined that it is the timing to make the announcement (step S17; Yes), the process moves to step S18.

Subsequently, the control function 154 extracts a medical keyword included in the notification item such as the test result or diagnosis result and a wording representing the level thereof (step S18). Next, the control function 154 causes the image generation function 152 and the voice generation function 153 to generate a speaker image and speaker voice for announcement using the expression method corresponding to the extracted set of words (step S19), and proceeds to step S20. Transition.

In the following step S20, the control function 154 determines whether the user of the user terminal 20 has instructed to end the conversation. Here, if the end of the conversation is not instructed (step S20; No), the control function 154 returns the process to step S13 and continues the conversation with the user.

Note that when the method of expressing the speaker image and the speaker's voice is changed in the previous step S19, the control function 154 may control the processing from step S13 onwards while maintaining the method of expression. Further, the control function 154 may perform the processing from step S13 onward while suppressing the amount of change within a predetermined range so that the expression method does not change suddenly.

On the other hand, if the control function 154 detects a phrase indicating the end of the conversation (bye-bye, excuse me, etc.) from the utterances of the user of the user terminal 20, it determines that the end of the conversation has been instructed (step S20; Yes). ). In this case, the control function 154 notifies the user of the end of the conversation via the speaker image and the speaker's voice, and then ends this process.

In this way, the medical support device 10 has a conversation with the user of the user terminal 20 via the speaker image and speaker voice generated by the image generation function 152 and the voice generation function 153, and performs medical information to notify the user. The method of expressing the speaker image and the speaker's voice is changed depending on the announcement matter, etc. Thereby, the medical support device 10 can notify the results of examinations and diagnoses, etc. in a way of speaking and interacting with the user that takes into account the user's feelings, so it is possible to reduce the anxiety of the user receiving telemedicine.

In addition, the embodiment mentioned above can also be modified and implemented as appropriate by changing a part of the configuration or function of the medical support device 10. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. Note that, below, points that are different from the embodiments described above will be mainly explained, and detailed explanations of points that are common to those already explained will be omitted. Further, the modified examples described below may be implemented individually or in appropriate combinations.

(Modification 1)
In the above-described embodiment, an example has been described in which data derived from the user terminal 20 (attribute data, history data, etc.) is used as a method for selecting character data and voice data used to generate a speaker image and speaker voice. However, the method of selecting character data and voice data is not limited to this.

For example, the control function 154 of this modification controls the image generation function 152 so that some character data is displayed on the user terminal 20, such as when starting a conversation. Then, the control function 154 estimates the user's reaction to the character data from the user's imaging data and voice data, and selects character data (voice data) that shows a positive reaction.

As a result, the medical support device 10 can generate a speaker image and a speaker's voice from a character that the user is interested in or a favorite character without requiring any direct operation by the user. It is possible to reduce the anxiety and discomfort that people feel toward the speaker.

(Modification 2)
In the embodiment described above, an example was explained in which a character image was used as an object representing a speaker. However, the format of the object is not limited to this. For example, a communication robot capable of conversation and movement may be used as an object representing a speaker.

In this case, the communication robot is installed at the user's home or the like and operates on the user terminal 20 itself or in cooperation with the user terminal 20. Furthermore, instead of generating a speaker image, the image generation function 152 of the medical support device 10 operates the communication robot by generating a control signal for operating the communication robot as first data. Furthermore, the control function 154 of the medical support device 10 selects the type of voice data to be used to generate the speaker's voice based on the user's attribute data, the content of the scheduled conversation, and the like.

Also in this modification, the control function 154 controls the image generation function 152 in the same way as in the above-described embodiment, and changes the method of expressing the motion of the object, that is, the motion of the communication robot. Thereby, in this modification 1 as well, it is possible to achieve the same effects as in the above-described embodiment.

(Modification 3)
In the embodiments described above, the medical support device 10 is realized by computer equipment such as a workstation or a server device. However, the method for implementing the medical support device 10 is not limited to this. For example, the medical support device 10 may be realized by a plurality of computer devices (cloud) using cloud computing.

[Second embodiment]
Next, a second embodiment will be described. In the first embodiment, a case has been described in which the user's test results or diagnosis results are prepared in advance. In the second embodiment, a case will be described in which the medical support device 10 automatically diagnoses the health condition of the user of the user terminal 20. In the following, the second embodiment will be mainly described with a focus on processing that is different from the first embodiment.

FIG. 7 is a diagram showing a configuration example of the medical support device 10 according to the present embodiment. As shown in FIG. 7, the medical support device 10 includes a communication interface 110, a storage circuit 120, an input interface 130, a display 140, and a processing circuit 150. Further, the processing circuit 150 has a diagnostic function 156 in addition to the above-mentioned receiving function 151, image generating function 152, sound generating function 153, control function 154, and transmitting function 155.

Here, for example, each processing function executed by the reception function 151, image generation function 152, sound generation function 153, control function 154, transmission function 155, and diagnosis function 156, which are the components of the processing circuit 150, can be executed by a computer. The program is stored in the storage circuit 120 in the form of a program. The processing circuit 150 reads each program from the storage circuit 120 and executes it, thereby realizing a function corresponding to each program. In other words, the processing circuit 150 in a state where each program is read has each function shown in the processing circuit 150.

The diagnostic function 156 has a function of automatically diagnosing the health condition of the user of the user terminal 20 based on various data transmitted from the user terminal 20. Here, the diagnostic method of the diagnostic function 156 is not particularly limited, and any known technique can be used. For example, the diagnostic function 156 may provide the user with questions related to diagnosing the health condition through the speaker's voice or the like, and may diagnose the user's health condition from the answers to the questions included in the speaker's voice or the like. Furthermore, for example, the diagnostic function 156 may diagnose the user's health condition from imaging data obtained by imaging the user. In this case, the diagnostic function 156 can diagnose the state of the wound, for example, from the image of the user's wound included in the imaging data.

Furthermore, when the diagnostic function 156 diagnoses the user's health condition, the control function 154 of this embodiment uses a method similar to that of the first embodiment described above in accordance with the medical notice included in the diagnosis result. , change the representation method of the speaker image and the speaker's voice.

For example, when questions and answers related to the diagnosis of a health condition are exchanged during the processing from step S13 to step S16 in FIG. Announcement will be made.

In the case of the present embodiment, the control function 154 performs control to switch the types of speaker images and speaker voices to be presented to the user when the diagnosis function 156 diagnoses the health condition and when notifying the user of the diagnosis results. Good too. For example, the control function 154 may present a speaker image representing a virtual character such as a robot to the user while diagnosing a health condition, and may present a speaker image representing a virtual character such as a robot to the user, and may display a realistic image of a doctor or You may also switch to a speaker image representing a character.

As a result, as in the first embodiment, the medical support device 10 can announce the diagnosis result in a manner of speaking and interacting with the user that takes into consideration the user's feelings, thereby reducing the anxiety of the user receiving telemedicine. can.

Note that in the embodiment and modification described above, an example has been described in which the functional configuration of the medical information processing apparatus 100 is realized by the processing circuit 280, but the embodiment is not limited to this. For example, the functional configurations in this specification may be implemented using only hardware or a combination of hardware and software.

Furthermore, the term "processor" used in the above explanation refers to, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device. (For example, it refers to circuits such as Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA)). The processor realizes functions by reading and executing programs stored in the storage circuit 220. Note that instead of storing the program in the storage circuit 220, the program may be directly incorporated into the circuit of the processor. In this case, the processor implements its functions by reading and executing a program built into the circuit. Further, the processor of this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining a plurality of independent circuits to realize its functions.

Here, the program executed by the processor is provided by being pre-installed in a ROM (Read Only Memory), a storage circuit, or the like. This program is a file in a format that can be installed or executable on these devices, such as CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), DVD (Digital Versatile Disk), etc. It may be provided recorded on a computer readable storage medium. Further, this program may be stored on a computer connected to a network such as the Internet, and may be provided or distributed by being downloaded via the network. For example, this program is composed of modules including each of the above-mentioned functional units. In actual hardware, a CPU reads a program from a storage medium such as a ROM and executes it, so that each module is loaded onto the main storage device and generated on the main storage device.

According to the embodiment described above, it is possible to alleviate the anxiety of a user receiving telemedicine.

Although several embodiments (modifications) of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

1 Telemedicine system 10 Medical support device 20 User terminal 151 Receiving function 152 Image generation function 153 Voice generation function 154 Control function 155 Transmission function 156 Diagnosis function

Claims (11)

a first generation unit that generates first data for operating an object representing a speaker who has a conversation with a user;
a second generation unit that generates second data representing a voice to be uttered by the object;
The motion of the object and /or the method of expressing the voice may be changed in accordance with the emotional state of the user estimated based on the medical notice included in the utterance content of the speaker or the reaction of the user. a control unit that controls ;
Equipped with
The control unit is a medical support device that prioritizes the announcement item and the emotional state, and controls to change the motion of the object and/or the method of expressing the voice according to the priority. The control unit changes the motion of the object and /or the method of expressing the voice according to a predetermined event related to the announcement item included in the utterance content and the degree of influence of the event on the user. The medical support device according to claim 1, wherein the medical support device is controlled as follows . The control unit estimates the emotional state of the user from the user's utterance content obtained by a sound collection device or the captured image of the user obtained by an imaging device, and according to the estimated emotional state of the user. The medical support device according to claim 1 or 2, wherein the medical support device controls to change the motion of the object and /or the method of expressing the sound. The medical support device according to claim 1, wherein the control unit controls to change the motion of the object and/or the method of expressing the sound to be the highest priority, depending on the notification item. The medical support device according to claim 1, wherein the control unit changes the type of the object or the type of the sound based on attributes of the user. The medical support device according to claim 1, wherein the control unit changes the type of the object or the type of the voice based on content to be made to speak by the object. The medical support device according to claim 1, wherein the control unit changes the type of the object or the type of the sound based on the user's action history. When the type of the object or the type of the audio is specified by the user, the control unit controls the first data and /or the second data using the object or the audio of the specified type. The medical support device according to claim 1, wherein the medical support device generates. The first generation unit generates, as the first data, a speaker image in which the character image, which is the object, is moved;
The medical support device according to any one of claims 1 to 8 , wherein the control unit changes the behavior of the speaker image by controlling the first generation unit. The first generation unit generates the first data for operating the communication robot that is the object,
The medical support device according to any one of claims 1 to 8 , wherein the control unit changes the operation of the communication robot by controlling the first generation unit. In the computer of medical support equipment,
a first generation function that generates first data for operating an object representing a speaker who has a conversation with a user;
a second generation function that generates second data representing a voice to be uttered by the object;
The motion of the object and /or the method of expressing the voice may be changed in accordance with the emotional state of the user estimated based on the medical notice included in the utterance content of the speaker or the user's reaction. a control function to control;
The control function is characterized in that the notification item and the emotional state are prioritized, and control is performed to change the movement of the object and/or the method of expressing the voice according to the priority. program.

Images