JP7488591B2 - System for modifying images based on audio - Patents.com - Google Patents

Description

This invention relates to a system for changing an image based on voice. More specifically, this invention relates to a system in which a shared image is changed in real time based on a conversation.

A typical web conferencing system displays presentation materials prepared by the speaker on each device. Even if participants make comments related to the materials, no changes are made to the materials.

JP 2020-089641 A describes a system that performs command operations based on voice. Systems that recognize voice in this way and perform various operations are well known. However, no system is known that changes a displayed image in real time based on two-way conversation.

JP 2020-089641 A

The aim of this invention is to provide a system in which shared images are changed in real time, for example based on conversation.

This invention is based on the knowledge that a shared image can be changed in real time based on a conversation by using information about the content contained in the audio information and information about changes to the content to change the position of the content in an image that represents the content.

The system 1 includes a voice information input unit 3, a voice analysis unit 5, and an image modification unit 7.
The voice information input unit 3 is an element for inputting voice information.
The audio analysis unit 5 is an element for analyzing the audio information input by the audio information input unit 3. The image change unit 7 is an element for changing the position of an image representing a content, using information on the content included in the audio information analyzed by the audio analysis unit 5 and information on changes in the content included in the audio information analyzed by the audio analysis unit 5.

The image change unit 7 of this system 1 may change either the shape or the color of the content whose position has been changed, or both.

A preferred embodiment of this system 1 further includes a first operation input unit 21 for inputting first operation information which is operation information of a first participant, and a second operation input unit 23 for inputting second operation information which is operation information of a second participant.
The image changing unit 7 changes the position of the content in the image based on the information on the content included in the voice information analyzed by the voice analysis unit 5, the information on the variation of the content analyzed by the voice analysis unit 5, the first operation information, and the second operation information. The image changing unit 7 may change either the shape or the color of the content whose position has been changed, or both.

The system 1 provides a method for modifying an image based on audio using a system having a computer.
This method includes a voice information input step of inputting voice information into the system 1;
The system 1 includes a voice analysis step for analyzing voice information inputted by the voice information input step;
The system 1 includes an image changing step of changing the position of the content in an image representing the content, using information about the content included in the audio information analyzed in the audio analysis step and information about the change in the content.

This specification provides a program for causing a computer to function as the above system, and an information recording medium on which the program is stored.

This invention can provide, for example, a system that can change shared images in real time based on conversations.

FIG. 1 is a block diagram illustrating a system for modifying an image based on audio. FIG. 2 is a block diagram showing the basic configuration of a computer. FIG. 3 is a conceptual diagram showing an example of a system according to the present invention. FIG. 4 shows an example of a presentation material. FIG. 5 is a conceptual diagram showing an image after the position of the content in the image representing the content has been changed by the image change section. FIG. 6 is a conceptual diagram showing an image after the position, shape and color of the content in the image representing the content have been changed by the image change section. FIG. 7 is a conceptual diagram showing an image after the position of the content in the image representing the content has been changed based on operation information.

Below, the embodiments for carrying out the present invention will be explained using the drawings. The present invention is not limited to the embodiments described below, but also includes appropriate modifications of the embodiments below within the scope that would be obvious to a person skilled in the art.

In the following, a system 1 for modifying an image based on audio is described, which is a computer-based system.
Fig. 1 is a block diagram for explaining a system for changing an image based on audio. As shown in Fig. 1, this system 1 has an audio information input unit 3, an audio analysis unit 5, and an image change unit 7. This system 1 may further have a presentation material storage unit 11 that stores presentation materials and the like, a presentation output unit 13 that outputs presentation materials, an operation information input unit 15 to which various types of operation information are input, and an audio output unit 17 that outputs audio information.

Figure 2 is a block diagram showing the basic configuration of a computer. As shown in this figure, the computer has an input unit 31, an output unit 33, a control unit 35, a calculation unit 37, and a storage unit 39, and each element is connected by a bus 41 or the like so as to be able to exchange information. For example, the storage unit may store a control program or various information. When specific information is input from the input unit, the control unit reads out the control program stored in the storage unit. Then, the control unit appropriately reads out the information stored in the storage unit and transmits it to the calculation unit. The control unit also appropriately transmits the input information to the calculation unit. The calculation unit performs calculation processing using the various information received and stores it in the storage unit. The control unit reads out the calculation results stored in the storage unit and outputs them from the output unit. In this way, various processes are performed. Each element described below may correspond to any element of the computer.

Figure 3 is a conceptual diagram showing an example of a system of the present invention. As shown in Figure 3, the system of the present invention (a system including the device of the present invention) may include a terminal 45 connected to the Internet or an intranet 43, and a server 47 connected to the Internet or an intranet 43. Of course, a single computer or mobile terminal may function as the device of the present invention, and multiple servers may exist.

The voice information input unit 3 is an element (e.g., a microphone) for inputting voice information. An example will be described in which the system is composed of multiple terminals and a server connected to the multiple terminals. In this case, the voice information input unit 3 is, for example, an input unit in the server that inputs voice information input from an input unit (e.g., a microphone) of each terminal. In this case, the input unit in the server may function as the voice information input unit 3, or the input unit of each terminal may function as the voice information input unit 3. In either case, the voice information is input into the server.

The following explanation is based on an example where a medical representative (MR) and a medical professional have a conversation via a web conference via their respective computer terminals (terminals). Conference access information is input in advance into the MR's terminal and the medical professional's terminal. Examples of conference access information include information that identifies the conference (URL), a conference ID, and a conference password.

The MR terminal 45a and the medical staff terminal 45b use the conference access information to access the web conference server. When the conference access information is input to the MR terminal 45a and the medical staff terminal 45b, these terminals access the conference server. The server has an input section and an output section, so that each image and audio information can be exchanged via the server. In this example, the server has a presentation material storage section 11. In this example, each terminal of the participants in the web conference accesses the server and has a conversation based on image data stored in the server, and the image data is updated to reflect the conversation.

For example, the MR uses an input device (mouse) to read out the presentation materials to be discussed from the presentation material storage unit 21. The read presentation materials are output from the output unit of the server to the MR's terminal 45a and the medical staff's terminal 45b. The presentation materials are input to the MR's terminal 45a and the medical staff's terminal 45b. The presentation materials are displayed on monitors, which are the output units of the MR's terminal 45a and the medical staff's terminal 45b. Note that conversation information between the MR or medical staff may be input to the server, the conversation information may be analyzed, and the presentation materials to be discussed may be read out from the presentation material storage unit 11 based on the analyzed conversation information. In this case, the presentation material storage unit 11 may store a readout word associated with each presentation material, and if the analyzed conversation information contains a readout word, the presentation materials associated with that readout word may be read out.

Figure 4 shows an example of a presentation. In the example in Figure 4, the text reads, "What is your impression of the effectiveness and safety of the following antihypertensive drugs?" The figure shows a graph with the vertical axis showing the drug's effectiveness (efficacy) and the horizontal axis showing safety.

Through the MR's terminal 45a and the medical staff's terminal 45b, the two parties engage in various conversations. For example, the MR might say, "The diagram shows the efficacy and safety of drugs A to D. What do you think, doctor?" Then, voice information is input to the MR's terminal 45a from the microphone of the MR's terminal 45a. The input voice information is temporarily stored in the memory of the MR's terminal 45a. The voice information is then output from the output section of the MR's terminal 45a to the server 47. The server 47 receives the voice information, which is then input to the server 47 from the server's input section and stored in the memory section of the server 47. In this way, voice information is input to the system 1 (voice information input process).

Suppose a medical professional were to say, for example, "Drug C is more effective than drug B and is a little safer." Then, in the same manner as above, voice information is input to system 1 (voice information input process).

The voice analysis unit 5 is an element for analyzing the voice information inputted by the voice information input unit 3 (for example, a known voice recognition unit).
For example, the server's memory unit stores content (e.g., drug A, drug B, drug C, and drug D) and the position of each content in the image (the coordinates of drug A, drug B, drug C, and drug D) in relation to the presentation materials. The voice analysis unit 5 then obtains information about the content "drug C" and "drug B" as analysis results. The voice analysis unit 5 also obtains information about changes in content, such as "drug C is more effective than drug B," as analysis results. The voice analysis unit 5 also obtains information about changes in content, such as "drug C is a little safer," as analysis results.
In this way, the system 1 analyzes the voice information input in the voice information input step, and obtains an analysis result (voice analysis step). The obtained analysis result is appropriately stored in the memory unit of the server. The analysis result may be output from the output unit of the server to the MR terminal 45a or the medical staff terminal 45b, and stored in the memory unit of these terminals. This process is the same below.

The image change unit 7 changes the position of the content in the image representing the content, using information about the content included in the voice information analyzed by the voice analysis unit 5 and information about the change in the content included in the voice information analyzed by the voice analysis unit 5. For example, the image change unit 7 may specify the content to be changed based on information about the content, and perform processing to change the content based on information about the change in the content. The changed image is displayed on the display unit. For example, the server reads out information about the content "Drug C" and "Drug B", which are the analysis results, from the storage unit of the server, information about the change in the content, "Drug C is more effective than Drug B", and information about the change in the content, "Drug C is a little safer". Then, the position of Drug C in the image of the content is changed using the read analysis results. The image with the changed position of the content is stored in the storage unit of the server. Then, it may be output to the MR terminal 45a and the medical staff terminal 45b. Then, the changed image is displayed on the output unit (monitor) of the MR terminal 45a and the medical staff terminal 45b. The image change unit 7 may identify the person who made the sound (speaker) regarding the audio information, and adjust the amount of change according to the identified speaker. The speaker may be identified when inputting the terminal ID or web conference. The amount of change may be adjusted by storing a coefficient used for the adjustment in association with the speaker's ID and reading it out according to the speaker's ID. In this way, changes according to the speaker's personality can be made automatically (without the speaker realizing). Changing the shape or color of the content without changing the position of the content is another embodiment described in this specification.

Figure 5 is a conceptual diagram showing an image after the position of the content in the image representing the content has been changed by the image changing unit. In this way, the system 1 changes the position of the content in the image representing the content based on the information about the content contained in the audio information analyzed in the audio analysis process and the information about the change in the content (image changing process).

After looking at Figure 5, the doctor uses an input device (e.g., a mouse or touch panel) on the medical staff terminal 45b to drag drug C in the image. The doctor then says, "I think drug C is around here." This statement is entered into the system, as explained above. Furthermore, operation information from the medical staff terminal 45b is output from the output section of the medical staff terminal 45b to the server 37. Having received the operation information, the server 37 inputs the operation information into the server 37 from an input section of the server 37. The server 37 then changes the position of drug C in the content image based on the operation information.

Figure 6 is a conceptual diagram showing an image after the position, shape, and color of the content in the image representing the content have been changed by the image change unit. As shown in Figure 6, the image change unit 7 may change either or both of the shape and color of the content so that the content whose position has been changed can be identified. The shape may be changed by changing the size, making it disappear, making it a different content image, or changing the shape of the frame. As will be described later, either or both of the shape and color of the content after the change may be different between when the position of the content has been changed based on operation information and when the position of the content has been changed based on voice.

Figure 7 is a conceptual diagram showing an image after the position of the content in the image representing the content has changed based on the operation information. In this way, this system may change the position of the content based on the operation information. The image with the changed position of the content is stored in the storage unit of the server. It may then be output to the MR terminal 45a and the medical staff terminal 45b. Then, the changed image is displayed on the output unit (e.g., monitor) of the MR terminal 45a and the medical staff terminal 45b. A preferred embodiment of this system 1 further includes a first operation input unit 21 for inputting the first operation information, which is the operation information of the first participant, and a second operation input unit 23 for inputting the second operation information, which is the operation information of the second participant. In the above example, for example, the input device of the MR terminal 45a functions as the first operation input unit 21. Also, the input device of the medical staff terminal 45b functions as the second operation input unit 23. For example, the MR terminal 45a receives input information that the drug C has been dragged to the right by the mouse. The received input information is transmitted to the server as operation information. The server that receives the operation information causes the image change unit 7 to change the image based on the operation information. The changed image is stored in the server's storage unit. The stored image is output to the MR terminal 45a and the medical staff terminal 45b. Then, the MR terminal 45a and the medical staff terminal 45b display the received image on their respective display units. The same applies to the operation information by the medical staff terminal 45b. In this way, the image change unit 7 changes the position of the content in the image based on information about the content contained in the voice information analyzed by the voice analysis unit 5, information about the fluctuation of the content analyzed by the voice analysis unit, the first operation information, and the second operation information. The above is an example of operation information that changes the position. If the operation information changes the shape or color of the content, the image change unit 7 may change either or both of the shape and color of the content whose position has been changed.

This specification provides a program for causing a computer to function as the above-mentioned system, and an information recording medium storing the program. This program is implemented in a computer and causes the computer to function as a system for changing an image based on audio, the system having audio information input means for inputting audio information, audio analysis means for analyzing the audio information input by the audio information input means, and image change means for changing the position of the content in an image representing the content, using information about the content included in the audio information analyzed by the audio analysis means and information about changes in the content.

This invention can be used in voice recognition systems and can therefore be utilized in the information industry.

1 System 3 Voice information input unit 5 Voice analysis unit 7 Image change unit

Claims (4)

A voice information input unit (3) for inputting voice information;
a voice analysis unit (5) for analyzing the voice information input by the voice information input unit (3);
an operation input unit (21) for manually inputting operation information;
and an image change unit (7) that uses information about the content included in the audio information analyzed by the audio analysis unit (5) and information about a change in the content to identify the content and change either a shape or a color of the content in an image representing the content,
The image changing unit (7) can change the position of the content based on the operation information inputted by the operation input unit (21),
The image change unit (7) can identify a speaker who has issued the audio information, and can adjust the amount of change in either or both of the shape and color of the content depending on the speaker;
The image changing unit (7) can change the position of the content by using information about the change in the content, and when the position of the content is changed, changes either or both of the shape and color of the content whose position has been changed;
changing the position of the content based on the operation information and changing the position of the content using the voice information to make one or both of a shape and a color of the content after the change different from each other;
A system for modifying an image based on audio (1). 1. A method for modifying an image based on audio using a system having a computer, comprising:
a voice information input step of inputting voice information into the system;
the system includes a voice analysis step for analyzing the voice information inputted by the voice information input step;
the system includes an image changing step of identifying the content using information on the content included in the audio information analyzed in the audio analyzing step and information on a change in the content, and changing either a shape or a color of the content, or both, in an image representing the content;
The image changing step includes identifying a speaker who has issued the audio information, and adjusting the amount of change in either or both of a shape and a color of the content in accordance with the speaker;
The image changing step can change a position of the content by using information about the change in the content, and when the position of the content is changed, one or both of a shape and a color of the content whose position has been changed is changed ,
The image changing step changes one or both of a shape and a color of the content after the change between a case where the position of the content is changed based on manual operation information and a case where the position of the content is changed using the audio information,
Method. Computer,
A voice information input means for inputting voice information;
a voice analysis means for analyzing the voice information input by the voice information input means;
an operation input means for manually inputting operation information;
an image change means for specifying the content by using information on the content included in the audio information analyzed by the audio analysis means and information on a change in the content, and for changing either a shape or a color of the content in an image representing the content,
the image changing means is capable of changing a position of the content based on the operation information inputted by the operation input means,
the image change means identifies a speaker who has issued the audio information, and adjusts the amount of change in either or both of the shape and color of the content in accordance with the speaker;
The image changing means can change a position of the content by using information about the change in the content, and when the position of the content is changed, changes either or both of a shape and a color of the content whose position has been changed;
changing the position of the content based on the operation information and changing the position of the content using the voice information to make one or both of a shape and a color of the content after the change different from each other;
A program for functioning as a system for changing images based on audio. An information recording medium storing the program according to claim 3.

Images