JP6182464B2 - Image display system and image display method - Google Patents

Description

  The present invention relates to an image display system and an image display method for displaying an image of a speaker to a conversation partner, and more particularly to an image display system and an image display method capable of realizing a more realistic dialogue.

  Communication techniques for remote parties such as a teleconference system for performing a conversation while watching each other's images are already well known. In recent years, in order to realize a dialogue with a sense of presence, a technique has been developed that intentionally changes the line of sight of a conversation person in a conversation person image displayed on a display screen (for example, Patent Document 1). reference).

  In the image display system described in Patent Document 1, when one of the conversation persons A is viewing the image of the other conversation person B on the display when the conversation persons are performing a conversation while watching each other's images. Based on the detection result, the video of the conversation person A is changed by changing the shape and position of the pupil in the video of the conversation person A that the conversation person B is watching on the display. . With such an arrangement, in the image display system described in Patent Document 1, for example, when the conversation person A interacts with a plurality of conversation persons B, the line of sight is directed to a specific conversation person B. Can be perceived by each dialogue person B.

JP 2012-70081 A

  By the way, further improvement of a sense of reality is demanded for the dialogue performed by the interlocutors while looking at each other's images. In order to satisfy such a requirement, it is possible to apply a technology for changing the line of sight in the video of the interlocutor as in the image display system described in Patent Document 1 and make it feel as if the user is interacting with each other. System and method must be developed.

  Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to provide a more realistic dialogue as an image display system and method for displaying a speaker's image to a dialogue partner. It is an object to provide a system and method capable of realizing the above.

According to the image display system of the present invention, the subject is an image display system for displaying a speaker's image to a conversation partner, and is provided on the speaker side, and an image when the speaker is captured A data acquisition unit that acquires data indicating a display, a display data generation unit that generates display data for displaying the image corrected from the video, and the display partner that expands the display data An image display unit for displaying the image, and a detection unit that is provided on the conversation partner side and detects the conversation partner when the conversation partner performs an operation that satisfies a preset condition , wherein when the front Symbol detection unit detects the dialogue partner were performing the operation, the display data generation unit, which has been corrected so that the face is the speaker of the line-of-sight to the dialogue partner is present position, which was the operating Display images Run the first process of generating the display data of the eye, the when the detection unit does not detect the dialogue partner for a predetermined time of performing the operation, the display data generating unit, the talk in the predetermined position This is solved by executing a second process for generating the display data for displaying the image corrected so that the user's line of sight faces .

As described above, in the image display system of the present invention, when the conversation partner performs a predetermined operation, the detection unit detects this. Then, the display data generation unit generates display data of the image corrected so that the line of sight of the speaker is directed to the position where the conversation partner who has performed the above operation exists in conjunction with the detection operation. Thus, when an image of the speaker is displayed on the conversation partner side, the line of sight of the speaker in the image is directed toward the conversation partner. This change in line of sight gives the conversation partner the illusion that the speaker has responded to the above actions, and as a result, the conversation partner feels as if they are interacting with the speaker. It becomes possible.
Further, according to the above configuration, in the non-detection period in which the conversation partner performing the predetermined operation is not detected, the line of sight of the speaker is preset in the speaker image displayed for the conversation partner. It comes to the position. This makes it possible to suitably set the speaker's line of sight during the non-detection period.

In the image display system, when the display unit detects the conversation partner that performed the operation, the display data generation unit detects the speaker's line of sight and the position of the conversation partner who performed the operation. The display data for displaying the image corrected so that the face of the speaker faces may be generated.
According to the above configuration, in the speaker image displayed to the conversation partner, the speaker's line of sight and the speaker's face are directed to the position where the conversation partner who has performed the predetermined action is present, so that Realistic dialogue can be realized. In other words, not only the line of sight of the speaker but also the speaker's face is directed to the conversation partner who performed the above action, so that the conversation partner feels as if interacting with the speaker (face-to-face) However, it will increase further.

In particular, when there are a plurality of conversation partners, when the detection unit does not detect the conversation partner that has performed the operation for a predetermined time or more, the display data generation unit is the speaker among the plurality of conversation partners. It is preferable that the second process is repeatedly executed so that the conversation partner at the position where the line of sight is facing sequentially switches.
According to the above configuration, during the non-detection period, the display data generation unit repeatedly executes the second process so that the conversation partners at the position where the speaker's line of sight is sequentially switched. That is, in the speaker's image displayed to the conversation partner, the speaker's line of sight shows a movement in which the conversation partner positioned ahead changes sequentially. This makes it possible for each conversation partner to feel as if the speaker is looking around, thereby improving the preference of image display in the conversation.

Further, in the above image display system, a display screen forming unit that forms a part of a building material, furniture, or decoration arranged in a building where the conversation partner is located and forms a display screen of the image; Provided at the time of detecting at least one of an action performed by the conversation partner, a position where the conversation partner is located, a posture of the conversation partner, and a sound emitted by the conversation partner. A sensor that detects the detection target that satisfies the condition, and the display screen forming unit displays the display screen during a period in which the sensor does not detect the detection target that satisfies the second condition. It is preferable that the display screen is formed only during a period in which the appearance as the part is displayed without being formed and the sensor detects the detection target that satisfies the second condition.
According to the above configuration, an image display screen is formed by using at least one of the operation, position, posture and sound of the person on the conversation partner side as a trigger. In addition, the display screen is not formed during the period when the detection target that is the trigger is not detected, and instead, the appearance as a part of the building material, furniture, or decoration arranged in the building where the conversation partner is located is displayed. become. As a result, the display screen forming unit that forms the display screen functions as a building material or the like during a period in which no dialogue with the speaker is performed, making it difficult to stand out in the building. As a result, it is difficult to notice the presence of the display screen forming unit during the same period. On the other hand, as described above, when a detection target is detected on the conversation partner side, a display screen is formed using this as a trigger, so that no complicated operation is required for forming the display screen.

Further, the above-described problem is an image display method for displaying a speaker's image to a conversation partner according to the image display method of the present invention, wherein the data acquisition unit provided on the speaker side Obtaining data indicating a video when a speaker is imaged, a display data generation unit generating display data for displaying the image corrected from the video, and provided on the conversation partner side The displayed image display unit displays the image by expanding the display data, and the detecting unit provided on the dialog partner side performs an operation satisfying a preset condition. Detecting the conversation partner at the time, and when the detection unit detects the conversation partner that performed the operation , the display data generation unit is a position where the conversation partner who performed the operation is located. The speaker ’s line of sight Run the first process of generating the display data for displaying the corrected the image, the when the detection unit said no interaction partner is detected over a predetermined time of performing the operation, the display data generating unit This is solved by executing a second process of generating the display data for displaying the image corrected so that the line of sight of the speaker is directed to a preset position .

According to the image display system and the image display method of the present invention, when the conversation partner performs a predetermined operation, the image of the speaker displayed to the conversation partner is corrected, and the speaker's line of sight performs the above operation. It will be directed to the position where the conversation partner is. This makes it possible to realize a more realistic dialogue. That is, according to the present invention, when a speaker and a conversation partner at a distant place interact with each other while watching each other's images, the conversation partner is as if they are interacting with the speaker in an audiovisual effect. Can be obtained.
Further, in a non-detection period in which a conversation partner performing a predetermined operation is not detected, the speaker's line of sight is directed to a preset position in the speaker image displayed for the conversation partner. This makes it possible to suitably set the speaker's line of sight during the non-detection period.

It is the figure which showed a mode that the image of a speaker is displayed on the dialogue other party side. It is the figure which showed a mode that the image of the other party was displayed on the speaker side. 1 is a conceptual diagram illustrating an image display system according to an embodiment of the present invention. 1 is a block diagram illustrating a configuration of an image display system according to an embodiment of the present invention. 4A and 4B are diagrams showing an example of the display screen forming unit of the present invention. It is the figure which showed each structure of the speaker side server and the other party server from the functional surface. It is the figure which showed the flow of the 1st process of this invention. It is explanatory drawing regarding specification of an operation person position. It is explanatory drawing regarding decomposition | disassembly of a speaker image | video. It is explanatory drawing regarding a line-of-sight edit. It is the figure which showed a mode that the speaker image after correction | amendment is displayed. It is explanatory drawing regarding face orientation editing. It is the figure which showed a mode that the speaker image after re-correction is displayed. It is the figure which showed the flow of the 2nd process of this invention. (A), (B), and (C) of FIG. 14 are views showing a state in which a speaker image is displayed based on display data generated by the second process. It is the figure which showed the flow of the data processing which the image display system which concerns on one Embodiment of this invention performs (the 1). It is the figure which showed the flow of the data processing which the image display system which concerns on one Embodiment of this invention performs (the 2). It is the figure which showed the procedure at the time of displaying a speaker image.

  Hereinafter, an image display system and an image display method according to an embodiment of the present invention (hereinafter, this embodiment) will be described with reference to the drawings. In order to make the description easy to understand, a case where the speaker is Mr. A and the conversation partners are Mr. B, Mr. C, and Mr. D will be described below as a specific example. Here, the “speaker” is a person who starts a conversation (conversation) based on his own idea and talks to the conversation partner. On the other hand, a “dialogue partner” is a speaker who is a listener of a speaker's story and performs a dialogue in response to the speaker's story.

  In the following description, Mr. A who is a speaker is in a predetermined building (for example, Mr. A's home) at the time of conversation, and Mr. B, Mr. C, and Mr. D who are conversation partners are with Mr. A. An example will be described in which a person is in a different place (for example, in a different building from Mr. A's home) and all three people gather at the same place and interact with Mr. A.

<< Configuration of Image Display System According to Present Embodiment >>
The image display system (present system S) according to the present embodiment is used for a conversation between Mr. A who is a speaker and Mr. B, C, and D who are conversation partners while viewing each other's images. The That is, by using this system S, Mr. B, Mr. C, and Mr. D can interact while viewing the image of Mr. A as shown in FIG. 1A. Similarly, Mr. A can interact while viewing the images of Mr. B, Mr. C, and Mr. D as shown in FIG. 1B. Here, FIG. 1A is a diagram showing a state in which an image of a speaker is displayed on the conversation partner side, and FIG. 1B is a diagram showing a state in which an image of the conversation partner is displayed on the speaker side. It is.

  Referring to FIGS. 1A and 1B, the images of the speaker and the conversation partner displayed in the present embodiment will be described in more detail with reference to FIGS. 1A and 1B. As shown in FIG. It is projected on the display installed in the building where there is, and the whole body image of Mr. A and the surrounding space are to be displayed. Similarly, the images of Mr. B, Mr. C, and Mr. D who are conversation partners are displayed on the display installed at Mr. A's home, and the whole body image of the above three people and the surrounding space are displayed. ing.

  As described above, the speaker and the conversation partner can interact with each other while looking at each other's whole body image and the surrounding space at a location apart from each other. As a result, both the speaker and the conversation partner can interact while feeling as if they are in the same room, so that a realistic conversation can be realized. The “whole body image” is the appearance of the entire body from the head to the foot, and may be in a standing state or a sitting state, and partly hidden by an object placed in front. It is a concept that includes the state of being.

  And in this system S, in order to implement | achieve the realistic dialogue, as shown in FIG. 2, the communication units 1 and 2 are provided in the A side and the B side, respectively. FIG. 2 is a conceptual diagram of the system S. As shown in the figure, the communication units 1 and 2 are capable of data communication through a communication line 3 such as the Internet. Although not shown in FIG. 2, a relay server (proxy server) is generally interposed between the communication units 1 and 2. That is, data transmitted / received between the communication units 1 and 2 usually passes through the relay server.

  Hereinafter, the configuration of the communication units 1 and 2 will be described. First, the communication unit 1 provided on the speaker side (Mr. A side) will be described. As shown in FIG. 3, a server computer (hereinafter referred to as a speaker side server) 10A provided on the speaker side, audiovisual equipment, etc. It is constituted by. FIG. 3 is a block diagram showing a configuration of the system S including the communication units 1 and 2. As the audiovisual equipment, a sound collecting device 21, an imaging device 22, an audio reproducing device 24, and a display (strictly, a display / mirror 25 described later) are provided. Furthermore, the communication unit 1 according to the present embodiment includes a human sensor 23 as a sensor for detecting a speaker in front of the display.

  The speaker-side server 10A is a device that plays a central role in the communication unit 1, and as shown in FIG. 3, a CPU 11, a memory 12 composed of ROM and RAM, a hard disk drive 13 (denoted as HDD in FIG. 3), a communication interface. 14 (I / F for communication in FIG. 3) and an I / O port 15. The speaker-side server 10A receives data transmitted from an external device (for example, a partner-side server 10B described later) connected to the communication line 3, and stores the data in the memory 12 or the hard disk drive 13. Further, the memory 12 stores a program (hereinafter referred to as a dialogue program) that defines a series of data processing executed when the speaker A-san interacts with Mr. B-san. When this interactive program is read out and executed by the CPU 11, the video and audio of Mr. A are sent to Mr. B, and the image and sound of Mr. B are displayed / reproduced on the A side. It will be.

  The sound collecting device 21 is a device that collects a voice spoken by Mr. A and a sound to be emitted, and includes a known device such as a microphone. The sound collecting device 21 outputs a sound signal indicating the collected sound, and the sound signal is input to the I / O port 15 provided in the speaker-side server 10A.

  The imaging device 22 is a device that images the appearance of Mr. A and the surrounding space, and is configured by a known device such as a video camera. The imaging device 22 outputs a video signal indicating the captured video, and the video signal is input to the I / O port 15 provided in the speaker-side server 10A.

  When the human sensor 23 exists in the detection area, the human sensor 23 detects the position of the person and outputs a signal indicating the detection result to the speaker side server 10A. More specifically, the human sensor 23 according to the present embodiment is known in its structure, and detects the position of a speaker (Mr. A) when the speaker (Mr. A) is in front of the display. is there. That is, the human sensor 23 according to the present embodiment uses the position where the person is present as a detection target, and detects the detection target when the position satisfies a preset condition.

  The human sensor is not limited to the one that detects the position of the person, and at least one of the action, posture, and sound emitted by the person is set as the detection target, and the detection target is previously set. What is necessary is just to detect the said detection object, when the set conditions are satisfy | filled. For example, a sound sensor that reacts to human footsteps or spoken voice may be used to detect the sound when the sound volume exceeds a predetermined level. As for the configuration in which sound is detected as described above, it is possible to use a known configuration, for example, a configuration described in JP2013-73505A or JP2005-78347A.

  The sound reproducing device 24 is a device that reproduces sound, and includes a known device such as a speaker. The voice reproduction device 24 receives a reproduction command output when the speaker-side server 10A expands data indicating the voice of the conversation partner. As a result, the voice of the conversation partner is played back by the voice playback device 24.

  The display is a device that forms a display screen for displaying Mr. B's images (hereinafter referred to as partner images), and corresponds to a display screen forming unit. This display accepts a display command that is output when the speaker-side server 10A expands the partner image display data. As a result, the partner image is displayed on the display screen on the display.

  The display according to the present embodiment normally functions as an ornament arranged in the building (home) where Mr. A is present, specifically as a figure, and forms a display screen only when a dialogue is executed. It has become. Hereinafter, the display according to the present embodiment will be described with reference to FIGS. 3 and 4. FIG. 4 is a diagram showing a display according to the present embodiment, where FIG. 4A shows a non-interactive state when no dialogue is performed, and FIG. It shows the state at the time of dialogue.

  As described above, the display according to the present embodiment constitutes a part of the appearance arranged in Mr. A's house, specifically a specular part. As shown in FIG. Sometimes the appearance as a mirror surface part is shown without forming a display screen. On the other hand, the display according to the present embodiment forms a display screen as shown in FIG. 4B when the conversation is started, and the partner image is displayed on the display screen.

  As described above, the display according to the present embodiment is constituted by the display / mirror 25, and the display screen can be freely formed and erased. More specifically, the display / mirror 25 includes a control circuit 26 and a light emitting unit 27 as shown in FIG. When the control circuit 26 receives the display screen formation command output from the speaker-side server 10A and turns on the light emitting unit 27 in accordance with the command, the display screen is formed on the mirror surface portion of the display / mirror 25. become.

  On the other hand, during a period when there is no output of the display screen formation command, the control circuit 26 keeps the light emitting unit 27 in the unlit state, so that the mirror surface portion of the display / mirror 25 exhibits its original appearance. become. In this way, the display-use mirror 25 that forms a display screen at the time of dialogue functions as an appearance during non-dialogue when no dialogue is performed, so that it becomes difficult to stand out in the home of the dialoguer (the other party dialoguer). The presence of the display screen becomes difficult to notice. In addition, about the structure of the mirror 25 for display, a well-known structure can be utilized like the structure described in international publication 2009/122716, for example.

  Incidentally, the display that makes it difficult to notice the presence of the display screen during normal (non-interactive) time is not limited to the above-mentioned display-use mirror 25 and may be other than the appearance. In other words, for the display that forms the display screen for displaying the partner image, it is a part of the building material, furniture, or decoration arranged in the building, and the display screen can be switched between formation and deletion freely If it is. For example, it is good also as using the building material which comprises the door and inner wall of a building, or some furnitures, such as a chiffon, as said display.

  Next, the communication unit 2 provided on the conversation partner side (Mr. B) will be described. As shown in FIG. 3, a server computer (hereinafter referred to as partner server) 10B and audiovisual equipment provided on the conversation partner side. Etc. are constituted. The audio-visual equipment includes the same devices as the communication unit 1 on the speaker side, that is, a sound collection device 21, an imaging device 22, an audio reproduction device 24, and a display (strictly, a display-use mirror 25). Since these devices are the same as the devices provided in the communication unit 1 on the speaker side, description thereof will be omitted.

  Further, in the present embodiment, the communication unit 2 on the conversation partner side is also provided with the human sensor 23 as in the communication unit 1 on the speaker side. The human sensor 23 provided on the conversation partner side is an example of the sensor, and detects the position of the person when the conversation partner (Mr. B, Mr. C, or Mr. D) is in front of the display. It is. That is, the human sensor 23 provided on the conversation partner side detects a position where the person is present, and the position is equivalent to a preset condition (corresponding to the second condition, specifically, in front of the display. When the above condition is satisfied, the above detection target is detected.

  In addition, the human sensor 23 provided on the conversation partner side is not limited to the one that detects the position of the person, but at least one of the movement, posture, and sound generated by the person is a detection target, What is necessary is just to detect the said detection target, when the said detection target satisfy | fills the conditions set beforehand.

  The counterpart server 10B has substantially the same configuration as the speaker server 10A, and a dialogue program is stored in the memory 12 thereof. Then, the conversation program is read and executed by the CPU 11, so that Mr. B's video and audio are sent to Mr. A, and Mr. A's image and sound are displayed / played on Mr. B's side. Will be.

  By the way, the partner server 10B is equipped with a special function in addition to the function of the speaker server 10A. Explaining this function, when receiving the video data indicating the video of the speaker from the speaker side server 10A, the partner server 10B generates a speaker image (hereinafter referred to as a speaker image) based on the video data. Display data for display is generated. When this display data is expanded, a speaker image is displayed on a display provided on the side of Mr. B.

  And in this embodiment, the other party server 10B produces | generates the display data for displaying the image corrected from the video | video of the speaker which the video data received from 10A of speaker side shows at the time of display data production | generation. Is possible. In other words, in the present embodiment, the speaker image displayed to the conversation partner is an image obtained by correcting the image when the speaker is captured (in simple terms, the actual image of the captured speaker is Different images).

  In order to explain the correction of the speaker image more specifically, any one of Mr. B, Mr. C, and Mr. D reacts to Mr. A's story (for example, an operation such as speaking or laughing) Is assumed. At this time, the counterpart server 10B detects the above-described reaction operation, and specifies the position where the person who performed the operation (hereinafter referred to as the operation executor) is present. After that, the counterpart server 10B generates display data for displaying the speaker image corrected so that Mr. A's line of sight is directed to the position where the action executor is present based on the identification result.

  Then, when the display data is expanded, Mr. B's side displays a speaker image (hereinafter referred to as a line-of-sight image) in which Mr. A's line of sight is directed to the action performer as shown in FIG. It becomes like this. By displaying this line-of-sight image, the person performing the operation looks at Mr. A's line of sight in the image and feels that Mr. A has responded to his / her movement, as if Mr. A. You will feel like you are interacting directly with each other.

  In the present embodiment, the partner server 10B displays the display data for displaying the speaker image corrected so that Mr. A's line of sight and face are directed at the position where the operation executor is present after the line-of-sight image is displayed. Is further generated. When such display data is expanded, Mr. B's side displays a speaker image (hereinafter referred to as a secondary line-of-sight alignment image) in which Mr. A's line of sight and face are directed to the action performer, as shown in FIG. Become so. By displaying this secondary line-of-sight image, a more realistic dialogue is realized. If it explains in an easy-to-understand manner, not only Mr. A's line of sight but also Mr. A's face is directed toward the action performer, so the degree to which the action performer feels as if interacting with Mr. A, that is, face-to-face Will increase more.

<< Configuration of Speaker Server and Counterparty Server >>
The configuration of each of the speaker-side server 10A and the partner-side server 10B, particularly the hardware configuration, has already been described. In the following, the configuration of each server will be described again from the functional aspect with reference to FIG. I decided to. FIG. 5 is a diagram showing the configuration of each of the speaker-side server 10A and the partner-side server 10B in terms of functions.

  The configuration of the speaker-side server 10A will be described in terms of functions. As shown in FIG. 5, a data acquisition unit 31, a data transmission unit 32, a data reception unit 33, a partner image display unit 34, and a partner voice reproduction unit 35 are provided. ing. These correspond to data processing executed by the speaker-side server 10A, that is, a data processing unit. The five data processing units described above are realized by hardware devices such as the CPU 11, the memory 12, the hard disk drive 13, the communication interface 14, and the I / O port 15 cooperating with an interactive program as software. The Hereinafter, each data processing unit will be described individually.

(About the data acquisition unit 31)
The data acquisition unit 31 digitizes and encodes signals received from the sound collection device 21 and the imaging device 22 via the I / O port 15 by the speaker-side server 10A, specifically, an audio signal and a video signal, respectively. In this way, audio data and video data are acquired. Here, the voice data is data indicating the voice (speaking voice) of Mr. A collected by the sound collecting device 21. The video data is data indicating an actual video when Mr. A is imaged. In the present embodiment, the video data indicates a full-body image of A who is in front of the display / mirror 25 and video of the surrounding space. It has become.

  In the present embodiment, when the human sensor 23 provided on the Mr. A side detects the position of a person in the detection area, the sound collection device 21 and the imaging device 22 are activated using this as a trigger, and the voice Sound collection and video imaging are started. In conjunction with this, the data acquisition unit 31 starts to acquire the two data.

(About the data transmission unit 32)
When the data acquisition unit 31 acquires audio data and video data, the data transmission unit 32 transmits these data to the counterpart server 10B as needed. In the present embodiment, the data transmission unit 32 multiplexes audio data and video data and transmits them as one data (hereinafter referred to as dialogue data).

(About the data receiver 33)
The data receiving unit 33 receives the conversation data transmitted from the counterpart server 10B through the communication line 3. The received dialogue data is stored in a predetermined area of the memory 12 of the speaker-side server 10A or the hard disk drive 13.

(About the partner image display unit 34)
The partner image display unit 34 reads out the conversation data stored in the memory 12 or the hard disk drive 13, extracts the video data from the data, decodes and expands the video data, and displays it on the display on the Mr. A side. Output a display command. On the display side that has received this display command, the other party's image, that is, the whole body image of Mr. B and the video of the surrounding space are displayed on the display screen. In this way, the partner image display unit 34 displays the partner image by expanding the video data extracted from the conversation data.

  By the way, in this embodiment, as described above, the display provided on the speaker side is constituted by the display / mirror 25. The partner image display unit 34 executes a process for forming a display screen on the mirror surface portion of the display / mirror 25 when displaying the partner image. Such processing (hereinafter, display screen formation command processing) is executed by the partner image display unit 34 when the screen formation condition is satisfied in a state where the display mirror 25 does not form a display screen. Here, the screen formation condition is set in advance as a condition for forming a display screen on the display / mirror 25. Specifically, the human sensor 23 provided on the side of Mr. A is within the detection area. It is to detect the position of the person who is in the area.

  More specifically, the detection area of the human sensor 23 is set at a position slightly closer to the front position of the display / use mirror 25, for example, the position when the display / use mirror 25 is used as a look. Therefore, as shown in FIG. 4B, when Mr. A is located closer to the display / mirror 25 than usual, the human sensor 23 is located within the detection area. Is detected, and a signal indicating the detection result (hereinafter, a detection signal) is output to the speaker-side server 10A. When the detection signal is input to the speaker-side server 10A via the I / O port 15, the partner image display unit 34 executes a display screen formation command process, and a command (display) for forming the display screen on the display / mirror 25. A screen formation command is generated and the command is output to the display mirror 25.

  In the display-use mirror 25 that has received the display screen formation command, the control circuit 26 turns on the light emitting unit 27 in accordance with the command. As a result, a display screen is formed on the mirror surface portion of the display-use mirror 25 that has been shown as an appearance. The display mirror 25 is maintained in a state in which the display screen is formed while the human sensor 23 continues to detect the position of the person in the detection area. On the other hand, when a person who is in the detection area moves out of the area and the human sensor 23 no longer detects the detection target, the display-use mirror 25 erases the display screen, and the mirror surface portion has an appearance as a appearance. To come out. That is, in the present embodiment, the display-use mirror 25 forms a display screen only during a period in which the human sensor 23 is detecting the position of a person who is in the detection area.

(About the other party voice playback unit 35)
The other party's voice playback unit 35 reads out conversation data stored in the memory 12 or the hard disk drive 13, extracts voice data from the data, decodes and expands the voice data, and plays the voice of Mr. A. A reproduction command is output to the device 24. The audio reproduction device 24 that has received this reproduction command reproduces the audio indicated by the audio data, that is, the voices of Mr. B and others, according to the command.

  Next, the configuration of the counterpart server 10B will be described in terms of functions. As shown in FIG. 5, a data acquisition unit 41, a data transmission unit 42, a data reception unit 43, an operation performer detection unit 44, and a display data generation unit 45 are provided. , A speaker image display unit 46 and a speaker voice reproduction unit 47 are provided. These correspond to data processing performed by the counterpart server 10B, that is, a data processing unit. The seven data processing units described above are realized by hardware devices such as the CPU 11, the memory 12, the hard disk drive 13, the communication interface 14, and the I / O port 15 cooperating with an interactive program as software. The

  Among the above seven data processing units, the data acquisition unit 41, the data transmission unit 42, and the data reception unit 43 have different data contents, but the functions of the speaker side server 10A are the same. Since these are common to the units (specifically, the data acquisition unit 31, the data transmission unit 32, and the data reception unit 33), description thereof will be omitted. Hereinafter, the remaining data processing units will be described.

(About the operation executor detection unit 44)
The operation executor detection unit 44 corresponds to a detection unit, and detects an operation executor based on audio data and video data acquired by the data acquisition unit 41. More specifically, Mr. B, Mr. C, and Mr. D who are conversation partners are in front of the display / mirror 25 constituting the display during the conversation, and the display screen formed on the display / mirror 25 is displayed on the display screen. The image of Mr. A is viewed and the sound of Mr. A reproduced by the audio reproducing device 24 is heard. During this time, Mr. B's voice is collected by the sound collecting device 21, and the image of Mr. B's is captured by the imaging device 22. The audio signal and video signal are sequentially output toward the counterpart server 10B, and the data acquisition unit 41 acquires audio data and video data from the output signal.

  On the other hand, the operation executor detection unit 44 analyzes the audio data and the video data acquired by the data acquisition unit 41, and determines whether there is a person who performs the corresponding operation among Mr. B. Here, the corresponding action is an action that satisfies a preset condition for detecting the person who performs the action, and specifically, an action that produces a laughter or a voice. Then, when it is determined that there is a person who performs the corresponding operation, the action executor detection unit 44 detects the person.

  The method for detecting the person performing the operation is not particularly limited. For example, the volume and the pitch of the voice are specified from the voice data, and the position of the voice source is calculated from the specified result. It is conceivable to identify the position of each conversation partner from the video data and detect the person who is in the position corresponding to the calculated position of the sound generation source to detect the person performing the operation.

(About the display data generation unit 45)
The display data generation unit 45 extracts video data from the conversation data transmitted from the speaker side server 10A, and generates display data of the speaker image based on the video data. In this embodiment, there are two types of data processing (display data generation processing) by the display data generation unit 45, and which of the processing is executed is detected by the operation performer detection unit 44. It will change according to the presence or absence.

  More specifically, when the operation executor detection unit 44 detects an operation executor, the display data generation unit 45 executes display data generation processing for displaying the above-described line-of-sight image as a speaker image. Such a process corresponds to the first process of the present invention, and is hereinafter referred to as a line-of-sight image generation process.

  The line-of-sight image generation processing will be described along the procedure illustrated in FIG. FIG. 6 is a diagram illustrating a flow of the line-of-sight image generation processing. The line-of-sight image generation processing is executed with the operation executor detection unit 44 detecting the operation executor as a trigger, and starts from a step of identifying the position where the operation executor is present (S001). In this step S001, the display data generation unit 45 specifies the position of the operation performer from the position of the sound generation source determined when the operation performer detection unit 44 detects the operation performer. Hereinafter, the operation executor position specifying step S001 will be described in more detail with reference to FIG. FIG. 7 is an explanatory diagram regarding the specification of the position of the person performing the operation.

  The display data generation unit 45 analyzes the video data and specifies the position of the person who performs the operation in order to specify the position of the conversation partner who uttered the voice (the person on the rightmost side in FIG. 7). More specifically, the direction of the conversation partner who uttered the voice and the tilt angle from the front position of the imaging device 22 (shown by the symbol θ in FIG. 7) when identified from the imaging device 22 are specified. In addition, about the method of specifying an operation performer position, it is not limited to said content, For example, it is good also as specifying an operation performer's position using a distance sensor or a position sensor.

  After specifying the position of the person who performs the operation, the display data generation unit 45 extracts video data from the dialogue data received by the data receiving unit 43 from the speaker-side server 10A, and the speaker for the extracted video data. Video separation processing is executed (S002). In this process, the video of the speaker indicated by the video data extracted from the dialogue data is converted into the video of the speaker's pupil (hereinafter referred to as the pupil video) and the video of the head excluding the pupil (hereinafter referred to as the head) as shown in FIG. Video) and a video of the space other than the speaker's head and the surrounding space (hereinafter referred to as a torso video), which is realized by a known image processing technique. FIG. 8 is an explanatory diagram regarding the decomposition of the speaker video.

  After executing the speaker video decomposition process, the display data generation unit 45 executes a process of editing the pupil shape and the relative position with respect to the eyeball on the pupil video data divided from the speaker video (S003). This line-of-sight editing process is executed to change the line of sight of the speaker in accordance with the action executor position specified in the action executor position specifying step S001. The line-of-sight editing process will be described below with reference to FIG. FIG. 9 is an explanatory diagram regarding line-of-sight editing.

  In the line-of-sight editing process, the actual pupil image (the black portion in FIG. 9) is edited according to the position of the person performing the operation, and specifically, tilted in the direction when the person performing the operation is viewed from the imaging device 22. The shape and position of the pupil are changed so that the line of sight is shifted by an amount corresponding to the angle θ. Through this procedure, the pupil image is an image whose position and shape are changed according to the position of the person performing the operation as shown by the hatched portion in FIG. 9 (hereinafter referred to as an edited pupil image). Become. As a method for changing the pupil shape and position in the line-of-sight editing process, a known image editing technique can be used.

  After executing the line-of-sight editing process, the display data generating unit 45 synthesizes the head image and the torso image extracted in the speaker image decomposition step S002 and the edited pupil image obtained in the line-of-sight editing step S003. Display data for displaying is generated (S004). When the display data obtained in step S004 is developed, the line-of-sight image shown in FIG. 10 is displayed on the display as a speaker image. FIG. 10 is a diagram illustrating a state in which a line-of-sight alignment image is displayed.

  The line-of-sight image will be described in more detail. The image is different from the actual speaker image displayed by expanding the video data in the conversation data received from the speaker-side server 10A. More specifically, as shown in FIG. 10, the line-of-sight image shows an actual speaker image so that the speaker's line of sight is directed to the position of the person performing the operation (position D in the case shown in FIG. 10). It has been corrected. In other words, the display data generated by combining the partial videos divided from the speaker video is such that the speaker's line of sight is directed to the position of the action executor specified in the action executor position specifying step S001. This is data for displaying the speaker image corrected in (1). A known image processing technique can be used as a method for combining the partial images.

  In addition, it is desirable that the line of sight of the speaker in the line-of-sight image is gradually switched in accordance with the normal human movement, rather than instantaneously switching to the position of the person performing the operation. Therefore, in the line-of-sight editing step S003, it is preferable to edit the line of sight so as to gradually change toward the position of the person performing the operation.

  In addition, regarding the time from the start of the line-of-sight image generation process triggered by the detection of the person performing the operation until the display data generated by the process is expanded to display the line-of-sight image, The time required for switching the line of sight in the neural circuit is preferably matched. In such a case, the speaker's line of sight changes more naturally in the speaker image.

  After generating the display data of the line-of-sight image, the display data generating unit 45 further executes a process of editing the direction of the speaker's face on the head image data divided in the speaker image decomposition process. (S005). This process is executed to change the direction of the speaker's face according to the action performer position specified in the action performer position specifying step S001. The face orientation editing process will be described below with reference to FIG. FIG. 11 is an explanatory diagram regarding face orientation editing.

  In the face orientation editing process, the head image is edited according to the position of the person who performs the operation. Specifically, the center line of the face when the speaker is facing the front (indicated by a broken line in FIG. 11). The position of each part of the face such as the nose and mouth and the contour of the face so that the direction of the speaker's face changes by an amount corresponding to the tilt angle θ in the direction when the person performing the operation is viewed from the imaging device 22 with reference. To change. Through this procedure, the head image is an image in which the orientation of the face is changed according to the position of the person performing the operation from the state of facing the front as shown in FIG. 11 (hereinafter, the edited head image). It becomes. As a method for changing the face orientation in the face orientation editing process, a known image editing technique can be used.

  After executing the face orientation editing process, the display data generation unit 45 generates display data for displaying an image obtained by combining the edited head image obtained in the previous step S005 and the remaining partial image (S006). ). When the display data obtained in step S006 is developed, the secondary line-of-sight image shown in FIG. 12 is displayed on the display as a speaker image. FIG. 12 is a diagram illustrating a state in which a secondary line-of-sight alignment image is displayed.

  The secondary line-of-sight image will be described. Similar to the above-described line-of-sight alignment image, the image is different from the actual speaker image, and as shown in FIG. 12, the position of the person performing the operation (in the case shown in FIG. 10). The actual video image of the speaker is corrected so that the speaker's line of sight and the speaker's face face the position of Mr. D. In other words, the display data generated by recombining the partial videos divided from the speaker video is the speaker's line of sight and speech at the position of the motion executor identified in the motion executor position identifying step S001. This is data for displaying a speaker image corrected so that the person's face faces. As for the method of recombining each partial image to construct a secondary line-of-sight image, a known image processing technique can be used as in the case of combining each partial image to construct a line-of-sight image. is there.

  Moreover, it is desirable that the orientation of the speaker's face in the secondary line-of-sight image changes gradually in accordance with normal human movement, rather than instantaneously changing toward the action performer. Therefore, in the face direction editing step S005, it is preferable to edit the face direction of the speaker so as to gradually change toward the position of the person performing the operation. At this time, it is desirable that the line of sight of the speaker changes in conjunction with the change in the orientation of the face. Therefore, at the same time as the face direction editing step S005, the pupil video is re-edited in association with the change in the face direction, and in this re-editing step, the shape and position of the pupil are changed according to the amount of change in the face direction. And good.

  When the series of steps described above is completed, the line-of-sight image generation processing is completed. Then, the display data generation unit 45 repeatedly executes the line-of-sight image generation process every time the operation executor detection unit 44 detects the operation executor.

  On the other hand, when the motion executor detection unit 44 has not detected a motion executor for a predetermined time or longer, the display data generation unit 45 displays a speaker image in which the speaker's line of sight faces a predetermined position (hereinafter referred to as a look image). The display data generation process for displaying is executed. Such a process corresponds to the second process of the present invention, and is hereinafter referred to as a “layout image generation process”.

  The layout image generation processing will be described along the procedure illustrated in FIG. FIG. 13 is a diagram illustrating a flow of a side-by-side image generation process. The gaze image generation process is executed when the operation executor detection unit 44 does not detect the operation executor reaches a predetermined time. Note that the non-detection time of the operation executor as an execution condition of the side-by-side image generation process can be arbitrarily set.

  The gaze image generation process starts from a step of specifying a position where each conversation partner is present (S011). Specifically, in this step S011, the display data generation unit 45 analyzes the video data of the conversation partner imaged by the imaging device 22, and specifies the position of each conversation partner. More specifically, for each conversation partner, the direction when viewed from the imaging device 22 and the inclination angle θ from the front position of the imaging device 22 are specified. In addition, about the method of specifying the position of each dialogue partner, it is not limited to said content, For example, it is good also as specifying the position of an operation person using a distance sensor or a position sensor.

  After specifying the position of each conversation partner, the display data generating unit 45 extracts video data from the conversation data received by the data receiving unit 43 from the speaker-side server 10A, and the speaker is extracted from the extracted video data. Video separation processing is executed (S012). This process is the same as the speaker video decomposition process in the line-of-sight image generation process.

  After executing the speaker video decomposition process, the display data generating unit 45 executes a process of editing the pupil shape and the relative position with respect to the eyeball on the pupil video data divided from the speaker video (S013). ). This line-of-sight editing process is executed to direct the speaker's line of sight to a preset position, and specifically, directs to a position where one of a plurality of conversation partners B, C, and D is present. To be executed. Note that the procedure of the line-of-sight editing process is the same as that of the line-of-sight editing process in the line-of-sight image generation process, and a known image editing technique can be used as a method of changing the pupil shape and position in the process.

  After executing the line-of-sight editing process, the display data generation unit 45 combines the edited pupil image obtained in the previous step S013 with the head image and the torso image extracted in the speaker image decomposition step S012. Display data for displaying is generated (S014). When the display data obtained in step S014 is developed, a speaker image in which the speaker's line of sight is directed to a position where one of the conversation partners is present, that is, a look-ahead image, is displayed on the display.

  The look-ahead image will be described. Similar to the line-of-sight image, the image is different from the actual speaker image displayed by expanding the video data in the conversation data received from the speaker-side server 10A. More specifically, the look-ahead image is obtained by correcting an actual speaker video so that the speaker's line of sight is directed to a position where one of the conversation partners is present. In other words, the display data generated by synthesizing the partial videos divided from the speaker video is a predetermined conversation partner, for example, an imaging device among the positions of the respective conversation partners in the conversation partner position specifying step S011. 22 is data for displaying a speaker image corrected so that the speaker's line of sight is directed to a position where the conversation partner located on the rightmost side as viewed from 22 is present. A known image processing technique can be used as a method for combining the partial images.

  Further, it is desirable that the line of sight of the speaker in the look-ahead image is gradually switched in accordance with a normal human movement rather than instantaneously switching toward a position where a predetermined conversation partner is present. Therefore, in the line-of-sight editing step S013, it is preferable to edit the line of sight so as to gradually change toward a position where a predetermined dialogue partner is present.

  At the time when the series of steps described above is completed, the side-by-side image generation processing ends. Then, the display data generation unit 45 repeatedly executes the look-ahead image generation process at regular intervals while the non-detection period in which the operation performer is not detected continues. At this time, the display data generation unit 45 repeats the look-ahead image generation process so that the conversation partner at the position where the line of sight of the speaker is facing is sequentially switched among the plurality of conversation partners.

  More specifically, in the one-time gaze image generation process, correction was performed so that the speaker's line of sight is directed to the position where the conversation partner (that is, Mr. B) located on the rightmost side as viewed from the imaging device 22 is located. It is assumed that display data for a side-by-side image is generated. In such a case, in the next-stage run-out image generation process, the run-out image display data corrected so that the speaker's line of sight is directed to the position of the conversation partner (ie, Mr. C) on the left side of B is generated. . Further, in the side-by-side image generation process executed in the next round, correction is performed so that the speaker's line of sight is directed to the position where the conversation partner (ie, Mr. D) is located on the leftmost side when viewed from the imaging device 22. Display data of a side-by-side image is generated. Thereafter, the look-ahead image generation process is repeatedly executed so that the speaker's line of sight is switched in the order of Mr. B, Mr. C, and Mr. D.

  As described above, by repeating the look-ahead image generation process, in the speaker image (that is, the look-ahead image) displayed on the display on the side of Mr. B, (A), (B), and (C ), The speaker's line of sight changes periodically, and the conversation partner at the end of the line of sight switches sequentially. (A), (B), and (C) of FIG. 14 are views showing a state in which a speaker image is displayed based on display data generated by the lookout image generation process. The above-mentioned visual effect makes it possible for each conversation partner to feel as if the speaker is paying attention, thereby improving the preference of the conversation performed while looking at the speaker image. .

  In this embodiment, it is assumed that only the speaker's line of sight faces the position of the predetermined conversation partner in the look-ahead image, but the speaker's face faces the predetermined conversation partner's position together with the speaker's line of sight. It is also possible to display a fine-grained image. In other words, in the look-ahead image generation process, display data for displaying a speaker image (a look-ahead image) corrected so that the speaker's line of sight and the speaker's face are directed to the position of a predetermined conversation partner is generated. It is good.

(About the speaker image display unit 46 and the speaker voice reproduction unit 47)
The speaker image display unit 46 functions as an image display unit in cooperation with the display on the side of Mr. B. Strictly speaking, the speaker image display unit 46 expands the display data generated by the display data generation unit 45 and outputs a display command to the display on the side of Mr. B. On the display side that has received this display command, the above-described line-of-sight image, secondary line-of-sight image, or gaze image is displayed on the display screen. Thus, the speaker image display unit 46 displays the speaker image by expanding the display data.

  Since the display on the side of Mr. B is also composed of the display / mirror 25, the speaker image display unit 46 forms a display screen on the mirror surface of the display / mirror 25 when displaying the speaker image. Therefore, a display screen formation command process is executed. Since the display screen formation command process is the same as the process performed by the partner image display unit 34 of the speaker-side server 10A, a description of the specific procedure of the process is omitted.

  The speaker voice playback unit 47 extracts voice data from the conversation data received from the speaker side server 10A, decodes and expands the voice data, and issues a playback command to the voice playback device 24 on the Mr. B side. Output. The audio reproducing device 24 that has received the reproduction command reproduces the audio indicated by the audio data, that is, Mr. A's audio in accordance with the command.

<< Procedure for Image Display Method According to Present Embodiment >>
Next, an image display method according to the present embodiment will be described. In the following description, the case where the speaker is Mr. A and the conversation partners are Mr. B, Mr. C, and Mr. D is given as a specific example in the same manner as described above.

  The image display method according to the present embodiment is realized by the communication units 1 and 2 of both the speaker and the conversation partner in the system S. Specifically, the server computer (speaker-side server 10A) of each communication unit 1 and 2 is used. And the counterpart server 10B) execute data processing sequentially in the flow shown in FIGS. FIGS. 15 and 16 are diagrams illustrating a flow of data processing executed by the system S in the dialogue between Mr. A and Mr. B. Hereinafter, a flow of a series of data processing will be described with reference to FIGS.

  First, Mr. A moves to the front of the display / mirror 25 installed in his / her home, and the human sensor 23 detects the position of Mr. A in the detection area (S021). When the speaker-side server 10A receives a signal indicating the detection result of the human sensor 23 via the I / O port 15 (Yes in S021), the speaker-side server 10A executes display screen formation command processing. As a result, the display-use mirror 25 on the side of Mr. A transitions from a state in which the mirror surface portion shows the appearance of appearance, and forms a display screen on the mirror surface portion (S022). At the time when the display screen is formed on the display-use mirror 25 on the side of Mr. A, the display screen is a predetermined standby screen.

  On the other hand, the sound collecting device 21 and the image pickup device 22 are operated in accordance with the execution of the display screen formation command process, whereby the whole body image of A and the surrounding space are picked up, and Mr. A's speaking voice is collected. (S023). Thereafter, the speaker-side server 10A generates conversation data based on the output signals from the sound collecting device 21 and the imaging device 22, and transmits the data to the counterpart server 10B (S024).

  The partner server 10B receives the dialog data via the communication line 3, and stores the dialog data in the internal memory 12 or the hard disk drive 13 (S025). Thereafter, the counterpart server 10B reads the stored conversation data, extracts voice data from the data, and outputs a command to play the voice indicated by the voice data to the voice playback device 24. Receiving the reproduction command, the audio reproducing device 24 reproduces the audio indicated by the audio data (S026). As a result, Mr. A's voice (speaking voice) can be heard in the building where Mr. B is.

  On the other hand, Mr. B moves to the front of the display / mirror 25 installed in the building where Mr. B is in response to Mr. A's voice. If any of Mr. B enters the detection area of the human sensor 23 while the voice of Mr. A is being reproduced, the human sensor 23 detects the position of the person (S027). . When the partner server 10B receives a signal indicating the detection result by the human sensor 23 via the I / O port 15, the display server forming command process is executed accordingly. As a result, the display-use mirror 25 on the side of Mrs. B transitions from the state in which the mirror surface portion shows the appearance of appearance, and forms a display screen on the mirror surface portion (S028).

  After the display screen is formed, the counterpart server 10B executes data processing related to the display of the speaker image. By executing the data processing, Mr. A's voice is reproduced by the voice reproduction device 24, and a speaker image is displayed on the display screen (S029). The flow of data processing related to the display of the speaker image will be described in detail later.

  Further, when the human sensor 23 detects the position of a person in the detection area in the building where Mr. B and others are located, the sound collecting device 21 and the imaging device 22 are activated. Thereby, Mr. B's whole body image and the image of the surrounding space are imaged, and Mr. B's voice is collected (S030). Thereafter, the partner server 10B generates dialogue data based on the output signals from the sound collecting device 21 and the imaging device 22, and transmits the generated dialogue data to the speaker side server 10A (S031).

  The speaker-side server 10A receives the dialog data via the communication line 3, and stores the dialog data in the internal memory 12 or the hard disk drive 13 (S032). Thereafter, the speaker-side server 10A reads the stored conversation data, extracts audio data and video data from the data, and develops these two data. Then, the speaker-side server 10A outputs a command for playing back the voice indicated by the extracted voice data to the voice playback device 24, and a command for displaying the video indicated by the extracted video data (that is, the partner image). Output to the mirror 25 for display. As a result, Mr. A's voice reproduction device 24 reproduces Mr. B's voice, and the display screen formed by the display and mirror 25 displays Mr. B's whole body image and the image of the surrounding space. (S033).

  Thereafter, while the conversation between the two continues, the series of data processing described above is repeatedly executed in each of the communication units 1 and 2. Then, in the building where Mr. A's house or Mr. B's are located, when the human sensor 23 no longer detects the position of the person in the detection area (S034, S035), the dialogue is terminated, and the series described above. The execution of this data processing is also terminated. Thereby, the sound collector 21 and the imaging device 22 are stopped. When the human sensor 23 no longer detects the position of the person in the detection area, the display screen formed on the display mirror 25 is erased in conjunction with this, and the display mirror 25 Appearance appears as a figure on the mirror surface.

  Next, of the above-described data processing, data processing related to display of a speaker image will be described in detail with reference to FIG. FIG. 17 is a diagram illustrating a procedure for displaying a speaker image. As shown in FIG. 17, the data processing related to the display of the speaker image is divided into two patterns depending on whether or not the operation server is detected by the partner server 10B.

  One processing pattern is a pattern when the partner server 10B detects an operation performer (Yes in S041). In this processing pattern, the counterpart server 10B executes the above-described line-of-sight image generation processing (S042). The display data generated in the line-of-sight image generation processing is successively developed by the counterpart server 10B (S043). Thereby, a speaker image comes to be displayed on the display of Mr. B's side. More specifically, when the line-of-sight image generation processing is executed, first, a line-of-sight image in which the speaker's line of sight faces the position of the person performing the operation is displayed, and then the line of sight and face of the speaker are displayed. A secondary line-of-sight alignment image facing the position of the person performing the operation is displayed. When the line-of-sight image generation processing is executed, the line-of-sight image (strictly, the secondary line-of-sight image) continues to be displayed on the display for a certain period of time.

  When a certain time has elapsed since the line-of-sight image generation processing is executed (Yes in S044), the counterpart server 10B again determines whether or not the operation performer has been detected (S041).

  The other processing pattern is a pattern when the counterpart server 10B does not detect the operation performer (No in S041). In such a processing pattern, the counterpart server 10B executes the above-described gaze image generation processing (S045). The display data generated by the side-by-side image generation process is successively developed by the counterpart server 10B (S046). Accordingly, a look-ahead image in which the line of sight of the speaker is directed to the position of the predetermined conversation partner is displayed as the speaker image on the display on the side of Mrs. B and others.

  Then, the side-by-side image generation process is repeatedly executed at regular intervals during the non-detection period in which the counterpart server 10B does not detect the operation performer (S047, S048). Thereby, in the speaker image displayed on the display on the side of Mr. B, the direction of the line of sight of the speaker is switched in the order of Mr. B, Mr. C, and Mr. D.

  When the other server 10B detects an action performer after executing the gaze image generation process (Yes in S047), as shown in FIG. 17, the processing pattern is changed from the pattern for executing the gaze image generation process. The process shifts to a pattern for executing the line-of-sight image generation processing.

<< Other Embodiments >>
In the above embodiment, the image display system and the image display method of the present invention have been described by way of example. However, said embodiment is only an example for making an understanding of this invention easy, and does not limit this invention. The present invention can be changed and improved without departing from the gist thereof, and the present invention includes the equivalents thereof.

  Further, in the above embodiment, when displaying a speaker image in which the speaker's line of sight and the direction of the speaker's face are corrected with respect to the actual speaker image, the actual speaker image is divided into partial images. It was. Then, in the divided partial video, the pupil video and the head video are edited, and the edited partial video and the remaining partial video are combined to generate the final speaker image display data. . Such a procedure is merely an example of a procedure for generating speaker image display data, and the speaker image display data may be generated by another procedure. For example, it is possible to edit only the pupil or the head in a state where the speaker video is not divided into partial videos and is still in the state of the speaker video (that is, not divided).

  In the above embodiment, the partner server 10B generates the display data of the speaker image. However, the apparatus that generates the display data is not limited to the partner server 10B, and may be performed by the speaker server 10A. Alternatively, a third server (a server not shown, such as an ASP server or a cloud service server) that can communicate with both the speaker server 10A and the partner server 10B may generate display data. .

  In the above embodiment, when there is a conversation partner who laughs or speaks during the conversation, the partner server 10B detects the conversation partner as an action executor, and executes a line-of-sight image generation process using this as a trigger. It was decided to. However, the condition for detecting the person who performs the operation is not limited to laughing out or speaking, and performing other operations may be set as the detection condition. For example, it is good also as setting as detection conditions performing operation | movement which moves a body like raising hand operation | movement, standing-up operation | movement, etc.

  In the above embodiment, the speaker image is described as an example of the image obtained by correcting the line of sight and the face direction. However, the line of sight and the face direction are actually applied to the conversation partner image (the partner image). It is good also as displaying after correcting from the image | video of.

  In the above-described embodiment, a case where there are a plurality of conversation partners with respect to one speaker has been described as an example, but the present invention is not limited to this. For example, there may be a plurality of speakers. In such a case, when generating the display data of the speaker image, the speaker performs data processing (specifically, dividing, editing, and synthesizing the speaker video) to correct the speaker's line of sight and face orientation. It will be executed every time. Further, the conversation partner may be a single person (one person). However, from the viewpoint of more effectively demonstrating the function of the system S, the above-described configuration in which there are a plurality of conversation partners is more desirable.

S System (image display system)
1, 2 Communication unit 3 Communication line 10A Speaker server 10B Partner server 11 CPU, 12 Memory 13 Hard disk drive 14 Communication interface 15 I / O port 21 Sound collecting device, 22 Imaging device 23 Human sensor, 24 Voice reproduction Device 25 Display / mirror 26 Control circuit, 27 Light emitting unit 31, 41 Data acquisition unit 32, 42 Data transmission unit 33, 43 Data reception unit 34 Partner image display unit, 35 Partner voice playback unit 44 Operation performer detection unit, 45 Display Data generation unit 46 Speaker image display unit 47 Speaker audio playback unit

Claims (5)

An image display system for displaying a speaker's image to a conversation partner,
A data acquisition unit which is provided on the speaker side and acquires data indicating an image when the speaker is imaged;
A display data generation unit for generating display data for displaying the image corrected from the video;
An image display unit provided on the conversation partner side and displaying the image by expanding the display data;
A detection unit that is provided on the conversation partner side and detects the conversation partner when the conversation partner performs an operation that satisfies a preset condition ;
Wherein when the front Symbol detection unit detects the dialogue partner were performing the operation, the display data generation unit, which has been corrected so that the face is the speaker of the line-of-sight to the dialogue partner is present position, which was the operating Performing a first process for generating the display data for displaying an image ;
When the detection unit does not detect the conversation partner who has performed the operation for a predetermined time or longer, the display data generation unit displays the image corrected so that the line of sight of the speaker faces at a preset position. An image display system characterized by executing a second process for generating the display data for the purpose .   When the detection unit detects the conversation partner that performed the operation, the display data generation unit is configured so that the line of sight of the speaker and the face of the speaker are directed to a position where the conversation partner performing the operation is present. The image display system according to claim 1, wherein the display data for displaying the corrected image is generated. In the case where there are a plurality of the conversation partners, when the detection unit does not detect the conversation partner that has performed the operation for a predetermined time or more, the display data generation unit is the line of sight of the speaker among the plurality of the conversation partners. the image display system according to claim 1, characterized in that repeatedly executes the second processing the dialogue as partner sequentially switched to being in a position where the facing. A display screen forming unit that forms a part of a building material, furniture, or decoration arranged in a building where the conversation partner is located and forms a display screen of the image;
It is set in advance when the object to be detected is at least one of the action performed by the conversation partner, the position where the conversation partner is located, the posture of the conversation partner, and the sound emitted by the conversation partner. A sensor that detects the detection target that satisfies the second condition,
The display screen forming unit displays the appearance as the part without forming the display screen during a period in which the sensor does not detect the detection target that satisfies the second condition. the image display system according to any one of claims 1 to 3, characterized in that the condition is satisfied the formation of the display screen only during the period in which to detect the detection target. An image display method for displaying an image of a speaker to a conversation partner,
A data acquisition unit provided on the speaker side acquires data indicating an image when the speaker is imaged;
A display data generation unit generating display data for displaying the image corrected from the video;
An image display unit provided on the conversation partner side displays the image by expanding the display data;
A detection unit provided on the conversation partner side detects the conversation partner when the conversation partner performs an operation that satisfies a preset condition;
When the detection unit detects the conversation partner who performed the operation , the display data generation unit corrects the image so that the line of sight of the speaker faces the position where the conversation partner who performed the operation exists. Executing a first process for generating the display data for displaying
When the detection unit does not detect the conversation partner who has performed the operation for a predetermined time or longer, the display data generation unit displays the image corrected so that the line of sight of the speaker faces at a preset position. An image display method comprising: executing a second process for generating the display data for the purpose .

Images