JP6550307B2 - Image display system and image display method - Google Patents

Description

  The present invention relates to an image display system and an image display method, and more particularly to an image display system and an image display method capable of reducing a transmission load of image data of a frame image constituting a video of a user.

  Image display systems using ICT (information communication technology) are already known. Such a system is used, for example, when users in different spaces interact with each other. In such a case, each user can talk to the other party while viewing the image of the other party (more specifically, a video composed of a plurality of frame images) displayed on the display such as a screen. It is. This enables the user who is looking at the other party of the dialogue through the display to interact in the same atmosphere (realism) as when actually facing the other party of the dialogue.

  On the other hand, the sense of realism of dialogue improves as the image of the other party of the dialogue displayed on the display device has higher image quality. However, on the other hand, the higher the quality of the image of the conversation partner, the larger the data capacity of the image data sent from the conversation partner, and the load (communication load) associated with transmission and reception of the image data increases. It will As a measure against such a problem, for example, when an image is newly acquired, it is possible to send image data of only an image (that is, an image of a moved part) corresponding to the difference from the previously acquired image. (See Patent Document 1). With such a configuration, it is possible to reduce the data capacity at the time of transmission of the image data and to display a high quality image (video) in the image receiving apparatus.

JP 2003-299088 A

  By the way, when the person displayed in the frame image moves, after the moved portion (specified portion) is specified, the image data of the specified portion is transmitted. On the other hand, when specifying the specified part, it is naturally required to specify with high accuracy by an appropriate procedure. In particular, in the dialog using the above system, whether or not the specified part is properly identified may affect the realism of the dialog.

  Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to reduce the transmission load of image data and to move the person displayed in the frame image. It is providing the image display system which can identify a part appropriately. Similarly, another object of the present invention is to provide an image display method capable of appropriately specifying a part when a person shown in a frame image moves while reducing transmission load of image data. It is to provide.

According to the image display system of the present invention, the subject includes (A) an imaging device for imaging a first user, and (B) a measurement device for measuring a measurement target value regarding the position of each part of the first user. (C) a first computer for acquiring a frame image constituting an image of the first user captured by the imaging device; (D) a second computer for communicating with the first computer for acquiring the frame image; (E) a display for displaying the frame image acquired by the second computer to a second user located at a place different from the first user, and (F) the first computer (F1) The above-mentioned each of the body parts based on the change in the measurement result of the measurement target value during the period from the previous acquisition of the frame image to the current acquisition of the frame image. (F2) a process of extracting an area including the identified part from the person image of the first user in the frame image acquired this time by the first computer; And (f3) a process of generating image data of the area and transmitting it to the second computer, and (G) the second computer receives the image data of the area, The display unit is configured to display the frame image configured by superimposing the image of the area on a position corresponding to the area among the frame images displayed on the display before receiving the image data. Solved by

In the image display system of the present invention configured as described above, based on the change in the measurement result of the measurement target value regarding the position of each part of the body of the first user, in the body of the first user, from the time of previous frame image acquisition A portion (that is, a specified portion) moved during the period up to the frame image acquisition time this time is identified. That is, in the image display system of the present invention, the identified part is identified based on the change in the measurement result of the measurement target value, so that the identified part can be identified accurately.
On the other hand, the first computer extracts an area including the specified part from the person image of the first user in the frame image acquired this time, and transmits the image data of the area to the second computer. As a result, it is possible to reduce the data transmission load as compared to the case of transmitting the image data of the entire person image of the first user.
As described above, according to the image display system of the present invention, it is possible to appropriately identify a moved part (that is, a specified part) in the body of a person shown in a frame image while reducing the transmission load of image data. It becomes possible.

Further, to describe a suitable configuration of the image display system according to the present invention, in the process of identifying the specified portion, the first computer determines the configuration based on a change in the measurement result of the measurement target value during the period. Of the plurality of setting sites set in the skeleton of the first user, the setting site moved during the period may be specified, and the specified portion may be specified to include at least the setting site.
In the above configuration, it becomes possible to specify the specified part by looking at the presence or absence of movement for the plurality of set parts set in the skeleton of the first user. With such a configuration, it is possible to identify the specified part more easily because it is sufficient to confirm the presence or absence of the movement at each setting site when specifying the specified part.

Further, to describe a more preferable configuration of the image display system according to the present invention, when the first computer specifies the set site moved during the period in the process of specifying the specified portion, the first computer may The determination as to whether the displacement amount of the setting site is equal to or more than a threshold may be performed for each setting site, and the setting site having the displacement amount equal to or more than the threshold may be specified as the setting site moved during the period.
In the above configuration, it is determined for each set site whether the displacement amount of the set site is equal to or greater than a threshold. Through such determination for each set site, it is possible to more easily determine the presence or absence of movement in each set site, in other words, the specified site.

Further, to describe a further preferable configuration of the image display system of the present invention, when the first computer performs the determination for each of the set regions, the first set region after the determination of the predetermined region. When performing the determination on the set site located next to the target area and extracting the area including the specified part, the set site moved during the period is included in the area. It is good to extract the area.
In the above configuration, after the determination on a certain set site, the determination on the set site located next to the certain set site is performed. Then, when extracting the area including the specified part, the area is extracted so as to include all the set parts moved during the period from the previous frame image acquisition to the current frame image acquisition. As a result, in the person image of the first user, the image of the moved part is properly extracted. Then, by superimposing the image of the extracted area on the previous display image (frame image), the frame image acquired this time by the first computer (strictly speaking, the person image of the first user in the frame image) is appropriately used. It is possible to reproduce in

Further, to describe a further preferable configuration of the image display system of the present invention, at least one of the plurality of set parts set is a part on the body axis of the upper body of the first user, The one computer may extract the image of the upper body as the area when it is determined that the displacement amount is equal to or more than the threshold value in the determination of the set region on the body axis.
In the above configuration, when it is determined that the displacement amount of the set region on the body axis is equal to or more than the threshold, the image of the upper body is extracted as a region. By performing area extraction in units of upper body images in this manner, processing relating to area extraction can be performed more simply.

Further, to describe the still more preferable configuration of the image display system of the present invention, the first computer generates background image data indicating a background image in the frame image separately from image data other than the background image. And the frequency at which the first computer executes the process of transmitting the background image data is the frequency at which the first computer acquires the frame image from the imaging device. Better than less.
In the above configuration, the background image data indicating the background image in the frame image is generated separately from the image data other than the background image and transmitted to the second computer. Further, the transmission frequency of the background image data is smaller than the frequency at which the first computer acquires the frame image from the imaging device. This is because this generally reflects that the change in the background image is small. That is, for the image data of the background image, the number of transmissions can be reduced. For this reason, it is possible to further reduce the data transmission load by setting the transmission frequency of background image data to be lower than the acquisition frequency of frame images as in the above configuration.

Further, to describe the configuration of the image display system according to the present invention, the positional relationship between the second user and the display and the posture of the second user in the state where the second user is in front of the display. Information providing apparatus for providing the second computer with information related to at least one content, the first computer processing for acquiring the at least one content specified by the second computer from the information Further, in the process of generating the image data of the area, the first image displayed in the range of the area determined according to the content of the at least one in the image of the area is more The image data of the area may be generated such that a second image displayed in a different area has a low image quality.
In the above configuration, the image quality of an image other than an image in a predetermined range (for example, an image in the central visual field of the second user when displayed on the display) in the extracted area is degraded. This reflects that the images other than the central visual field are difficult to be visually recognized, so even if the image quality of the image is relatively low, the influence on the sense of realism of the dialogue felt by the second user is small. It is because Therefore, according to the above configuration, it is possible to further reduce the data transmission load. Such an effect is more effectively exhibited as the extracted area becomes wider.

Further, to describe the particularly preferable configuration of the image display system according to the present invention, a distance measuring device for measuring a distance between the second user and the display in a state where the second user is in front of the display. And the first computer acquires the measurement result of the distance from the second computer, and reduces the image quality of the person image to a predetermined image quality when the distance is greater than or equal to a preset size. Preferably, low-quality person image data representing the person image of the image quality after deterioration is generated and transmitted to the second computer.
In the above configuration, when the distance between the second user and the display is equal to or greater than the preset size, the image quality of the person image of the first user is degraded, and data indicating the person image of the image quality after degradation (Low-quality human image data) is generated and transmitted to the second computer. This reflects that the influence on the sense of realism of the dialogue felt by the second user is small even if the image quality of the image displayed on the display is slightly degraded when the above distance becomes larger than the set value. It is because Therefore, according to the above configuration, it is possible to reduce the data transmission load while securing the sense of reality of the dialogue.

Another object described above, according to the image display method of the present invention, a first computer imaging apparatus acquires a frame image constituting the video of a first user who has captured the in order to obtain the frame image first An image display method for displaying the frame image acquired by the second computer using a display to a second user who is at a different place from the first user using a second computer in communication with the one computer (A) the first computer executes processing for acquiring the measurement result of the measurement target value from a measurement device that measures the measurement target value regarding the position of each part of the first user's body; B) The first computer measures the measurement target value during the period from the previous acquisition of the frame image to the current acquisition of the frame image. Performing, on the basis of the change in the part of the body, a process for identifying a specified part that has moved during the period; and (C) the first computer in the frame image acquired this time. Performing a process of extracting an area including the specified part in the person image of the user; and (D) the first computer generates image data of the area and transmits it to the second computer (E) when the second computer receives the image data, a position corresponding to the area in the frame image displayed on the display before the image data is received And displaying the frame image configured by superimposing the image of the area indicated by the image data on the display.
According to the above method, while reducing the transmission load of the image data, in the person image of the first user in the frame image, it is possible to appropriately identify the moved part (that is, the identified part) in the body of the first user. Is possible.

  According to the image display system and the image display method of the present invention, it is possible to appropriately identify a portion of the person image of the user in the frame image that has moved within the body of the user while reducing the transmission load of the image data. Is possible. As a result, while realizing smoother image data transmission and reception, it is possible to secure a sense of reality (real feeling) of the dialogue performed while displaying the person image of the dialogue partner on the display.

FIG. 1 is a schematic view of an image display system according to an embodiment of the present invention. It is a figure showing equipment composition of a communication unit which constitutes an image display system. It is a figure which shows the frame image and depth data of the imaging | video which the imaging device imaged. It is a figure which shows the state of the indicator used in one Embodiment of this invention, and (A) in the figure shows the state at the time of non-interaction, and each shows the state at the time of interaction. It is an explanatory view about separation and composition of a background image and a person image. (A), (B) and (C) of FIG. 6 are explanatory diagrams of the image quality reduction process. (A), (B), (C) and (D) of FIG. 7 are explanatory diagrams related to clipping of an image. It is explanatory drawing about an image quality adjustment process. It is a figure showing the flow of the dialogue communication flow. It is a figure showing the flow of communication pre-processing. It is a figure showing the flow of present information notification processing. It is a figure showing the flow of image processing transmission processing. It is a figure showing the flow of selection processing of a logging field. It is a figure showing the flow of calculation processing of a logging field. It is a figure showing the flow of image quality adjustment processing. It is the figure which showed the flow of the reconstruction process of a display image.

  Hereinafter, an embodiment of the present invention (hereinafter, the present embodiment) will be described. The embodiments described below are merely examples for facilitating the understanding of the present invention, and are not intended to limit the present invention. That is, the present invention can be modified and improved without departing from the gist thereof, and the present invention naturally includes the equivalents thereof.

<< Application of image display system according to the present embodiment >>
First, the application of the image display system (hereinafter, the present system S) according to the present embodiment will be outlined. The present system S is used to allow users in distant places to interact with each other while looking at each other. That is, in a dialogue (hereinafter, dialogue communication) using the present system S, each user comes to feel as if he / she is actually talking to the other party. In the following description, the above-mentioned visual effect will be referred to as realism.

Note that the dialog communication of this embodiment is performed when each user is in a predetermined room (own room) of each home. However, the present invention is not limited to this, and it is useful when the user is in a place other than home, for example, a meeting place or commercial facility, a school classroom or learning school, a public facility such as a hospital, a company or office, etc. Dialogue communication may be performed by the system S. In addition, interactive communication may be performed when users in the same building are in different rooms in the building.
As described above, the present system S can be widely used in situations where persons in different places interact with each other while looking at the face of the other.

  Hereinafter, a case in which the user A and the user B communicate with each other will be described as an example. Also, in the following, it will be explained from the viewpoint of Mr. B (in other words, the position to see Mr. A's figure). In such a case, Mr. A corresponds to the "first user", and Mr. B corresponds to the "second user". Here, “first user” and “second user” are relative concepts that are switched according to the relationship between the viewer and the viewer of the image. It corresponds to the "first user" and Mr. A corresponds to the "second user".

  Both Mr. A and Mr. B will enter their own room when conducting dialogue communication. Specifically, a mirror-type display (specifically, the display 5 shown in FIG. 2) is disposed in each room. Mr. A and Mr. B move to the front position of the display when performing dialogue communication. At this time, if the present system S is activated, interactive communication is started. The system activation timing is not particularly limited, and may be a timing different from the above contents as long as it is a suitable timing.

  When the dialog communication is started, the image of Mr. A is displayed on the display of Mr. B's side. This image is an image captured by a camera 2 (corresponding to an imaging device) provided on the side of Mr. A, and described strictly, it is a frame image constituting a video of Mr. A captured by the camera 2. That is, the image displayed on the display on the side of Mr. B is switched at a constant speed (specifically, a speed corresponding to an acquisition speed of a frame image). As a result, a continuous image of Mr. A, that is, an image, is displayed on the display, and Mr. B feels as if he / she is facing Mr. A (realism).

  By the way, the whole body image of Mr. A is to be displayed in a life-size on the indicator of Mr. B's side. Specifically, as described above, the display is configured by the mirror-type display 5 and has the same shape and size as a general look, and displays a full-length image of Mr. A in a full-size view It has a shape and size suitable for With this configuration, Mr. B sees life-size Mr. A reflected in the display, and feels as if he is meeting Mr. A through the glass.

<< About the Configuration of the Image Display System According to the Present Embodiment >>
Next, the specific configuration of the present system S will be described. The present system S is configured by a unit for information communication (hereinafter referred to as a communication unit) prepared in both A's home and B's home. Specifically, the present system S is configured by a first communication unit 100A used by Mr. A at Mr. A's home and a second communication unit 100 B used by Mr. B at Mr. B's home. The configurations of the first communication unit 100A and the second communication unit 100B will be described below.

  Note that “first communication unit 100A” and “second communication unit 100B” are concepts determined in accordance with the relationship between the first user and the second user described above, and when Mr. A is viewed as the first user, The communication unit used by Mr. A corresponds to the first communication unit 100A, and the communication unit used by Mr. B corresponds to the second communication unit 100B. Conversely, when Mr. A is viewed as the second user, the communication unit used by Mr. B corresponds to the first communication unit 100A, and the communication unit used by Mr. A corresponds to the second communication unit 100B.

  The first communication unit 100A and the second communication unit 100B have substantially the same mechanical configuration, and to be specific, as shown in FIG. 1, the home server 1, the camera 2, and the microphone 3 in any unit. The infrared sensor 4, the display 5 and the speaker 6 are mounted. Among these devices, the camera 2, the microphone 3, the infrared sensor 4, the display 5 and the speaker 6 are disposed in the room of each user at home (the room to be entered when performing face-to-face interaction). FIG. 1 is a conceptual view showing the configuration of the present system S. As shown in FIG.

  The home server 1 is a device that forms the core of the present system S, and is a computer having a CPU, a memory such as a ROM and a RAM, a communication interface, a hard disk drive, and the like. The home server 1 of the first communication unit 100A corresponds to a first computer, and the home server 1 of the second communication unit 100B corresponds to a second computer.

  Further, a program for interactive communication is installed in the home server 1. By the CPU executing this program, the home server 1 exerts an interactive communication function described later. The home servers 1 are communicably connected via an external communication network GN such as the Internet, and transmit / receive various data to / from each other. Here, the data transmitted and received by the home server 1 is data necessary for dialogue communication, and is, for example, image data and audio data of various images.

  The camera 2 is an imaging device that captures an image of an object within an imaging range (angle of view), and is configured by a known network camera in the present embodiment. In addition, when the user (Mr. A, Mr. B) stands in front of the display 5, the camera 2 captures a full-length image of the user. That is, the camera 2 included in the first communication unit 100A captures an image of Mr. A and the periphery thereof when Mr. A stands in front of the display 5 installed in Mr. A's room. Similarly, when Mr. B stands in front of the display 5 installed in Mr. B's room, the camera 2 included in the second communication unit 100B images Mr. B and the periphery thereof.

  In the present embodiment, as shown in FIG. 2, the lens of the camera 2 faces the display screen 5 a of the display 5. Here, the mirror surface panel of the display 5 constituting the display screen 5a is made of transparent glass. Therefore, the camera 2 captures an image of the user standing in front of the display 5 through the above-mentioned mirror panel. FIG. 2 is a view showing the device configuration of each communication unit, and an explanatory view of the arrangement position of each device. However, the arrangement position of the camera 2 is not limited to the position illustrated in FIG. 2 and may be a position away from the display 5.

  By the way, when the user is not standing in front of the display 5, the camera 2 is to image the internal space of the room in which the camera 2 is installed (strictly speaking, the range within the angle of view of the camera 2). ing. The frame image of the video captured at this time is to be used as a "background image".

  Then, the frame image constituting the captured image of the camera 2 is digitized and transmitted to the home server 1 (strictly speaking, the home server 1 belonging to the same communication unit).

  The microphone 3 is a device for collecting a sound generated in a room in which the microphone 3 is installed, such as a user's speaking voice. Then, the microphone 3 outputs an audio signal indicating the collected sound to the home server 1 (strictly, the home server 1 belonging to the same communication unit). In the present embodiment, as shown in FIG. 2, a microphone is installed at a position immediately above the display 5.

  The infrared sensor 4 is a so-called depth sensor, and is a sensor that measures the depth of an object to be measured by an infrared method. Specifically, the infrared sensor 4 irradiates infrared light from the light emitting unit 4 a toward the measurement object, and measures the depth by receiving the reflected light by the light receiving unit 4 b. Here, “depth” is the distance from the reference position to the measurement object (ie, the depth distance). Incidentally, in the present embodiment, the position of the display screen 5a (front surface) of the display 5 is set as the reference position. That is, the infrared sensor 4 measures the distance between the measurement object in the normal direction of the display screen 5a and the display screen 5a as the depth. However, the reference position is not limited to the above position, and can be set to any position.

  In addition, the measurement result of the depth is obtained for each of the pixels when the frame image of the image captured by the camera 2 is divided into a predetermined number of pixels. Then, the depth data illustrated in FIG. 3 can be obtained by collecting the measurement results of the depth obtained for each pixel in frame image units. This depth data is data indicating the measurement result of the depth for each pixel in the frame image, and as shown in FIG. 3, bit map data obtained by setting the color and density of each pixel according to the measurement result of the depth It has become. FIG. 3 is a diagram showing a frame image and depth data of the frame image.

  Describing in more detail about depth data, depth data is to be acquired for each of the frame images that constitute the captured image of the camera 2. Further, as shown in FIG. 3, in the depth data, pixels belonging to the image of the subject positioned on the back side in the frame image (black pixels in the figure) and pixels belonging to the image of the subject positioned on the front side ( Of course, in the case of white pixels in the figure, the measurement results of depth differ. By using such a property, it is possible to distinguish and separate the pixels belonging to the background image and the pixels belonging to the human image among the pixels constituting the depth data.

  The above infrared sensor 4 is installed in both the room A and the room B. That is, when Mr. A stands in front of the display 5 installed in Mr. A's room, the infrared sensor 4 of the first communication unit 100A measures the depth of each part of Mr. A's body. That is, the infrared sensor 4 of the first communication unit 100A corresponds to a measurement device that measures depth as a measurement target value regarding the position of each part of the person A.

  Similarly, when Mr. B stands in front of the display 5 installed in Mr. B's room, the infrared sensor 4 of the second communication unit 100 B measures the depth of each part of Mr. B's body. That is, the infrared sensor 4 of the second communication unit 100B corresponds to a distance measurement device that measures the depth in a state where Mr. B is in front of the display 5, in other words, the distance between Mr. B and the display 5.

  The device (measuring device) that measures the measurement target value regarding the position of each part of the body is not limited to the infrared sensor 4; for example, a sensor (motion that is mounted on the user and directly measures the position of each part of the body It may be a sensor for capture. Further, the method of measuring the distance to the display 5 is not limited to the method of using the infrared sensor 4. For example, the standing position of the user is detected by a sensor or the like. The distance between and may be measured. Alternatively, the distance may be determined by analyzing a captured image of the camera 2.

  The speaker 6 is a device that emits sound (reproduction sound) to be reproduced by expanding the sound data received by the home server 1. Specifically, when the home server 1 of the first communication unit 100A receives voice data from the home server 1 of the second communication unit 100B, the home server 1 expands the voice data, and the voice collected in Mr. B's room is collected. Are reproduced by the speaker 6. On the other hand, when the home server 1 of the second communication unit 100B receives the audio data from the home server 1 of the first communication unit 100A, it expands the audio and reproduces the audio collected in the room of Mr. A by the speaker 6 Let In the present embodiment, as shown in FIG. 2, a plurality of (four in FIG. 2) speakers 6 are provided at positions sandwiching the display 5 in the widthwise direction of the display 5.

  The display 5 is a display for displaying the frame image acquired by the home server 1 on the display screen 5a. More specifically, the display 5 of the first communication unit 100A displays the frame image acquired by the home server 1 of the first communication unit 100A to Mr. A. On the other hand, the display 5 of the second communication unit 100B displays the frame image acquired by the home server 1 of the second communication unit 100B to Mr. B.

  Moreover, the display 5 which concerns on this embodiment is comprised by the display of a mirror type as mentioned above. Furthermore, the display 5 according to the present embodiment normally functions as furniture arranged in a room as shown in FIG. That is, since the frame image is not displayed on the display screen 5a of the display 5 at the time of non-interaction (when the dialogue communication is not performed), the display screen 5a functions as a mirror surface. On the other hand, at the time of dialogue (when dialogue communication is performed), as shown in FIG. 4B, a frame image is displayed (reproduced) on the display screen 5a. (A) and (B) of FIG. 4 are diagrams showing a configuration example of the display 5 according to the present embodiment, (A) showing the non-interactive state and (B) showing the interactive state. It shows.

  As described above, the display 5 according to the present embodiment is used as a look at non-interaction, and displays a frame image on the display screen 5a at the time of meeting. This makes it difficult to notice the presence of the display screen 5a when not interacting. On the other hand, at the time of dialogue, the user is made to feel the visual effect as if they were facing the other party of the dialogue through the glass.

  In addition, about the structure which combines the display of an image and a look, a well-known structure like the structure described, for example in the international publication 2009/122716 can be utilized. In addition, the display 5 is not limited to the configuration used as a look-aside. The device used as the display 5 may have a size sufficient to display a full-length image of the other party. And from the viewpoint of making it hard to notice the presence of the display screen 5a at the time of non-interaction, the other furniture or building material installed in the room and having a mirror surface portion is preferable, for example, a door (glass door) or A window (glass window) may be used as the display 5. In addition, about the display 5, it is not limited to what is also used as furniture and a building material, The normal display which always forms the display screen 5a may be sufficient during starting.

<< Home Server Functions >>
Next, the interactive communication function provided in the home server 1 of each communication unit will be described. In the following, among the interactive communication functions, only the function related to image display will be described, and the description of functions related to sound reproduction and the like will be omitted. Also, in the following, in order to make the description easy to understand, an example is shown in which the image delivered from Mr. A (that is, the first communication unit 100A) is displayed by Mr. B (that is, the second communication unit 100B). I will list and explain. In addition, the contents to be described below hold true even when the viewpoint is changed. That is, in the following description, the function of the home server 1 of the first communication unit 100A is also included in the home server 1 of the second communication unit 100B, and the function of the home server 1 of the second communication unit 100B is , And the home server 1 of the first communication unit 100A.

The home server 1 of the first communication unit 100A functions as a server on the image distribution side, and specifically includes the following functions (1) to (5).
(1) Frame image acquisition function (2) Skeletal model identification function (3) Current information identification / notification function (4) Opposite party visual field estimation function (5) Image processing / transmission function

Further, the home server 1 of the second communication unit 100B functions as a server on the image display side, and specifically, has the following function (6).
(6) Display Image Reconstruction Function Hereinafter, each function will be described in detail.

(Frame image acquisition function)
The home server 1 of the first communication unit 100A acquires a frame image captured by the camera 2 at an interval corresponding to the frame rate of the camera 2 belonging to the unit. More specifically, when Mr. A is in front of the display 5 in a room (strictly speaking, a room that enters the room during interactive communication), the camera 2 images Mr. A and the background thereof. For this reason, the home server 1 acquires a frame image including a person image of Mr. A and its background image. On the other hand, when Mr. A is not in the room, the home server 1 acquires a frame image consisting only of the background image (image of the internal space of the room).

  When acquiring a frame image, the home server 1 of the first communication unit 100A acquires depth data for the frame image. As described above, the depth data of the frame image indicates the measurement result of the depth for each pixel when the frame image is divided into predetermined pixels, and more specifically, the bit map illustrated in FIG. 3 It consists of data.

(Skeletal model identification function)
As described above, each time the frame image is acquired, the home server 1 of the first communication unit 100A acquires depth data of the frame image. Then, the home server 1 specifies the skeletal model of Mr. A based on the frame image (strictly, the person's human image in the frame image) and the depth data of the frame image. Specifically, in depth data of a frame image including a person image of Mr. A, as shown in FIG. 3, pixels belonging to the person image (in FIG. 3, white pixels) and the other images are included. The depth is obviously different from that of the pixel to which the pixel belongs (in FIG. 3, the black pixels and the hatched pixels). Using such a feature, the home server 1 extracts pixels belonging to a person image in depth data. Then, the home server 1 specifies a skeleton model of Mr. A from the extracted pixels.

  As shown in FIG. 3, the skeletal model is a simplified model of position information on the human skeleton, in particular, the head, shoulders, elbows, hands, legs, hips, hips, knees, and feet. Here, the above-described site set in the skeletal model corresponds to the "set site" in the present invention. In addition, the setting site includes a site on the body axis of the upper body of the first user, and specifically, the head and the waist correspond. Incidentally, as a method of specifying a skeletal model, known methods (for example, methods described in Japanese Patent Application Laid-Open Nos. 2014-155693 and 2013-116311) can be used.

  Then, the home server 1 of the first communication unit 100A specifies the skeleton model each time depth data is acquired, in other words, each time a frame image is acquired. Thereby, it is possible to detect the presence or absence of movement (displacement) for the change in position of each part of the body of Mr. A represented as a skeletal model, more specifically, for each of a plurality of setting parts set in the skeletal model. Become.

  Further, as shown in FIG. 3, the home server 1 of the first communication unit 100A can extract a person image from the frame image based on the skeleton model specified from the depth data of the frame image. It is. In the present specification, a description of a method of extracting a human image from a frame image based on a skeletal model is omitted, but if a rough procedure is described, a person in depth data based on a skeletal model identified Identify pixel groups that belong to the image. Thereafter, an area corresponding to the specified pixel group is extracted from the frame image. The image extracted by this procedure corresponds to the person image in the frame image.

(Current information identification / notification function)
The home server 1 of the first communication unit 100A specifies information (hereinafter referred to as current information) related to the current state of the user A in dialog communication, and transmits the current information to the home server 1 of the second communication unit 100B. . Here, the "current information" refers to the content related to at least one of the positional relationship between the user A and the user in the state of being in front of the display 5 and the posture of the user A, and the present embodiment Then, it is the distance between A and the display 5 (depth distance), the height of A, and the direction of A's face. The contents specified as current information are not limited to the above contents, and include other information such as the direction of the line of sight of Mr. A and the position of the face (position in both the vertical direction and the horizontal direction) It may be

  Regarding the method of identifying each piece of current information, regarding the distance between Mr. A and display 5, the measurement result of the depth when infrared sensor 4 measures with Mr. A standing in front of display 5, ie, , And can be specified from depth data. That is, the home server 1 of the first communication unit 100A specifies the distance between the user A and the display 5 based on the measurement result of the infrared sensor 4. In other words, it can be said that the infrared sensor 4 corresponds to the information providing device that provides the home server 1 with the measurement result of the depth as the information on the distance between the user A and the display 5.

  The height of Mr. A can be identified based on the distance between Mr. A and the display 5 identified by the above method and the skeletal model identified from the depth data. More specifically, the home server 1 of the first communication unit 100A determines the height of Mr. A on the skeletal model (hereinafter, the height on the model). Further, the home server 1 calculates the ratio of the height on the model to the actual height of Mr. A from the distance between Mr. A and the display 5. Then, the home server 1 specifies the height (actual height) of Mr. A based on the calculated height on the model and the calculated ratio.

  The direction of the face of Mr. A can be specified from a frame image when the camera 2 captures an image while Mr. A is standing in front of the display 5. More specifically, the home server 1 of the first communication unit 100A applies known image analysis processing to the above-described frame image to specify the direction of Mr. A's face. In other words, it can be said that the camera 2 corresponds to an information providing device that provides the home server 1 with a frame image including a person's image of the person A as information on the posture (the direction of the face) of the person A.

  After specifying the three pieces of current information, the home server 1 of the first communication unit 100A notifies the home server 1 of the second communication unit 100B of these. On the other hand, identification and notification of current information are similarly performed in the home server 1 of the second communication unit 100B. That is, the home server 1 of the second communication unit 100B specifies the distance between the B and the display 5, the height of the B and the direction of the B's face while the B is in front of the display 5. , These are notified to the home server 1 of the first communication unit 100A. The infrared sensor 4 of the second communication unit 100B provides the home server 1 with information on the distance between the user B and the display 5, more specifically, the measurement result of the depth, as an information providing device. In addition, the camera 2 of the second communication unit 100B provides the home server 1 with information on the posture (face orientation) of Mr. B, more specifically, a frame image including Mr. B's person image, as an information providing device. .

  Then, when the home server 1 of the second communication unit 100B notifies the present information of Mr. B, the home server 1 of the first communication unit 100A notifies the present information (that is, the home server 1 of the second communication unit 100B). The content specified based on the provided information from the infrared sensor 4 or the camera 2 is acquired.

(The other party's view estimation function)
The home server 1 of the first communication unit 100A estimates a region corresponding to the field of view of the user B, more specifically, a range corresponding to the central region of view, based on the acquired current information of the user B. More specifically, the home server 1 determines the height (view height) and the direction (view direction) of Mr. B's eyes from the information on the height and the direction of the face of Mr. B. Then, the home server 1 specifies a range expanded by a predetermined angle (viewing angle) based on the virtual line extending from the above-mentioned eye height toward the above-mentioned eye direction. This range corresponds to a range corresponding to the central visual field of Mr. B (hereinafter, referred to simply as a central visual field).

  The home server 1 of the first communication unit 100A estimates the central visual field area of Mr. B by the above method, and then stores the position indicating the estimation result. Here, the "position indicating the estimation result" is the relative position of the central visual field of Mr. B with respect to the display screen 5a of the display 5 of the second communication unit 100B.

  As described above, in the present embodiment, it is possible to appropriately estimate the central visual field of the conversation partner based on the height and the face orientation of the conversation partner. In addition, as a method of estimating a central visual field area, it is not limited to said method, As long as it is a method suitable for estimating a central visual field area, you may employ | adopt another method.

(Image processing and transmission function)
The home server 1 of the first communication unit 100A directs the image data to the home server 1 of the second communication unit 100B in order to display a frame image including a person image of Mr. B on the display 5 of the second communication unit 100B. Send. Here, the image data to be transmitted will be described. In order to ensure the presence of interactive communication, in principle, high-quality image data is transmitted. On the other hand, the transmission load at the time of data transmission (hereinafter, data transmission load) becomes larger as the image data is of higher quality. Therefore, the home server 1 of the first communication unit 100A performs predetermined processing on the frame image acquired from the camera 2 in order to reduce the data transmission load, and the data (image data) of the processed image is obtained. It is supposed to be sent.

  Hereinafter, processing for reducing the data transmission load will be described with reference to FIGS. 5 to 8. FIG. 5 is an explanatory diagram of processing for separating a background image of a frame image and a person image. (A), (B) and (C) in FIG. 6 are explanatory diagrams of the image quality reduction process, (A) in the figure shows the positional relationship between the B and the display 5, (B) Shows a display image of the display 5 when Mr. B is at a position close to the display 5, and (C) shows a display image of the display 5 when Mr. B is at a position away from the display 5. There is. (A), (B), (C) and (D) of FIG. 7 are explanatory diagrams related to clipping of an image selected from among frame images, and (A) in the figure is a frame image of the previous time. It is a figure which contrasted with this frame picture, (B) is a figure which contrasted the last skeleton model and this skeleton model, (C) was cut out as a transmitting object out of this frame image. (D) is a figure which shows the procedure which reconstructs a display image using the cut-out image. FIG. 8 is an explanatory view of the image quality adjustment process.

  First, the image separation processing will be described with reference to FIG. When interactive communication is started, the home server 1 of the first communication unit 100A acquires frame images (captured images) sequentially sent from the camera 2. Then, when the person image of Mr. A and its background image are included in the acquired frame image, the home server 1 extracts the person image from the frame image as shown in FIG. 5, and the person image and the background image And separate. Then, the home server 1 transmits only image data of a person image.

  On the other hand, the image data of the background image is generated separately from the image data other than the background image, and is transmitted to the home server 1 of the second communication unit 100B. In the present embodiment, the execution frequency of the transmission process of the background image data is smaller than the frequency at which the home server 1 of the first communication unit 100A acquires a frame image from the camera 2.

  More specifically, the home server 1 of the first communication unit 100A acquires a frame image consisting of only a background image from the camera 2 immediately after the start of the dialogue communication or in the pre-communication processing to be described later. After obtaining the frame image, the home server 1 transmits the image data of the frame image as the image data of the background image. Thereafter, the home server 1 does not transmit the image data of the background image until the interactive communication ends. The reason why the transmission of the image data of the background image is limited to the start time of the dialogue communication and the like in this way is that it generally reflects that the change in the background image is small.

  When the home server 1 transmits image data of the background image once at the start of the dialogue communication, the home server 1 transmits only the image data of the person image in the frame image thereafter, and transmits the image data of the background image. do not do. This makes it possible to reduce the data transmission load as compared with the case of transmitting image data of the entire frame image (that is, both image data of a person image and a background image).

  The separated background image and person image are recombined by the home server 1 of the second communication unit 100B. More specifically, the home server 1 of the second communication unit 100B receives the image data of the background image transmitted by the home server 1 of the first communication unit 100A at the time of interactive communication, and the person image transmitted thereafter. The image data of (1) is received and expanded, and an image (synthetic image) in which both images are synthesized is constructed. The composite image substantially matches the frame image at the time when the home server 1 of the first communication unit 100A acquires it from the camera 2, that is, the frame image before being separated into the human image and the background image.

  The home server 1 of the second communication unit 100B acquires a new frame image by combining the background image and the person image as described above. Then, the newly acquired frame image is displayed on the display 5 as a display image of this time.

  Next, the image quality reduction processing will be described with reference to (A), (B) and (C) of FIG. As described above, the home server 1 of the first communication unit 100A extracts the person image of Mr. A from the frame image acquired from the camera 2, and transmits the data of the person image. On the other hand, the home server 1 of the first communication unit 100A acquires the distance between the user B and the display 5 from the home server 1 of the second communication unit 100B as current information of the user B.

  Then, when the distance between the user B and the display 5 is less than the threshold (for example, when it is the distance indicated by the symbol d1 in FIG. 6A), the home server 1 of the first communication unit 100A The image data which displays the extracted person image with the image quality as it is is produced | generated, and the said image data is transmitted toward the home server 1 of 2nd communication unit 100B. Here, the threshold is a reference value when determining whether or not the image quality reduction processing is to be performed, and is a value of a preset magnitude with respect to the above distance. The specific value of the threshold is not particularly limited, but is preferably set to a value suitable for determining the execution of the image quality reduction process.

  On the other hand, when the distance between Mr. B and the display 5 is equal to or larger than the threshold (for example, when it is the distance indicated by symbol d2 in (A) of FIG. 6), the home server 1 of the first communication unit 100A The image quality reduction process is performed on the extracted human image. In this image quality reduction process, the image quality of the extracted person image is reduced to a predetermined image quality, and image data (hereinafter, low image quality person image data) indicating a person image of the image quality after reduction is generated. Here, "reduce the image quality" means to reduce the resolution. Further, with regard to the “predetermined image quality” described above, the image quality of the frame image at the time when at least the home server 1 of the first communication unit 100A acquires from the camera 2, that is, the image quality lower than the image quality of the original image In addition, desirably, the image quality should be set to a degree that does not impair the sense of reality of interactive communication.

  Then, after the low-quality person image data is generated, it is transmitted to the home server 1 of the second communication unit 100B. The data transmission load at this time is reduced by the amount of image quality degradation.

  As described above, the image quality of the person image distributed by the home server 1 of the first communication unit 100A when the distance between Mr. B and the display 5 is equal to or greater than the threshold and when the distance is less than the threshold. Will be different. For this reason, the image quality of the human image in the frame image (that is, the composite image of the human image and the background image) displayed on the display 5 of the second communication unit 100B also changes in accordance with the above distance. Specifically, when the distance between Mr. B and the display 5 is less than the threshold, as shown in FIG. 6B, the person image in the display image of the display 5 is the first communication The image quality is substantially the same as the human image in the frame image (original image) acquired by the home server 1 of the unit 100A from the camera 2.

  On the other hand, when the distance between Mr. B and the display 5 is equal to or larger than the threshold, as shown in FIG. 6C, the person image in the display image of the display 5 is the one of the first communication unit 100A. The image quality is somewhat lower than that of the person image in the frame image acquired by the home server 1 from the camera 2 (low resolution). However, in this case, even if the image quality of the human image in the display image is lowered, Mr. B looking at the display 5 is far from the display 5, and therefore does not feel a sense of discomfort due to the image quality deterioration. That is, if the above distance is equal to or more than the threshold value, the dialog communication is performed even if the low image quality person image data is transmitted to the home server 1 of the second communication unit 100B by executing the image quality reduction processing on the person image. The sense of reality (realism) is not impaired. As a result, it is possible to reduce the data transmission load by an amount corresponding to the reduction in image quality while securing the presence of interactive communication while reducing the image quality of the person image in the display image.

  Next, clipping of an image will be described with reference to (A), (B), (C) and (D) of FIG. 7. The home server 1 of the first communication unit 100A extracts the person image of Mr. A from the frame image acquired from the camera 2 as described above. Thereafter, the home server 1 generates image data of the extracted person image. At this time, when the distance between the user B and the display 5 is less than the threshold value, as described above, the image data of the human image is generated so as to have the same image quality as the original image. Such image data causes a larger data transmission load due to the higher image quality.

  On the other hand, as shown in FIG. 7A, when two frame images (previous frame image and current frame image) acquired continuously are compared, a frame image is displayed for the person image in the frame image. There are parts that differ between the two, and parts that are common to the frame images. That is, of the two frame images, in the human image in the frame image acquired this time, there are a portion moved from the frame image acquired last time and a portion not moved.

  Then, the home server 1 of the first communication unit 100A cuts out the image of the moved part of the person image in the frame image acquired this time, generates the image data of the cut out image, and generates the home of the second communication unit 100B. It is supposed to be sent to the server 1. Here, the “image of the moved part” refers to an image of a part of the body of Mr. A who moved during the period from the time of acquisition of the previous frame image to the time of acquisition of the current frame image. .

  As described above, in the present embodiment, the image data of the moving part of the person image in the frame image acquired this time is transmitted to the home server 1 of the second communication unit 100B. This makes it possible to reduce the image data of the transmitted person image by the amount of the image data of the non-moving part in the person image. As a result, it is possible to further reduce the data transmission load when transmitting the image data of a person image.

  By the way, when generating the image data of the moved part, a part of Ms. A's body that moved during the period from the time of acquisition of the previous frame image to the time of acquisition of the current frame image (hereinafter referred to as specified Part) needs to be identified. And in this embodiment, when specifying a to-be-specified part, it is supposed that a to-be-specified part is specified based on the change of the measurement result of the infrared sensor 4 of 1st communication unit 100A in said period.

  More specifically, as shown in FIG. 7B, the skeletal model is specified from each of the depth data for the previously acquired frame image and the depth data for the currently acquired frame image. Then, the identified part is identified by comparing the two skeletal models. Incidentally, in the case illustrated in FIG. 7B, the hand and the elbow are identified as the identified part. A specific procedure for specifying the specified part will be described later.

  As described above, in the present embodiment, when identifying the specified part (that is, the moved part in the body of A) in the person image of A in the frame image, the skeletal model is compared with each other for the skeletal model. Identify the identified part from the difference (change) between As a result, the specified part can be properly and accurately identified.

  After specifying the specified part, the home server 1 of the first communication unit 100A includes an area including the specified part in the person image of Mr. A in the frame image acquired this time (hereinafter, also referred to as cutout area or cutout image) Extract Specifically, the home server 1 extracts the cutout region so as to include the set portion that has moved during the period from the time of acquisition of the previous frame image to the time of acquisition of the current frame image. When the case of FIG. 7B is described as an example, when the hand and the elbow are identified as the identified part, the home server 1 displays a person image of Mr. A, as shown in FIG. 7C. Images of the range from the hand to the elbow (ie, the hand and forearm part) are extracted as cutout regions.

  Further, in the present embodiment, the home server 1 of the first communication unit 100A extracts the area including the entire face of Mr. A (ie, the head image) as the cutout area in addition to the area extracted by the above procedure. It has become. This is because it reflects that the facial expression and mouth movement of Mr. A are likely to change in the dialogue communication.

  After the area extraction (selection of the cutout area) is performed as described above, the home server 1 of the first communication unit 100A then generates image data of the extracted area, and the home server 1 of the second communication unit 100B. Send towards. Note that display position data indicating the display position of the area (strictly speaking, relative position to the frame image) is incorporated in the image data of the cutout area.

  On the other hand, when the home server 1 of the second communication unit 100B receives the image data of the cutout area, the home server 1 combines the image obtained by expanding the image data (that is, the cutout image) with the previously displayed frame image. Get the frame image to be displayed this time. Here, the “frame image displayed last time” is a frame image (display image) displayed on the display 5 immediately before receiving the image data of the cutout area.

  More specifically, the home server 1 of the second communication unit 100B analyzes the display position data in the received image data, and specifies the position corresponding to the cutout area (that is, the display position of the cutout image). Then, as shown in (D) of FIG. 7, the home server 1 superimposes the cutout image on the position of the specified cutout area among the person images of Mr. A in the frame image displayed last time. As a result, as shown in the figure, a frame image to be displayed this time (strictly speaking, a person image of Mr. A in the frame image) can be obtained.

  Next, the image quality adjustment processing will be described with reference to FIG. As described above, the home server 1 of the first communication unit 100A includes the person image of Mr. A and a partial image of the person image in the frame image captured by the camera 2 (hereinafter, these are collectively Image data is generated for the transmission image). On the other hand, as described above, the home server 1 of the first communication unit 100A estimates the central visual field of Mr. B.

  Then, the home server 1 of the first communication unit 100A performs the image quality adjustment process on the transmission image. In this image adjustment process, an image displayed in an area other than the central visual field than the image (first image) displayed in the central visual field of Mr. B on the display screen 5a of the display 5 in the transmission image (second Reduce the image quality). Note that “to lower the image quality of the second image than the first image” means to lower the resolution of the second image than the resolution of the first image. Further, the degree of reduction in the image quality of the second image (reduction degree) is not particularly limited, but when displaying the second image after the image quality reduction on the display 5, Mr. B feels a sense of discomfort It is good to be set to no extent.

  Further, in the image adjustment process, the home server 1 of the first communication unit 100A generates, as the image data of the transmission image, the image data of the transmission image such that the second image has a lower image quality than the first image. It transmits toward the home server 1 of 2nd communication unit 100B.

  When the image data of the transmission image is received by the home server 1 of the second communication unit 100B, a frame image including the transmission image is displayed on the display 5 of the second communication unit 100B. In the displayed image, the first image (the hatched portion in FIG. 8) displayed in the central visual field of Mr. B is a higher quality image, but it is other than the central visual field. The second image displayed (that is, in the peripheral vision region) is a lower quality image. Even with such a display image, the image (second image) displayed outside the central visual field is difficult to be recognized visually, so Mr. B looking at the display 5 feels a sense of discomfort I can not feel it. That is, even if there is a portion with different image quality in the display image, if the portion displayed in the central visual field has high image quality, the influence on the realism of the dialog communication is reduced. Therefore, in the present embodiment, it is possible to reduce the data transmission load by an amount corresponding to the decrease in image quality while securing the presence of interactive communication while reducing the image quality of the second image in the display image.

  In addition, in selecting a range (i.e., the second image) to be lowered in image quality in the transmission image, the central visual field area of Mr. B is estimated, but in the present embodiment, as described above, the height of Mr. B is The central visual field area is estimated on the basis of the face orientation. As a result, the central visual field area of Mr. B can be appropriately estimated, and as a result, an appropriate range of the human image of Mr. A is obtained for the second image determined according to the central visual field area of Mr. B as well. It will be selected.

(Display image reconstruction function)
The home server 1 of the second communication unit 100B receives the image data transmitted from the home server 1 of the first communication unit 100A, and displays an image obtained by expanding the image on the display 5. Here, when the image data transmitted from the home server 1 of the first communication unit 100A is described, as described above, the image data of the background image and the image data of the person image are separately transmitted. There is. Therefore, the home server 1 of the second communication unit 100B receives the respective image data, develops the image data, and combines the background image and the person image. Thus, the home server 1 of the second communication unit 100B reconstructs each image (received image) received from the home server 1 of the first communication unit 100A, and displays a frame image (display image) to be displayed on the display 5 this time. To get

  In addition, when the home server 1 of the second communication unit 100B receives image data of a part of the person image (that is, image data of the cutout region), the home server 1 of the second communication unit 100B By superimposing the cutout images, a person image of Mr. A to be displayed this time is acquired.

  Then, the home server 1 of the second communication unit 100B causes the display 5 to display the acquired frame image. At this time, the home server 1 of the second communication unit 100B adjusts the display size of the person image of Mr. A in the frame image to the actual size of Mr. A (life size). Specifically, the home server 1 of the second communication unit 100B determines the distance between the A and the display 5 and the A among the current information of the A acquired from the home server 1 of the first communication unit 100A. Adjust the display size of the person image of Mr. A according to the distance of.

<< Flow of dialogue using the image display system according to the present embodiment >>
Next, user interaction performed using the present system S, that is, a specific flow of dialogue communication (hereinafter, dialogue communication flow) will be described with reference to FIGS. FIG. 9 is a diagram showing the flow of the interactive communication flow. FIG. 10 is a diagram showing the flow of communication pre-processing. FIG. 11 is a diagram showing a flow of current information notification processing. FIG. 12 is a diagram showing the flow of the image processing transmission process. FIG. 13 is a diagram showing a flow of selection processing of the cutout region. FIG. 14 is a diagram showing a flow of calculation processing of the cutout region. FIG. 15 is a diagram showing a flow of image quality adjustment processing. FIG. 16 is a diagram showing a flow of display image reconstruction processing.

  By the way, in the interactive communication flow described below, the image display method of the present invention is adopted. That is, according to the image display method of the present invention, each device of the system S, in particular, the home server 1 (corresponding to the first computer and the second computer) of each of the first communication unit 100A and the second communication unit 100B has their own functions. It is realized by exerting

  First, a rough flow of the dialog communication flow will be described with reference to FIG. 9. At the start of the dialog communication flow, communication pre-processing is executed (S001). The communication pre-processing is a process executed to determine whether or not to start dialogue communication, and before the start of the dialogue communication flow, for example, when Mr. A or Mr. It is executed when entering the room).

  When dialogue communication is started after execution of communication preprocessing, current information notification processing (S002), reception of partner current information (S003), image processing / transmission processing (S004), reception of partner image (S005), and the like A display image reconstruction process (S006) is performed. These processes are performed in the home server 1 of both the first communication unit 100A and the second communication unit 100B, and are repeatedly performed until the interactive communication is completed (S007). Then, when Mr. A or Mr. B performs an operation of ending the dialog communication in the dialog communication, the present system S receives the ending operation, and as a result, the dialog communication is ended.

  Next, the flow of each process S001 to S007 in the interactive communication flow will be described. The flow of processing executed by the communication unit on the A side (that is, the first communication unit 100A) and the flow of processing executed by the communication unit on the B side (that is, the second communication unit 100B) It is substantially the same. For this reason, in the following, only the flow of processing performed in the first communication unit 100A will be described except for the display image reconstruction processing described later, and the second communication unit 100B will be described for the display image reconstruction processing. The flow of processing to be performed will be described.

  First, communication pre-processing will be described with reference to FIG. The communication preprocessing starts from the point where the home server 1 acquires a captured image (frame image) in the room by the camera 2 capturing an image of a room in which the camera 2 is installed (S011). At this time, the home server 1 acquires depth data on the frame image together with the frame image (S012).

  Then, the home server 1 determines whether or not Mr. A is present in front of the display 5 based on the frame image and depth data acquired in the previous steps S011 and S012 (S013). If it is determined that Mr. A is present in front of the display 5, the home server 1 waits until the home server 1 of the other party obtains a similar determination result (that is, a determination result that Mr. B is in front of the display 5). . Then, when both home servers 1 obtain the above determination result, communication can be started (S014), and at this time, the communication pre-processing ends.

  On the other hand, when it is determined that Mr. A is not present in front of the display 5, the home server 1 determines whether it has reached the update time of the background image (S015). If it is determined that the background image has been updated, the home server 1 transmits the image data of the frame image acquired in the previous step S011 to the other home server 1 (S016). The image data transmitted at this time is an image in which only Mr. A is not shown but only the inside of the room is shown, that is, image data of a background image.

  As described above, the home server 1 transmits the image data of the background image every time it reaches the update time of the background image while Mr. A is not present in front of the display 5 in the communication pre-processing. The background image update cycle (time interval) is not particularly limited, and can be set arbitrarily.

  Next, the current information notification process will be described with reference to FIG. The present information notification process is performed in a state where Mr. A is in front of the display 5, and notifies the other party's home server 1 as the present information on the position and posture of Mr. A in such a state. Specifically, in the current information notification process, the home server 1 calculates the distance between the user A and the display 5 based on the depth data acquired together with the frame image in which the user A appears (S021). . Further, the home server 1 specifies a skeleton model of Mr. A from the depth data and the frame image described above (S022). Further, the home server 1 calculates the height of Mr. A from the calculation result of the distance calculated in step S021 and the skeletal model specified in step S022 (S023). Furthermore, the home server 1 specifies the direction of the face of the person A from the person image of the person A in the acquired frame image (S024).

  Then, the home server 1 notifies the home server 1 of the other party of the present information obtained by the above steps, that is, the distance between Mr. A and the display, the height of Mr. A and the orientation of Mr. A's face. (S025). At this point in time, the current information notification process ends.

  Next, reception of the other party current information will be described. The home server 1 acquires the other party current information notified by the other party home server 1 (that is, the current information of Mr. B) through communication with the other party's home server 1. Specifically, the home server 1 receives, from the other party's home server 1, data indicating the distance between the B and the display 5, the height of the B and the orientation of the B's face.

  Next, the image processing transmission process will be described with reference to FIG. The image processing and transmission process is executed each time the home server 1 acquires a frame image from the camera 2. In this process, the acquired frame image or a part of the image data of the frame image is transmitted to the other party's home server 1 . Then, the type of image data transmitted in the image processing and transmission process changes in accordance with the elapsed time after the start of the dialog communication, the current information of the acquired Mr. B, and the like.

  Specifically, immediately after the start of the dialogue communication, image data of the background image is to be transmitted (S031, S032). The image data of the background image transmitted at this time is a frame image acquired in advance by the home server 1 at the preliminary stage of communication start (for example, the above-mentioned communication preprocessing), more specifically, Mr. A displays 5 The image data indicates a frame image when the camera 2 captures an image before moving to the front of the image.

  When the image data of the background image is transmitted immediately after the start of communication, it is not transmitted until after the interactive communication ends. That is, the process of transmitting the image data of the background image is executed at a frequency less than the frequency at which the home server 1 acquires a frame image from the camera 2. As a result, during interactive communication, once image data of the background image is transmitted immediately after the start of communication, it is not necessary to transmit image data of the background image, and the data transmission load is reduced accordingly. .

  On the other hand, after transmitting the image data of the background image, the image data of the person image of Mr. A is transmitted exclusively. That is, after transmission of the image data of the background image, the home server 1 extracts the person image of Mr. A from the frame image acquired from the camera 2 (S033). Thereafter, the home server 1 determines the subsequent processing contents based on the distance between the user B and the display 5 among the acquired current information of the user B.

  If it explains concretely, home server 1 will judge whether the distance between Mr. B and display 5 is more than a threshold (S034). When the above distance is equal to or greater than the threshold value in this determination, the home server 1 executes the image quality reduction process on the person image of Mr. A extracted in step S033 (S035). As a result, the image quality of the extracted person image of Mr. A is reduced to a predetermined image quality (resolution). Then, the home server 1 generates image data representing low-quality person images, that is, low-quality person image data, and transmits it to the other party's home server 1 (S036). The low-quality person image data transmitted at this time is data for displaying the person image of Mr. A, more strictly, the whole-body image of Mr. A with the image quality after reduction.

  As described above, when the distance between Mr. B and the display 5 is equal to or more than the threshold value, low-quality person image data so that the person image of Mr. A displayed to Mr. B becomes a lower quality image. Generate Then, the home server 1 transmits the generated low image-quality person image data to the home server 1 of the other party. By transmitting low-quality human image data in this manner, the data transmission load can be reduced by the amount of image quality degradation.

  On the other hand, when the distance between Mr. B and the display 5 is less than the threshold value, the home server 1 cuts out a partial area from the person image of Mr. A and transmits the image data of the cutout area. It has become. At this time, the home server 1 executes a process of selecting which area is to be cut out of the person image of Mr. A, that is, a process of selecting a cutout area (S037).

  The procedure of the process of selecting the cutout region will be described with reference to FIG. 13. In this process, first, displacement amounts of the set region on the body axis of person A, specifically, the head and the waist are calculated ( S101). Here, the “displacement amount” refers to the movement amount of the home server 1 in a period from the previous frame image acquisition time to the current frame image acquisition time (hereinafter, an image acquisition interval). Then, in the present embodiment, the change of the skeletal model of Mr. A identified in the current information notification process (specifically, the skeletal model identified at the previous frame image acquisition, and the skeletal model identified at the current frame image acquisition) The above displacement amount is calculated from the difference of

  After calculating the displacement amount, the home server 1 determines whether or not the displacement amount of at least one of the head and the waist is equal to or greater than a threshold (S102). Here, the “threshold” is a value set for selection of the cutout region, and is a reference value for determining whether each set site in the skeletal model has moved during the image acquisition period. . The specific value of the threshold is not particularly limited, but is preferably set to a suitable value for appropriately selecting the cutout region.

  Then, when at least one displacement amount of the head and the waist is equal to or more than the threshold, the home server 1 further calculates the displacement amount of each foot (S103). Thereafter, the home server 1 determines whether the displacement amount of each foot is equal to or more than a threshold (S104). If it is determined in this determination that the displacement amount of at least one foot is equal to or greater than the threshold value, the home server 1 cuts out the upper body image and the lower body image, that is, the whole body image in the person image of Mr. A (S105). Conversely, if it is determined that the displacement amounts of the two feet are both less than the threshold value, the home server 1 cuts out the upper body image in the person image of Mr. A (S106).

  As described above, in the present embodiment, when the displacement amount of at least one of the head and the waist is equal to or larger than the threshold value, the upper body image is cut out from the person image of Mr. A. This is because it is assumed that the body axis, that is, the upper body is moving and displaced if at least one of the head and the waist is moving. Then, if the cutout region is selected in units of upper body images, it is possible to more simply execute the processing related to the selection.

  On the other hand, when the displacement amounts of the head and the waist are both less than the threshold value, the home server 1 calculates displacement amounts for each of the limbs (two hands and two legs) (S107). Then, the home server 1 determines whether the displacement amount of each of the four limbs is equal to or more than a threshold (S108). In this determination, when it is determined that any displacement amount is less than the threshold, the home server 1 cuts out a head image from the person's image of Mr. A (S109).

  On the other hand, when it is determined that at least one displacement amount is equal to or more than the threshold value, the home server 1 executes cutout area calculation processing as processing for determining the cutout area in more detail (S110). The procedure of calculation processing of the cutout region will be described with reference to FIG. 14. In this processing, first, the displacement amount is calculated for the setting portion other than the setting portion (that is, the head, the waist and the limbs) for which the displacement amount has already been calculated. (S121). More specifically, the home server 1 specifies a part of the four limbs that is equal to or greater than the threshold of the displacement amount, and calculates the displacement amount for a set part adjacent to the part. Note that “a setting site adjacent to a certain site” means a setting site located next to a certain site among a plurality of setting sites set in the skeletal model, and more strictly, a side closer to the body axis with a certain site It is a setting part adjacent to each other.

  Then, the home server 1 determines whether the calculated displacement amount is equal to or more than a threshold (S122). When it is determined in this determination that the displacement amount is equal to or greater than the threshold value, the home server 1 determines the coordinates of the previous frame image with respect to the set portion (hereinafter referred to as the corresponding portion) determined to have the displacement amount equal to or more than the threshold value. And the coordinates in the frame image of the image are stored (S123). Here, “coordinates in the previous frame image” is coordinates (two-dimensional coordinates) representing the relative position of the corresponding part with respect to the frame image acquired by the home server 1 from the camera 2 in the previous time. “Coordinates in” means coordinates (two-dimensional coordinates) representing the relative position of the relevant part with respect to the frame image acquired by the home server 1 from the camera 2 this time.

  After that, the home server 1 determines whether there is a set part adjacent to the corresponding part (S124), and if there is a set part adjacent to the corresponding part, calculates the displacement amount for the set part (S125) Then, it is determined whether the calculation result is equal to or more than a threshold (S126). If it is determined in this determination that the displacement amount is equal to or greater than the threshold value, the home server 1 determines the previous frame with respect to the set portion determined to have the displacement amount equal to or greater than the threshold value (ie, the set portion to be newly corresponding). The coordinates in the image and the coordinates in the current frame image are stored (S123).

  Thereafter, the home server 1 repeats the calculation of the displacement amount (S125), the comparison with the threshold (S126), and the storage of the coordinates (S123) for the setting site that is newly the corresponding site and the setting site adjacent thereto. Then, when reaching the set region where the displacement amount is less than the threshold, ie, the non-moving region, the home server 1 reads the coordinates stored so far and specifies the X component and the Y component of each coordinate. Do. Then, the home server 1 specifies the maximum value and the minimum value for each component (S127). After that, the home server 1 sets an area defined by the minimum value and the maximum value of each component (specifically, a rectangular area having the minimum value and the maximum value of each component as vertex coordinates) as a cutout area (S128) .

  The series of steps S121 to S128 described above are repeated until the processing is completed for all the set parts (S129). Then, when there are no unprocessed set parts, the home server 1 ends the cutout area calculation process.

Returning to the explanation of the selection processing of the cutout region, when the calculation processing of the cutout region is executed, the home server 1 performs the image of the region calculated (decided) in the calculation processing and the head image of the person A's person Cut out from the image (S111).
Then, when the cutout area is selected according to the procedure described above, the home server 1 ends the cutout area selection process.

  As described above, in the present embodiment, when the distance between Mr. B and the display 5 is less than the threshold, the home server 1 cuts out a partial area from the personal image of Mr. A, and the image of the area Only the data is transmitted to the home server 1 of the other party. As a result, the data transmission load can be reduced as compared with the case of transmitting the image data of the entire person image of Mr. A. In addition, as a region to be cut out, a region including a set region which moves in a period from the previous frame image acquisition to the current frame image acquisition (period between image acquisition) in the body of Mr. A and a head image And are to be selected.

  On the other hand, in the present embodiment, when specifying the setting site moved during the image acquisition period, it is specified based on the change of the skeletal model (specifically, the difference between the previous skeletal model and the present skeletal model). ing. This makes it possible to appropriately and accurately identify the part (specified part) that has moved during the image acquisition period in the body of Mr. A.

  Further, in the present embodiment, the presence or absence of movement during the image acquisition period is to be confirmed in units of set parts. As a result, it becomes possible to easily identify the part (specified part) which has moved during the image acquisition period in the body of Mr. A. Further, in the present embodiment, when confirming the presence or absence of the movement of each setting site during the image acquisition interval, the displacement amount during the image acquisition interval is calculated for each setting site, and the calculation result of the displacement amount is a threshold. It is to be judged whether it is above or not. With such a procedure, it is possible to more easily identify the portion moved during the image acquisition period.

  Furthermore, in the present embodiment, after the displacement amount and the threshold value are compared (decided) with respect to a certain set site in the process of calculating the cutout region, next, the judgment with respect to the set site located next to the certain set site is performed. It has become. Then, when selecting the cutout region, the region is selected so as to include all the set parts (corresponding parts) moved during the inter-image acquisition period. Specifically, the coordinates in the previous frame image and the coordinates in the current frame image for each corresponding portion are determined. In addition, the maximum value and the minimum value are specified for the X component and the Y component of the coordinates obtained for each corresponding part. Then, a region defined by the maximum value and the minimum value of each of the identified components is selected as a cutout region.

  By selecting the cutout region according to the above-described procedure, an image of a portion that has moved during the image acquisition period can be appropriately selected in the person image of person A. Furthermore, by superimposing the cut-out image on the previous display image (frame image) to construct the display image of this time, the frame image acquired this time by the home server 1 (strictly speaking, in the frame image, Mr. A's It is possible to appropriately reproduce a person image).

  Returning to the description of the image processing and transmission process, after selecting the cutout area, the home server 1 confirms the data capacity of the image data of the cutout area (that is, the image data to be transmitted). Then, the home server 1 determines whether the data capacity is equal to or more than the set value (S039). Here, the “set value” is a value set in advance as a reference value for determining the presence or absence of the execution of the image quality adjustment process on the transmission image. The specific value of the setting value is not particularly limited, but is preferably set to a suitable value in order to appropriately determine whether or not the image quality adjustment processing is to be performed.

  If the data capacity is less than the set value in the above determination, the home server 1 does not perform the image quality adjustment process on the image of the cutout area (clipped image), and the image data of the cutout area is processed by the home server 1 of the other party. Send toward (S040). On the other hand, when the data capacity is equal to or greater than the set value in the above determination, the home server 1 executes the image quality adjustment processing on the cutout image (S041). After the end of the image quality adjustment processing, the home server 1 generates image data for displaying the clipped image (that is, the image quality adjusted image) subjected to the image quality adjustment processing, and transmits it to the other party's home server 1 (S042) ).

  The procedure of the image quality adjustment process will be described with reference to FIG. 15. In this process, first, the present information of Mr. B, specifically, the height of Mr. B and the direction of Mr. B's face from Mr. B's central visual field An area is estimated (S131). After that, the home server 1 determines whether the data of the cutout image to be transmitted is the data of the whole-body image of Mr. A (S132).

  When the data of the cut-out image is data of a whole-body image (intelligibly, when the selection processing of the cut-out area has reached step S105), the home server 1 displays on the display screen 5a of the display 5 in the cut-out image. At the time, the image quality of the image (second image) displayed in the area other than the central visual field area is lowered than the image (first image) located in the central visual field area of Mr. B (S133).

  On the other hand, when the data of the cutout image is not the data of the whole body image, the home server 1 selects the cutout image (S134). Then, the home server 1 determines whether or not there is an image (second image) to be displayed in an area other than the central visual field of Mr. B when displayed on the display screen 5a of the display 5 in the selected cutout image. (S135). In this determination, when it is determined that the portion corresponding to the second image is present in the selected cutout image, the home server 1 performs the first on the image (first image) displayed in the central visual field of Mr. B. The image quality of the two images is reduced (S133).

  Thereafter, the home server 1 determines whether or not an unprocessed cutout image remains (S136), selects an image with respect to the unprocessed cutout image (S134), determines the presence or absence of a second image (S135), and The image quality reduction of the two images (S133) is repeated. Then, when there is no unprocessed cutout image, the home server 1 ends the image quality adjustment processing.

  As described above, in the present embodiment, when the capacity of the image data of the cutout image to be transmitted is equal to or more than the set value, the image quality adjustment processing is performed to reduce the image quality of part of the cutout image. As a result, the image data of the cutout image after processing becomes smaller than the image data before processing, and the transmission load of the image data is reduced. Such an effect is more effectively exhibited as the area (that is, the cutout area) cut out from the person image of Mr. A becomes wider.

  In addition, in selecting a portion (second image) to be degraded in the cut-out image, the central visual field area of Mr. B is estimated. Then, in the cutout image, the image quality of the portion displayed in the area (peripheral visual field area) outside the central visual field area of Mr. B estimated on the display screen 5a of the display 5 is reduced to a predetermined image quality. This reflects that the image within the peripheral visual field is difficult to visually recognize, and even if the image quality of the image is somewhat low, the influence on the realism of the dialog communication felt by the viewer of the display image is small. ing. As a result of the above, the portion (the second image) that lowers the image quality is appropriately selected from the cut-out image, so that the data transmission load can be effectively reduced without losing the sense of realism of the dialog communication. It becomes possible.

  Then, the home server 1 ends the image processing and transmission process when transmission of various image data is completed.

  Next, display image reconstruction processing will be described with reference to FIG. The present process reconstructs each image obtained by expanding the image data received by the home server 1 of the second communication unit 100B from the home server 1 of the first communication unit 100A, and displays this on the display 5 (frame (frame Image) is acquired.

  More specifically, the home server 1 of the second communication unit 100B receives the image data of the background image immediately after the start of the dialogue communication (No in S051). After that, the home server 1 of the second communication unit 100B receives the image data of the person image of Mr. A (Yes in S051). If the image data received at this time is the data of the whole-body image of Mr. A (Yes in S052), the home server 1 responds to the current information of Mr. A (specifically, the height of Mr. A). The display size of the image is adjusted to be the actual size (life size size) of Mr. A (S054). Thereafter, the home server 1 acquires a frame image (display image) to be displayed on the display 5 this time by combining the background image already acquired and the person image of Mr. A acquired this time (S055).

  On the other hand, when the image data received from the home server 1 of the first communication unit 100A is image data of a part of the person's image of A (that is, the cutout image) (No in S052), the home of the second communication unit 100B. The server 1 reconstructs the person image of Mr. A using the image data described above.

  Explaining in detail, the home server 1 of the second communication unit 100B superimposes the image (cutout image) indicated by the image data received this time and the person image of Mr. A displayed on the display 5 last time (S053). At this time, the home server 1 analyzes the display position data incorporated in the image data received this time, specifies the display position of the cutout image, and displays the display position in the person image of Mr. A displayed on the display 5 last time. The cutout image is superimposed. The display position of the cutout image to be identified is a position corresponding to the cutout region in the frame image (that is, the previous display image) displayed on the display 5 immediately before the reception of the image data of the cutout region, that is, a cutout region. The position corresponds to the selected rectangular area.

  As described above, the home server 1 of the second communication unit 100B reconstructs (acquires) the person image of Mr. A displayed on the display 5 this time using the cutout image and the person image of Mr. A displayed last time. Do. After that, the home server 1 of the second communication unit 100B adjusts the display size of the person image of the person A in the same procedure as the above-described procedure, and then the background image and the person image of the person A obtained this time To obtain the display image of this time (S055).

  Then, the home server 1 of the second communication unit 100B causes the display 5 to display the frame image (display image) acquired this time (S056). At this time, the home server 1 ends the display image reconstruction process.

  The series of processes described above are repeatedly executed until the dialogue communication is completed. As a result, while effectively reducing the load of data transmission, interactive communication with a sense of reality (realism) can be realized.

1 Home Server 2 Camera (Imaging Device, Information Providing Device)
3 Microphone 4 Infrared sensor (measuring device, information providing device, distance measuring device)
5 Display
5a Display Screen 6 Speaker 100A First Communication Unit 100B Second Communication Unit GN External Network S This System (Image Display System)

Claims (9)

An imaging device for imaging a first user;
A measurement device for measuring a measurement target value regarding the position of each part of the body of the first user;
A first computer for acquiring a frame image constituting an image of the first user captured by the imaging device;
A second computer in communication with the first computer to obtain the frame image;
And a display for displaying the frame image acquired by the second computer to a second user who is at a different place from the first user,
The first computer is
Of the various parts of the body, the specified part that moved during the period, based on the change in the measurement result of the measurement value during the period from the time of acquisition of the frame image to the time of acquisition of the frame image this time. And the process of identifying
A process of extracting an area including the specified part from the person image of the first user in the frame image acquired this time by the first computer;
Executing processing of generating image data of the area and transmitting it to the second computer;
When the second computer receives the image data of the area, the second computer superimposes the image of the area on a position corresponding to the area among the frame images displayed on the display before the image data is received. An image display system configured to display the frame image configured as described above on the display.   The first computer, in the process of identifying the specified part, the setting part among a plurality of setting parts set in the skeleton of the first user based on a change in the measurement result of the measurement target value during the period. The image display system according to claim 1, characterized in that the set site moved during a period is specified, and the specified portion is specified to include at least the set site.   When the first computer specifies the set site moved during the period in the process of specifying the specified portion, it is determined whether the displacement amount of the set site during the period is equal to or more than a threshold. The image display system according to claim 2, characterized in that it is performed for each setting site, and the setting site whose displacement amount is equal to or more than the threshold is specified as the setting site that has moved during the period.   When the first computer performs the determination for each setting site, the first computer performs the determination for the setting site located next to the setting site after the determination for the setting site, and 4. The image display according to claim 3, wherein when extracting the area including the specific part, the area is extracted such that all the set parts moved during the period are included in the area. system. At least one of the plurality of setting sites set is a site on the body axis of the upper body of the first user,
The first computer extracts the image of the upper body as the area when it is determined that the displacement amount is equal to or more than the threshold value in the determination on the set region on the body axis. Item 5. An image display system according to item 3 or 4. The first computer executes processing for generating background image data indicating a background image in the frame image separately from image data other than the background image, and transmitting the generated image data to the second computer.
The frequency at which the first computer executes the process of transmitting the background image data is less than the frequency at which the first computer acquires the frame image from the imaging device. The image display system according to any one of the items. Providing information relating to at least one of the positional relationship between the second user and the display and the posture of the second user while the second user is present in front of the display, to the second computer Has an information provision device,
The first computer further executes a process of acquiring the at least one content specified from the information by the second computer, and in the process of generating the image data of the area, the display is performed in the image of the area. The image data of the area is set such that the second image displayed in a range different from the first image is lower than the first image displayed in the range determined according to the at least one content in the device. The image display system according to any one of claims 1 to 6, wherein the image display system generates the image. It has a distance measurement device that measures the distance between the second user and the display while the second user is in front of the display.
The first computer acquires the measurement result of the distance from the second computer, and when the distance is equal to or larger than a preset size, the image quality of the person image is reduced to a predetermined image quality, The image display system according to any one of claims 1 to 7, wherein low-quality person image data indicating the person image of image quality is generated and transmitted to the second computer. The second computer comprises a first computer for acquiring a frame image constituting a video of a first user captured by an imaging device, and a second computer for communicating with the first computer to acquire the frame image. The image display method, wherein the frame image acquired by the computer is displayed by a display to a second user who is at a different place from the first user,
Performing a process of acquiring the measurement result of the measurement target value from a measurement device that measures the measurement target value regarding the position of each part of the body of the first user;
The first computer calculates the duration of the period among the body parts based on a change in the measurement result of the measurement target value during the period from the previous acquisition of the frame image to the current acquisition of the frame image. Executing processing for identifying the identified part moved to
Performing a process of extracting an area including the specified part from the person image of the first user in the frame image acquired this time;
Performing processing of generating image data of the area and transmitting the image data to the second computer;
When the second computer receives the image data, an image of the area indicated by the image data is displayed at a position corresponding to the area in the frame image displayed on the display before the image data is received. Displaying the frame image configured by superimposing on the display;
An image display method characterized by comprising:

Images