JP4564432B2 - Video composition apparatus, video composition method, and program - Google Patents

Description

  The present invention relates to a video composition device, a video composition method, and a program.

  In recent years, due to significant advances in communication technology, inexpensive and broadband network platforms have become widespread. In particular, by establishing communication services such as ADSL (Asymmetric Digital Subscriber Line) lines and FTTH (Fiber To The Home) lines, it is easy to build a broadband network environment not only in companies but also in general households. Is possible.

  With the widespread use of broadband networks, the provision of video and audio communication services has begun. By constructing a system in which information devices capable of transmitting and receiving data exchange data via a network, bidirectional communication using video and audio becomes possible. Such a system is called a video conference system. In particular, a multi-point video conference system that enables communication among a plurality of members by connecting a plurality of bases via a network has attracted attention in recent years.

  When constructing a multipoint video conference system, a method in which each conference terminal apparatus (also called a conference client apparatus or simply a client) exchanges images with each other and a multipoint control unit (MCU: Multipoint Control Unit). After receiving the video from each client using the conference server device or simply the server), the MCU performs enlargement / reduction / cutout etc. on these videos and synthesizes them into one video (mixing). There is a way to distribute to each client. In a server-client type system such as the latter, each client only needs to send and receive video from a single MCU, so the processing load for sending and receiving video for each client and the accompanying load are higher than the former. It is possible to reduce the network load. Therefore, in the currently provided multi-point video conference systems, services are often provided mainly by a server / client type configuration.

  There are various composition layouts for image composition performed by the MCU. For example, a layout in which videos received from each client are reduced to the same size and arranged without gaps, or a layout in which one video is made larger than the other videos and arranged without gaps, or one video is maximized. There is a picture-in-picture layout in which other images are reduced and placed on top of each other.

  Furthermore, for video composition performed by the MCU, in addition to a method of sharing the same composite video among all clients, a method of referring to a composite video having an independent layout for each client is considered. In the former method, the representative client designates the layout of the composite video, and the MCU performs a single video composition and distributes the copy to each client. In the latter method, each client designates the layout of the desired composite video and notifies the MCU of the layout. The MCU performs a plurality of video composition independently based on the layout notified from each client, and distributes the generated composite video to each client.

  In general, a video conference system is required to transmit high-quality video / audio in order to play a role as a stress-free communication tool. Therefore, it is inevitable that the proportion of the video / audio data transmission band that accompanies the use of the video conference system will be high in the network communication band available to the user. This is a situation where the available network communication bandwidth is limited, such as when the network bandwidth is shared by multiple users, or when the client is connected via a mobile phone network, for example, Alternatively, this is particularly problematic in situations where data transmission other than video / audio, such as desktop image sharing and application data sharing, is performed in parallel. Under such circumstances, how to save video / audio data and transmit it efficiently is an important issue.

As a conventional solution to such a problem, for example, there is a method of automatically controlling the quality of video / audio data in accordance with the congestion status of the network. This method investigates the network congestion status at the start of the conference or during the conference, and if it detects that the congestion status has deteriorated, it automatically reduces the transmission quality of video and audio data to save communication bandwidth. Plan.
JP-A-10-304335 JP 2004-187170 A

  In a video conference system that has a configuration in which a plurality of clients are connected to an MCU and each client can refer to an independent layout, even in an environment where the network congestion does not change significantly, the client to the server An important issue is how to control such that high-quality video data is not transmitted unnecessarily.

  Therefore, in Japanese Patent Laid-Open No. 10-304335 (Patent Document 1), information on a target portion of interest of a client user is detected and notified to the server, so that the target of interest among a plurality of video data transmitted from the server to the client. By increasing the quality of the video corresponding to the part and lowering the quality of the video corresponding to the other part, the communication band is saved.

  However, in the method disclosed in JP-A-10-304335, transmission control of video data from the server to the client is possible, but transmission control of video data from the client to the server cannot be performed.

  Therefore, in Japanese Patent Application Laid-Open No. 2004-187170 (Patent Document 2), when a client selects a screen template, the server notifies the client of image parameter information corresponding to the screen template, and determines the quality of image data transmitted from the client. There has been proposed a method of performing control according to the image parameter information.

  However, in the method disclosed in Japanese Patent Application Laid-Open No. 2004-187170, transmission control of video data from the client to the server is possible, but the reference status from other clients of the video is not taken into consideration in the transmission control. For this reason, even if the video sent from the client is not displayed by any other client, or even if it is displayed in a very small size, the client can display the high-quality video on the server. Try to send to. It is useless to send a video that no one pays much attention to in high quality, and as a result, this client is performing inefficient data transmission. This is undesirable because it uses the client's processing power unnecessarily and unnecessarily presses the network communication bandwidth.

  The present invention has been made to solve the above-described problems, and efficiently controls video data transmitted from a video display device to a video synthesis device, thereby enabling a communication bandwidth to be saved. Another object is to provide a video composition method and program.

  According to another aspect of the present invention, there is provided a video synthesizing apparatus, comprising: video receiving means for receiving first to Nth encoded video data from first to Nth (N is an integer of 2 or more) video display devices; Decoding means for decoding the 1st to Nth encoded video data and outputting the 1st to Nth decoded video data; and the 1st to Nth video composites from the 1st to Nth video display devices Layout information receiving means for receiving the first layout information, layout information storage means for storing the received first to Nth layout information, and first to first decoded video data represented by the first to Nth decoded video data. N video images are synthesized in accordance with the first to Nth layout information to generate first to Nth synthesized video images, and the first to Nth synthesized video images are combined with the first to Nth synthesized video images. Video transmission means for transmitting to the video display device, and the first to Nth rays. Based on at least the layout of the first video in the second to N-th layout information in the out information, a judgment value calculation means for calculating a judgment value for the first video display device, and the calculated judgment value Video quality control means for determining an encoding parameter to be applied in the first video display device; and video quality control signal transmission means for transmitting the determined encoding parameter to the first video display device. .

  The video display device as one aspect of the present invention includes video generation means for generating video data, encoding means for encoding the video data according to encoding parameters and outputting encoded video data, and the encoded video data Video transmission means for transmitting the video to the video synthesis apparatus, layout information transmission means for transmitting layout information for video synthesis to the video synthesis apparatus, video reception means for receiving the composite video data, and display of the composite video data Display means for receiving, layout information receiving means for receiving layout information respectively used by one or more other video display apparatuses, and a determination value for the own apparatus from the video layout of the own apparatus in each received layout information A determination value calculation means for calculating the coding parameter and a coding parameter to be applied from the calculated determination value Video quality control means for notifying the encoding means of encoding parameters.

  According to another aspect of the present invention, there is provided a video composition method, wherein first to Nth video data are generated in first to Nth video display devices (N is an integer of 2 or more), and the first to Nth video data are generated. The data is encoded according to the first to Nth encoding parameters to generate first to Nth encoded video data, and the first to Nth encoded video data is transmitted to a video synthesizer for video synthesis. Transmitting the first to Nth layout information for the video synthesizer, decoding the first to Nth encoded video data to generate the first to Nth decoded video data, The first to N-th synthesized videos are generated by synthesizing the first to N-th videos represented by the first to N-th decoded video data according to the first to N-th layout information, and the first to N-th synthesized videos are generated. N composite images are transmitted to the first to Nth image display devices, and the first to Nth layout information is transmitted. A determination value for the first video display device is calculated based on at least the layout of the first video in the second to N-th layout information, and the first video display device uses the calculated determination value to calculate the determination value. The first encoding parameter to be applied is determined.

  The program as one aspect of the present invention includes a step of receiving first to Nth encoded video data from first to Nth (N is an integer of 2 or more) video display devices; Decoding first encoded video data and outputting first to Nth decoded video data, and receiving first to Nth layout information for video composition from the first to Nth video display devices A step of storing the received first to Nth layout information, and a first to Nth video represented by the first to Nth decoded video data. Synthesizing according to layout information to generate first to N-th synthesized videos, transmitting the first to N-th synthesized videos to the first to N-th video display devices, and the first To at least the second to Nth layout information among the Nth layout information. Based on the layout of the first video in the out information, a step of calculating a determination value for the first video display device, and an encoding parameter to be applied in the first video display device from the calculated determination value A step of determining and a step of transmitting the determined encoding parameter to the first video display device are executed by a computer.

  A program as one aspect of the present invention includes a step of generating video data, a step of encoding the video data according to a coding parameter and outputting encoded video data, and transmitting the encoded video data to a video synthesizer A step of transmitting layout information for video composition to the video composition device, a step of receiving composite video data, a step of displaying the composite video data, and one or more other video display devices Receiving the layout information used by each, a step of calculating a determination value for the device from the layout of the image of the device in each received layout information, and encoding to be applied from the calculated determination value A step of determining a parameter;

  According to the present invention, it is possible to save the communication bandwidth by controlling the video data transmitted from the client to the server based on the video reference status of a plurality of clients participating in the conference.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 shows a system configuration of a multipoint video conference system to which the present invention is applied. FIG. 1 includes a conference server device 100 and four conference client devices 300A, 300B, 300C, and 300D. The conference client devices 300A, 300B, 300C, and 300D are connected to the conference server device 100 via the networks 500A, 500B, 500C, and 500D, respectively.

  The conference client apparatuses 300A, 300B, 300C, and 300D include camera apparatuses 310A, 310B, 310C, and 310D, respectively. Each conference client device has a function of transmitting video data captured using the camera device to the conference server device 100 through the networks 500A, 500B, 500C, and 500D. In addition, the conference client devices 300A, 300B, 300C, and 300D have a function of receiving the video data transmitted from the conference server device 100 and displaying it on the respective display devices 360A, 360B, 360C, and 360D. Further, it has a function of transmitting control information to the conference server apparatus 100 through the networks 500A, 500B, 500C, and 500D.

  The conference server apparatus 100 has a function of receiving video data transmitted from the conference client apparatuses 300A, 300B, 300C, and 300D. In addition, it has a function of generating a plurality of types of composite video having different layouts by combining a plurality of received video data into one video. It should be noted that a plurality of video compositions can be performed independently for each conference client device at the time of video composition. In addition, a composite video is transmitted to the conference client apparatuses 300A, 300B, 300C, and 300D through the networks 500A, 500B, 500C, and 500D. Similarly, it has a function of transmitting a control signal to each conference client device through the networks 500A, 500B, 500C, and 500D.

  Next, internal components related to the present invention in the conference server apparatus and the conference client apparatus will be described with reference to FIG. The functions represented by the internal components in the conference server device and the internal components in the conference client device may be realized by causing a computer to execute a program or may be realized by hardware.

  The conference server apparatus 100 includes, as its constituent elements, a video receiving unit 110 that receives video data using the communication path 510A, a decoding unit 120 that decodes video data, and a layout that uses the communication path 530A. A layout control signal receiving unit 130 that receives a control signal, a layout information storage unit 140 that stores layout information obtained by analysis from the layout control signal, a video synthesis unit 150 that performs video synthesis processing, and a synthesized video An encoding unit 160 that encodes the video, a video transmission unit 170 that transmits video data using the communication channel 520A, a determination value calculation unit 180 that calculates a determination value from the layout information, and the determination value Video quality control unit 190 for calculating video coding control parameters, and video coding control parameters using communication channel 540A And a video quality control signal transmitting unit 200 for transmitting. In addition, although the operation | movement of each internal component of the conference server apparatus 100 shown below has demonstrated the process performed between conference client apparatuses 300A, it is not restricted to conference client apparatus 300A, 300B, 300C , 300D.

  The conference client device 300A uses, as its constituent elements, a camera unit 310A, an encoding unit 320A that encodes synthesized video, a video transmission unit 330A that transmits video data through the communication channel 510A, and a communication channel 520A. A video receiving unit 340A that receives video data, a decoding unit 350A that decodes video data, a display unit 360A that displays video, a user interface unit 370A, and a layout using the communication path 530A. A layout control signal transmission unit 380A that transmits a control signal and a video quality control signal reception unit 390A that receives a video encoding control parameter using the communication path 540A are provided. In FIG. 2, only one conference client device 300A is shown, but the conference server device 100 is connected to other conference client devices 300B, 300C, and 300D, each of which is connected to the conference client device 300A. It shall have similar internal components.

  Hereinafter, the internal components of the conference server apparatus 100 will be described in detail.

  The video receiving unit 110 receives the encoded video data from the conference client device 300A using the communication path 510A and outputs it to the decoding unit 120.

  When the video data is input from the video reception unit 110, the decoding unit 120 decodes and outputs the decoded video data to the video synthesis unit 150. In this embodiment, MPEG4 is used as the video data encoding method, but the type of encoding method is not an essential problem in the present invention. 263, H.M. It may be a video encoding method such as H.264. In this case, the encoding method used in the decoding unit 120 and the encoding unit 320A supports the same video format.

  The layout control signal receiving unit 130 receives a layout control signal from the conference client apparatus 300A using the communication path 530A. The layout control signal is transmitted as an IP packet including an IP header portion and an IP payload portion. The IP header portion includes the IP address of the conference client apparatus 300A that is the source of the layout control signal, and the IP payload portion includes Includes the layout information 1000A specified in the conference client apparatus 300A. The layout control signal receiving unit 130 analyzes the layout control signal, specifies the conference client device 300A that is the transmission source of the layout control signal from the transmission source IP address included in the IP header unit, and determines the layout information from the IP payload unit. Extract 1000A.

  Here, the layout information is information for designating a method for synthesizing video synthesized in the video synthesizing unit 150. For example, as shown in FIG. It is a set of numerical values representing the relative position (hierarchy). The identification number is a number for identifying the conference client apparatuses 300A, 300B, 300C, and 300D. Here, the identification number of 300A is 1, 300B is 2, 300C is 3, and 300D is 4. The composite position / composite size is a set of (X, Y) values in the upper left corner of the video using the XY coordinate values when the horizontal and vertical directions of the composite video are both normalized to 100. The composite size is represented by a set of length (W, H) values in the horizontal and vertical directions of the video. In this embodiment, W = H. The relative position in the front-rear direction is represented by a natural number from 1 to 4, where 1 is the frontmost display position of the composite video and 4 is the rearmost display position. In the present embodiment, layout information is defined as a set of identification number, composite position, composite size, and relative position value in the front-rear direction. However, the present invention is not limited to this. For example, the layout information may include a composite cut position, composite cut size, etc. Good. The composite cropping position and composite cropping size are values that can be defined when the video composition unit 150 supports cropping of a part of the video. As shown in FIG. 5, the horizontal and vertical directions of the input video are shown. Using the XY coordinate values when both are normalized to 100, the combined cut position is defined by a set of (X, Y) values in the upper left corner of the cut area, and the combined cut size is the horizontal direction of the cut area • It is defined by a set of longitudinal length (W, H) values. When the layout control signal receiving unit 130 extracts the layout information 1000 </ b> A, the layout control signal receiving unit 130 outputs the layout information 1000 </ b> A to the layout information storage unit 140 and outputs a determination value calculation request related to the conference client device 300 </ b> A to the determination value calculation unit 180.

  When the layout information 1000A is input from the layout control signal receiving unit 130, the layout information storage unit 140 stores the input layout information 1000A. If a new input is made from the conference client apparatus 300A in this state, the layout information 1000A is updated with a new input value. In addition, when an acquisition request for layout information 1000 </ b> A is input from the video composition unit 150, the layout information storage unit 140 outputs the stored layout information 1000 </ b> A to the video composition unit 150. In addition, when layout information 1000B, 1000C, and 1000D is input from the conference client devices 300B, 300C, and 300D, the layout information storage unit 140 similarly stores the layout information. In addition, when an acquisition request for layout information 1000A, 1000B, 1000C, and 1000D is input from the determination value calculation unit 180, the layout information storage unit 140 calculates the stored layout information 1000A, 1000B, 1000C, and 1000D as a determination value. Output to the unit 180.

  When the video data is input from the decoding unit 120, the video synthesizing unit 150 outputs an acquisition request for the layout information 1000A of the conference client device 300A to the layout information storage unit 140. When the layout information 1000A of the conference client apparatus 300A is input from the layout information storage unit 140, for example, as shown in FIG. 4, the video is synthesized according to the layout information 1000A to generate a synthesized video 1010A. The generated composite video 1010A is input to the encoding unit 160.

  When the synthesized video data 1010A is input from the video synthesizing unit 150, the encoding unit 160 encodes this and outputs it to the video transmitting unit 170. In this embodiment, MPEG4 is used as the video data encoding method, but the type of encoding method is not an essential problem in the present invention. 263, H.M. It may be a video encoding method such as H.264. In this case, the encoding method used in the encoding unit 160 and the decoding unit 350A supports the same video format.

  When the encoded composite video data 1010A is input from the encoding unit 160, the video transmission unit 170 transmits the composite video data 1010A to the conference client device 300A using the communication path 520A.

  When the determination value calculation unit 180 receives a determination value calculation request from the conference client apparatus 300A from the layout control signal receiving unit 130, the layout value storage unit 140 stores the layout information 1000A, 1000B, Output 1000C and 1000D acquisition requests. When four pieces of layout information 1000A, 1000B, 1000C, and 1000D as shown in FIG. 6 are input from the layout information storage unit 140, determination values 1020A and 1020B as shown in FIG. 1020C and 1020D are calculated. The judgment value is a value defined for each conference client device, and how much or clearly the video of the camera device transmitted by each conference client device should be displayed (whether it is receiving attention) ) Degree. Here, the determination value 1020A of the conference client apparatus 300A is set to the horizontal length (W) of the video of 300A on the composite video viewed by the conference client apparatuses other than itself, that is, 300B, 300C, and 300D. It is defined as the maximum value. That is, in the three pieces of layout information 1000B, 1000C, and 1000D in FIG. 6, pay attention to the line whose identification number is equal to 1 (= identification number of 300A), and select the largest W value among them and select this as the determination value 1020A. (In the example of FIG. 6, the determination value 1020A is calculated as the maximum value of 50, 50, and 25, that is, 50). In the present embodiment, the determination value is calculated based on the three pieces of layout information excluding 1000A. However, the determination value may be calculated based on the four pieces of layout information including 1000A. In this example, the maximum value in the horizontal length of the video is used as the determination value, but the minimum value may be used as the determination value.

  Although the determination value is calculated based on the horizontal length (W) of the video here, the determination value is not limited to this, and the determination value may be calculated based on other information. For example, as a possible example, the determination value calculation unit 180 may perform composite cut-out of a video corresponding to 300A on a composite video viewed by a client device other than 300A when calculating a determination value related to the conference client device 300A. The determination value may be obtained based on the smallest or largest one of the three horizontal lengths of the combined cuts. As an example of the former, when at least one of the horizontal lengths of the three composite cuts is less than 100, regardless of the value of the horizontal length (W) of the 300A video on each composite video. The determination value may be always output as 100. When the horizontal lengths of the composite cuts are all equal to 100, the maximum value of the horizontal length (W) of the 300A video on each composite video is output as a determination value, as described above. For example, the determination value of 300A calculated from the layout information of FIG. 6 is 100 in this case (the horizontal length of the combined cutout of the video corresponding to 300A in the layout information 1000C is less than 100). Alternatively, as a determination value calculation method that is different from these, the determination value calculation unit 180, when calculating a determination value related to 300A, an image corresponding to 300A on a composite image viewed by a client device other than 300A. Focusing on the relative position in the front-rear direction, and determining the minimum or maximum value of these, depending on the value, if it is 1, the determination value is 100, 2 is the determination value 75, 3 For example, if the determination value is 50, the determination value 25 may be output. For example, the determination value of 300A calculated from the layout information of FIG. 6 has a minimum value of 2 because the relative positions in the front-rear direction of the images corresponding to 300A on each composite image other than 300A are 2, 4, and 4, respectively. 75 corresponding to is derived. After calculating the determination value 1020A, the determination value calculation unit 180 outputs the determination value 1020A to the video quality control unit 190.

  In the above, three methods for calculating judgment values have been shown, but these calculation methods may be used separately, or judgment values may be calculated using any combination of these three calculation methods. . For example, the determination value may be calculated by inputting the determination value obtained by each calculation method to a function that weights the determination value of each calculation method. Alternatively, the smallest or largest judgment value obtained by each calculation method may be used as the judgment value to be obtained.

  When the determination value 1020A is input from the determination value calculation unit 180, the video quality control unit 190 calculates the video encoding control parameter 1030A. The video encoding control parameter 1030A is a value used when the encoding unit 320A of the conference client apparatus 300A encodes the camera video, and here defines the resolution and frame rate at the time of encoding. When calculating the video coding control parameter 1030A, as shown in FIGS. 8 and 9, the video quality control unit 190 holds a correspondence table between the determination value and the video coding control parameter, and refers to it appropriately. To calculate. Normally, the resolution and frame rate values included in the video encoding control parameter 1030A increase in accordance with the magnitude of the determination value 1020A, which corresponds to a higher quality video. When the video coding control parameter 1030A is calculated, the video quality control unit 190 adds an IP header including the IP address of the destination conference client apparatus 300A to the video coding control parameter 1030A, and the video quality control signal transmission unit 200. Output to.

  When the video quality control signal transmission unit 200 receives the video encoding control parameter 1030A including the IP header from the video quality control unit 190, the video quality control signal transmission unit 200 receives the IP header including the IP address and the like of the conference server device 100 as the transmission source. Further, an IP packet is generated by adding. This IP packet is called a video quality control signal. The video quality control signal transmission unit 200 transmits a video quality control signal to the conference client apparatus 300A using the communication path 540A.

  Next, the internal components of the conference client apparatus 300A will be described in detail.

  The camera unit 310A outputs video data obtained by capturing a face image, a material image, a PC desktop image, or the like of a conference participant to the encoding unit 320A. In this embodiment, video data captured using a camera unit is used. However, in the present invention, in addition to the camera, for example, a hard disk, a floppy disk, an MO (Magneto-Optical disk), a CD-R (Compact Disk Recordable). ), Video data stored in an external storage device such as a magnetic tape may be read and used.

  When the video data is input from the camera unit 310A, the encoding unit 320A encodes and outputs the encoded data to the video transmission unit 330A. At the time of encoding, encoding is performed according to video encoding control parameters (resolution and frame rate) set in advance as default values. When a video encoding control parameter is input from the video quality control signal receiving unit 390A, the value of the held video encoding control parameter is updated with the input value. In this embodiment, MPEG4 is used as the video data encoding method.

  When the encoded video data is input from the encoding unit 320A, the video transmission unit 330A transmits the video data to the conference server apparatus 100 through the communication path 510A.

  Video receiving unit 340A receives video data from conference server apparatus 100 using communication path 520A and outputs the video data to decoding unit 350A.

  When the encoded video data is input from the video receiving unit 340A, the decoding unit 350A decodes and outputs the decoded video data to the display unit 360A. In this embodiment, MPEG4 is used as the video data encoding method.

  Display unit 360A displays the video data input from decoding unit 350A. The display unit 360A is a display device such as a CRT (Cathode Ray Tube) display or a liquid crystal display.

  The user interface unit 370A has a pointing device such as a mouse and an input device such as a keyboard. The user inputs layout information desired by the user using these devices. The user interface unit 370A updates the cursor display position of the pointing device displayed on the display unit 360A according to the operation of the user's input device, and when detecting the input of layout information, the user interface unit 370A sends the layout control signal to the layout control signal transmission unit 380A. Output layout information. When the user inputs layout information, the layout information can be visually changed in the same manner as a general window object operation. Alternatively, as shown in FIG. 10, choices of predetermined layout patterns 1040A, 1050A, and 1060A that can be used may be displayed on the display unit 360A according to a user instruction. In this case, the user selects a pattern determined to be closest to the desired layout using the input device. Each pattern corresponds to layout information, and the layout information corresponding to the selected pattern is regarded as the layout information desired by the user. When the layout information desired by the user is determined, the layout information is output to layout control signal transmission unit 380A.

  When layout information is input from the user interface unit 370A, the layout control signal transmission unit 380A adds an IP header including the IP address of the destination conference client apparatus 300A to the layout information and generates an IP packet. This IP packet is called a layout control signal. When the layout control signal transmission unit 380A generates the layout control signal, the layout control signal transmission unit 380A transmits the layout control signal to the conference server apparatus 100 using the communication path 530A.

  The video quality control signal receiving unit 390A receives the video quality control signal from the conference server device 100 using the communication path 540A. The video quality control signal receiving unit 390A extracts the video encoding control parameter 1030A from the IP payload portion of the video quality control signal, and outputs the extracted video encoding control parameter 1030A to the encoding unit 320A.

  Next, as an example of processing in the conference server device 100 and the conference client devices 300A, 300B, 300C, and 300D, the layout information managed inside the conference server device 100 by each conference client device is changed, A mode that the quality of the image | video which each conference client apparatus sends out is controlled dynamically is shown. The conference client devices 300A, 300B, 300C, and 300D are distinguished by identification numbers 1, 2, 3, and 4.

  Immediately after the start of the conference, as shown in FIG. 11, the layout information of each conference client device stored in the layout information storage unit 140 is set to default layout information 1070A, 1070B, 1070C, 1070D. . The layout information 1070A, 1070B, 1070C, and 1070D are all arranged with the same video size ((W, H) = (50, 50)) so as not to overlap each other. Based on this layout information, the video composition unit 150 synthesizes the video, and the display units 360A, 360B, 360C, and 360D of each conference client device have the same video size of each conference participant as shown in FIG. The composite screen is displayed. Further, as a default, in the encoding units 320A, 320B, 320C, and 320D of each conference client device, the video encoding control parameters when performing encoding are all 640 × 480 and the frame rate as shown in FIG. Is set to 30 fps.

  Next, consider a situation in which each user participating in the conference has changed his / her layout information. Users participating in the conference can make changes to the layout of the composite video that they are currently viewing using the user interface units 370A, 370B, 370C, and 370D, respectively. Now, it is assumed that layout information 1080A, 1080B, 1080C, 1080D as shown in FIG. The layout information is transmitted to the conference server apparatus 100 through the layout control signal transmission units 380A, 380B, 380C, and 380D of each conference client apparatus and the communication paths 530A, 530B, 530C, and 530D. When stored in the layout information storage unit 140 through the layout control signal receiving unit 130, the video composition unit 150 performs video composition as shown in FIG. 15 with reference to the layout information. At the same time, a determination value calculation request related to each conference client device is input from the layout control signal receiving unit 130 to the determination value calculation unit 180. When a determination value calculation request is input, the determination value calculation unit 180 obtains layout information 1080A, 1080B, 1080C, and 1080D from the layout information storage unit 140. Based on this information, a determination value for each conference client device is obtained. Calculation is performed as shown in FIG. For example, the determination value for the conference client apparatus 300A is calculated as follows. First, attention is paid to layout information other than that acquired from 300A among the acquired layout information, that is, 1080B, 1080C, and 1080D. Among these, paying attention to the line corresponding to the identification number 1 (= identification number of 300A), the horizontal length (W) of the video included in this line is extracted. In this case, the W value extracted from 1080B is 25, the W value extracted from 1080C is 25, and the W value extracted from 1080D is 25. The maximum value among these three values is set to a determination value 1090A of 300A. The determination value calculation unit 180 performs such processing also for the other conference client devices 300B, 300C, and 300D. In the case of the conference client apparatus 300B, the W values extracted from the layout information 1080A, 1080C, and 1080D are 50, 100, and 100, respectively, and therefore the determination value 1090B of 300B is calculated as 100. In the case of the conference client apparatus 300C, the W values extracted from the layout information 1080A, 1080B, and 1080D are 50, 100, and 25, respectively. Therefore, the determination value 1090C of 300C is calculated as 100. In the case of the conference client apparatus 300D, the W values extracted from the layout information 1080A, 1080B, and 1080C are 50, 25, and 25, respectively. Therefore, the determination value 1090D of 300D is calculated as 50. The determination value calculation unit 180 inputs the determination values 1090A, 1090B, 1090C, and 1090D calculated as described above to the video quality control unit 190, respectively.

  Receiving each determination value, the video quality control unit 190 converts the determination value into a video encoding control parameter. The video quality control unit 190 internally stores a correspondence table of determination values and video encoding control parameters as shown in FIG. 8, and calculates the video encoding control parameters as shown in FIG. 17 with reference to this correspondence table. . For example, in the case of the determination value 1090A, the video encoding control parameter 1100A corresponding to this is calculated with a resolution of 320 × 240 dots and a frame rate of 15 fps. Similarly, the video encoding control parameters 1100B, 1100C, and 1100D corresponding to the determination values 1090B, 1090C, and 1090D are calculated as illustrated in FIG.

  When the video encoding control parameters 1100A, 1100B, 1100C, and 1100D are calculated, these values are sent to each conference client device through the video quality information transmission unit 200 and the communication paths 540A, 540B, 540C, and 540D.

  The conference client apparatuses 300A, 300B, 300C, and 300D receive the video encoding control parameters 1100A, 1100B, 1100C, and 1100D, respectively. These video encoding control parameters are input to the encoding units 320A, 320B, 320C, and 320D, respectively, and the video quality at the time of encoding is adjusted accordingly. In this case, compared to the default video encoding control parameter, the video encoding control parameter 1100A of the video transmitted by the conference client apparatus 300A has been changed. (1100B, 1100C, and 1100D do not change from the default parameters shown in FIG. 13 as a result.) That is, in this case, the other conference client devices 300B, 300C, and 300D use this video for the video of the conference client device 300A. Is displayed in a smaller size than the others, it is considered that the judgment value of 300A is low. For this reason, the video quality of the camera video encoding in the encoding unit 320A is lowered, and as a result, the amount of data transmitted using the communication path 510A is saved.

  Thereafter, when each user gives an instruction to change the layout information, the video quality of the camera video encoding of each conference client device is adjusted in the same manner as described above.

  The detailed internal configurations and operations of the conference server device 100 and the conference client devices 300A to 300D have been described above as the first embodiment of the present invention. According to the present invention, it is possible to control video data transmitted from a client to a server based on video reference statuses of a plurality of clients participating in a conference, thereby saving communication bandwidth.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to FIGS. 1 and 3 to 18.

  FIG. 1 shows a system configuration of a multipoint video conference system to which the present invention is applied. FIG. 1 includes a conference server device 100 and four conference client devices 300A, 300B, 300C, and 300D. The conference client devices 300A, 300B, 300C, and 300D are connected to the conference server device 100 via the networks 500A, 500B, 500C, and 500D, respectively.

  The conference client apparatuses 300A, 300B, 300C, and 300D include camera apparatuses 310A, 310B, 310C, and 310D, respectively. Each conference client device has a function of transmitting video data captured using the camera device to the conference server device 100 through the networks 500A, 500B, 500C, and 500D. In addition, the conference client devices 300A, 300B, 300C, and 300D have a function of receiving the video data transmitted from the conference server device 100 and displaying it on the respective display devices 360A, 360B, 360C, and 360D. Further, it has a function of transmitting control information to the conference server apparatus 100 through the networks 500A, 500B, 500C, and 500D.

  The conference server apparatus 100 has a function of receiving video data transmitted from the conference client apparatuses 300A, 300B, 300C, and 300D. Also, it has a function of combining a plurality of received video data into one video. It should be noted that a plurality of video compositions can be performed independently for each conference client device at the time of video composition. In addition, a composite video is transmitted to the conference client apparatuses 300A, 300B, 300C, and 300D through the networks 500A, 500B, 500C, and 500D. Similarly, it has a function of transmitting a control signal to each conference client device through the networks 500A, 500B, 500C, and 500D.

  Next, internal components related to the present invention in the conference server apparatus and the conference client apparatus will be described with reference to FIG. The functions represented by the internal components in the conference server device and the internal components in the conference client device may be realized by causing a computer to execute a program or may be realized by hardware.

  The conference server apparatus 100 includes, as its constituent elements, a video receiving unit 110 that receives video data using the communication path 510A, a decoding unit 120 that decodes video data, and a layout that uses the communication path 530A. A layout control signal receiving unit 130 that receives a control signal, a layout information storage unit 140 that stores layout information obtained by analysis from the layout control signal, a video synthesis unit 150 that performs video synthesis processing, and a synthesized video An encoding unit 160 that performs encoding, a video transmission unit 170 that transmits video data using the communication channel 520A, and a layout information notification signal transmission unit that transmits a layout information notification signal using the communication channel 540A. 210. In addition, although the operation | movement of each internal component of the conference server apparatus 100 shown below has demonstrated the process performed between conference client apparatuses 300A, it is not restricted to conference client apparatus 300A, 300B, 300C , 300D is performed in the same manner.

  The conference client device 300A uses, as its constituent elements, a camera unit 310A, an encoding unit 320A that encodes synthesized video, a video transmission unit 330A that transmits video data through the communication channel 510A, and a communication channel 520A. A video receiving unit 340A that receives video data, a decoding unit 350A that decodes video data, a display unit 360A that displays video, a user interface unit 370A, and a layout using the communication path 530A. A layout control signal transmission unit 380A that transmits a control signal, a layout information notification signal reception unit 220A that receives a layout information notification signal using the communication path 540A, and a determination value that calculates a determination value from the layout information The video encoding control parameter is calculated from the calculation unit 180A and the determination value. And a video quality control section 190A that. In FIG. 18, only one conference client device 300A is shown, but the conference server device 100 is connected to other conference client devices 300B, 300C, and 300D, each of which is connected to the conference client device 300A. It shall have similar internal components.

  Hereinafter, the internal components of the conference server apparatus 100 will be described in detail.

  The video receiving unit 110 performs the same operation as the video receiving unit 110 described in the first embodiment.

  The decoding unit 120 performs the same operation as the decoding unit 120 described in the first embodiment.

  The layout control signal receiving unit 130 receives a layout control signal from the conference client apparatus 300A using the communication path 530A. The layout control signal is transmitted as an IP packet including an IP header portion and an IP payload portion. The IP header portion includes the IP address of the conference client apparatus 300A that is the source of the layout control signal, and the IP payload portion includes Includes the layout information 1000A specified in the conference client apparatus 300A. The layout control signal receiving unit 130 analyzes the layout control signal, specifies the conference client device 300A that is the transmission source of the layout control signal from the source IP address included in the IP header portion, and the layout information 1000A from the IP payload portion. To extract.

  Here, as shown in FIG. 3, the layout information is the same as the set of the identification number, the composite position, the composite size, and the relative position value in the front-rear direction described in the first embodiment. Of course, the present invention is not limited to this, and may include, for example, a combined cut position, a combined cut size, and the like. The definition of the combined cut position and the combined cut size is the same as the definition described in the first embodiment. When the layout control signal receiving unit 130 extracts the layout information 1000A, the layout control signal receiving unit 130 outputs the layout information 1000A to the layout information storage unit 140 and transmits the layout information notification signal related to each of 300B, 300C, and 300D to the layout information notification signal transmission unit 210. Output the request.

  When the layout information 1000A is input from the layout control signal receiving unit 130, the layout information storage unit 140 stores the input layout information 1000A. If a new input is made from the conference client apparatus 300A in this state, the layout information 1000A is updated with a new input value. In addition, when an acquisition request for layout information 1000 </ b> A is input from the video composition unit 150, the layout information storage unit 140 outputs the stored layout information 1000 </ b> A to the video composition unit 150. In addition, when layout information 1000B, 1000C, and 1000D is input from the conference client devices 300B, 300C, and 300D, the layout information storage unit 140 similarly stores the layout information. Further, when an acquisition request for the three layout information 1000B, 1000C, and 1000D is input from the layout information notification signal transmission unit 210, the layout information storage unit 140 notifies the stored layout information 1000B, 1000C, and 1000D to the layout information notification. The signal is output to the signal transmission unit 210.

  The video composition unit 150 performs the same operation as the video composition unit 150 described in the first embodiment.

  The encoding unit 160 performs the same operation as the encoding unit 160 described in the first embodiment.

  The video transmission unit 170 performs the same operation as the video transmission unit 170 described in the first embodiment.

  When the layout information notification signal transmission unit 210 receives a layout information transmission request from the layout control signal reception unit 130, the layout information storage unit 140 receives the layout information 1000B, 1000C, A 1000D acquisition request is output. When three pieces of layout information 1000B, 1000C, and 1000D as shown in FIG. 6 are input from the layout information storage unit 140, an IP header including the IP address and the like of the conference client device 300A that is the transmission destination for these layout information. To generate an IP packet. This IP packet is called a layout information notification signal. When the layout information notification signal transmission unit 210 generates the layout information notification signal, the layout information notification signal transmission unit 210 transmits the layout information notification signal to the conference client apparatus 300A using the communication path 540A.

  Next, the internal components of the conference client apparatus 300A will be described in detail.

  The camera unit 310A performs the same operation as the camera unit 310A described in the first embodiment.

  The encoding unit 320A holds video encoding control parameters inside and performs the same operation as the encoding unit 320A described in the first embodiment.

  The video transmission unit 330A performs the same operation as the video transmission unit 330A described in the first embodiment using the communication path 510A.

  The video reception unit 340A performs the same operation as the video reception unit 340A described in the first embodiment using the communication path 520A.

  The decoding unit 350A performs the same operation as the decoding unit 350A described in the first embodiment.

  The display unit 360A performs the same operation as the display unit 360A described in the first embodiment.

  The user interface unit 370A has the same input device as the user interface unit 370A described in the first embodiment, and performs the same operation.

  The layout control signal transmission unit 380A performs the same operation as the layout control signal transmission unit 380A described in the first embodiment using the communication path 530A.

  The layout information notification signal receiving unit 220A receives a layout information notification signal from the conference server apparatus 100 using the communication path 540A. The layout information notification signal receiving unit 220A extracts the layout information 1000B, 1000C, and 1000D of the three conference client devices from the IP payload portion of the layout information notification signal, and outputs the extracted layout information to the determination value calculation unit 180A.

  When three pieces of layout information 1000B, 1000C, and 1000D are input from layout information notification signal receiving unit 220A, determination value calculation unit 180A calculates determination value 1090A as shown in FIG. 16 based on these layout information. . The determination value 1090A represents the degree to which the camera video transmitted from the conference client apparatus 300A has received attention from users of other conference client apparatuses. Here, the determination value 1090A is defined as the maximum value in the horizontal length (W) of the 300A video on the composite video viewed by the conference client devices 300B, 300C, and 300D. That is, in the layout information 1000B, 1000C, and 1000D in FIG. 6, pay attention to the line where the identification number is equal to 1 (= the identification number of 300A), and select the largest W value among them and select this as the determination value 1090A. It is determined. In the present embodiment, the determination value is calculated based on the three pieces of layout information excluding 1000A. However, the determination value may be calculated based on the four pieces of layout information including 1000A. That is, the layout information notification signal from the layout information notification signal transmission unit 210 includes four pieces of layout information 1000A, 1000B, 1000C, and 1000D, and the maximum W value of 300A of these four pieces of layout information is determined as a determination value 1090A. It may be defined. When determination value calculation unit 180A calculates determination value 1090A, it outputs it to video quality control unit 190A.

  When the determination value 1090A is input from the determination value calculation unit 180A, the video quality control unit 190A calculates the video encoding control parameter 1030A. The video encoding control parameter 1030A is a value used when the encoding unit 320A of the conference client apparatus 300A encodes the camera video, and here defines the resolution and frame rate at the time of encoding. When calculating the video coding control parameter 1030A, as shown in FIGS. 8 and 9, the video quality control unit 190A holds a correspondence table between the determination value and the video coding control parameter, and is referred to as appropriate. To calculate. Normally, the resolution and frame rate values included in the video encoding control parameter 1030A increase in accordance with the magnitude of the determination value 1090A, which corresponds to a higher quality video. When the video encoding control parameter 1030A is calculated, the video quality control unit 190A outputs this parameter to the encoding unit 320A.

  Next, as an example of processing in the conference server device 100 and the conference client devices 300A, 300B, 300C, and 300D, the layout information managed inside the conference server device 100 by each conference client device is changed, A mode that the quality of the image | video which each conference client apparatus sends out is controlled dynamically is shown. The conference client devices 300A, 300B, 300C, and 300D are distinguished by identification numbers 1, 2, 3, and 4.

  Immediately after the start of the conference, as shown in FIG. 11, the layout information of each conference client device stored in the layout information storage unit 140 is set to default layout information 1070A, 1070B, 1070C, 1070D. . The layout information 1070A, 1070B, 1070C, and 1070D are all arranged with the same video size ((W, H) = (50, 50)) so as not to overlap each other. Based on this layout information, the video composition unit 150 synthesizes the video, and the display units 360A, 360B, 360C, and 360D of each conference client device have the same video size of each conference participant as shown in FIG. The composite screen is displayed. Further, as a default, in the encoding units 320A, 320B, 320C, and 320D of each conference client device, the video encoding control parameters when performing encoding are all 640 × 480 and the frame rate as shown in FIG. Is set to 30 fps.

  Next, consider a situation in which each user participating in the conference has changed his / her layout information. Users participating in the conference can make changes to the layout of the composite video that they are currently viewing using the user interface units 370A, 370B, 370C, and 370D, respectively. Now, it is assumed that layout information 1080A, 1080B, 1080C, 1080D as shown in FIG. The layout information is transmitted to the conference server apparatus 100 through the layout control signal transmission units 380A, 380B, 380C, and 380D of each conference client apparatus and the communication paths 530A, 530B, 530C, and 530D. When stored in the layout information storage unit 140 through the layout control signal receiving unit 130, the video composition unit 150 performs video composition as shown in FIG. 15 with reference to the layout information. At the same time, the layout control signal receiver 130 outputs a layout information notification signal transmission request related to each conference client device to the layout information notification signal transmitter 210. For example, when a layout information notification signal transmission request to the conference client apparatus 300A is input, the layout information notification signal transmission unit 210 acquires three layout information 1080B, 1080C, and 1080D excluding 1080A from the layout information storage unit 140. These pieces of information are sent to the conference client apparatus 300A through the communication path 540A. Similarly, when a layout information notification signal transmission request is input to the conference client apparatus 300B, the layout information 1080A, 1080C, 1080D is acquired and transmitted to 300B, and a layout information notification signal transmission request to 300C is input. If layout information 1080A, 1080B, 1080D is acquired and transmitted to 300B, and a layout information notification signal transmission request is input to conference client apparatus 300D, layout information 1080A, 1080B, 1080C is acquired. Send to 300D.

  Upon receiving the layout information from the conference server device 100, each conference client device inputs the layout information to the determination value calculation unit 180A through the layout information notification signal reception unit 220A and calculates the determination value. For example, in the case of the conference client apparatus 300A, as shown in FIG. 16, the determination value for 300A is calculated as follows. The layout information received by the conference client apparatus 300A is three pieces of 1080B, 1080C, and 1080D. First, in the acquired layout information, attention is paid to a line corresponding to an identification number of 1 (= 300A identification number), and the horizontal length (W) of the video included in this line is extracted. In this case, the W value extracted from 1080B is 25, the W value extracted from 1080C is 25, and the W value extracted from 1080D is 25. The largest value among these three values, that is, 25 is set as the determination value 1090A of the conference client apparatus 300A. When the determination value calculation unit 180A calculates the determination value 1090A, the determination value calculation unit 180A inputs the determination value 1090A to the video quality control unit 190A.

  Upon receiving the determination value 1090A, the video quality control unit 190A converts the determination value into a video encoding control parameter. The video quality control unit 190A internally stores a correspondence table of determination values and video encoding control parameters as shown in FIG. 8, and calculates the video encoding control parameters as shown in FIG. 17 with reference to this correspondence table. . For example, in the case of the conference client apparatus 300A, the video encoding control parameter 1100A corresponding to the conference client apparatus 300A is calculated to have a resolution of 320 × 240 dots and a frame rate of 15 fps.

  When the video encoding control parameter 1100A is calculated, this value is input to the encoding unit 320A, and the video quality at the time of encoding is adjusted accordingly.

  Such processing is similarly performed by the other conference client apparatuses 300B, 300C, and 300D. In the case of the conference client apparatus 300B, the W values extracted from the received layout information 1080A, 1080C, and 1080D are 50, 100, and 100, respectively. Therefore, the determination value 1090B is calculated as 100, and the encoding unit 320B The video encoding is adjusted according to the parameter 1100B so that the resolution is 640 × 480 dots and the frame rate is 30 fps. In the case of the conference client apparatus 300C, the W values extracted from the received layout information 1080A, 1080B, and 1080D are 50, 100, and 25, respectively. Therefore, the determination value 1090C is calculated as 100, and the encoding unit 320C The video encoding is adjusted according to the parameter 1100C so that the resolution is 640 × 480 dots and the frame rate is 30 fps. In the case of the conference client apparatus 300D, the W values extracted from the received layout information 1080A, 1080B, and 1080C are 50, 25, and 25, respectively. Therefore, the determination value 1090D is calculated as 50, and the encoding unit 320D performs the process. The video encoding is adjusted according to the parameter 1100D so that the resolution is 640 × 480 dots and the frame rate is 30 fps.

  In this case, compared to the default video encoding control parameter, the video encoding control parameter 1100A of the video transmitted by the conference client apparatus 300A has been changed. (1100B, 1100C, and 1100D do not change from the default parameters shown in FIG. 13 as a result.) That is, in this case, the other conference client devices 300B, 300C, and 300D use this video for the video of the conference client device 300A. Is displayed in a smaller size than others, it is considered that the determination value of the conference client apparatus 300A is low. For this reason, the video quality of the camera video encoding in the encoding unit 320A is lowered, and as a result, the amount of data transmitted using the communication path 510A is saved.

  Thereafter, when each user gives an instruction to change the layout information, the video quality of the camera video encoding of each conference client device is adjusted in the same manner as described above.

  The detailed internal configurations and operations of the conference server device 100 and the conference client devices 300A to 300D have been described above as the second embodiment of the present invention. Although the configuration is different from that of the first embodiment, according to the present invention, the video data transmitted from the client to the server is controlled based on the reference status of the video of a plurality of clients participating in the conference, and the communication bandwidth is saved. It is possible to do as well.

Schematic of a multipoint video conference system. The block diagram which shows the structure of the conference server apparatus which concerns on the 1st Embodiment of this invention, and a conference client apparatus. The figure which shows the layout information example 1 which concerns on the 1st Embodiment of this invention. The figure which shows the synthetic | combination image example 1 which concerns on the 1st Embodiment of this invention. The figure which shows the coordinate axis in the synthetic | combination image | video which concerns on the 1st Embodiment of this invention. The figure which shows the layout information example 2 which concerns on the 1st Embodiment of this invention. The figure which shows the judgment value example 1 which concerns on the 1st Embodiment of this invention. The figure which shows the corresponding | compatible table of the determination value and video coding control parameter based on the 1st Embodiment of this invention. The figure which shows the example of a video coding control parameter which concerns on the 1st Embodiment of this invention. FIG. 4 is a diagram showing an example of a layout pattern according to the first embodiment of the present invention. FIG. 6 is a diagram showing a layout information example 3 according to the first embodiment of the present invention. The figure which shows the example 1 of a display screen which concerns on the 1st Embodiment of this invention. The figure which shows the video encoding control parameter example 2 which concerns on the 1st Embodiment of this invention. FIG. 6 is a diagram showing a layout information example 4 according to the first embodiment of the present invention. The figure which shows the example 1 of a display screen which concerns on the 1st Embodiment of this invention. The figure which shows the example 2 of the judgment value which concerns on the 1st Embodiment of this invention. The figure which shows the video encoding control parameter example 3 which concerns on the 1st Embodiment of this invention. The block diagram which shows the structure of the conference server apparatus and conference client apparatus concerning the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Conference server apparatus 110 ... Video receiving part 120 ... Decoding part 130 ... Layout control signal receiving part 140 ... Layout information storage part 150 ... Video composition part 160 ... Encoding part 170 ... Video transmission part 180, 180A ... Determination value calculation part 190 , 190A ... Video quality control unit 200 ... Video quality information transmission unit 210 ... Layout information notification signal transmission unit 220A ... Layout information notification signal reception units 300A, 300B, 300C, 300D ... Conference client devices 310A, 310B, 310C, 310D ... Camera Units 320A, 320B, 320C, 320D ... Encoding unit 330A ... Video transmission unit 340A ... Video reception unit 350A ... Decoding units 360A, 360B, 360C, 360D ... Display units 370A, 370B, 370C, 370D ... User interface unit 380A, 380B, 380C, 380D ... Layout control signal transmission unit 390A ... Video quality control signal reception unit 500A, 500B, 500C, 500D ... Network 510A, 520A, 530A, 540A ... Communication paths 1000A, 1000B, 1000C , 1000D ... Layout information 1010A ... Composite video 1020A, 1020B, 1020C, 1020D ... Determination value 1030A ... Video coding control parameter 1040A, 1050A, 1060A ... Layout pattern 1070A, 1070B, 1070C, 1070D ... Layout information 1080A, 1080B, 1080C, 1080D ... Layout information 1090A, 1090B, 1090C, 1090D ... Determination values 1100A, 1100B 1100C, 1100D ... video encoding control parameters

Claims (21)

Video receiving means for receiving first to Nth encoded video data representing the first to Nth videos from first to Nth (N is an integer of 2 or more) video display devices;
Decoding means for decoding the first to Nth encoded video data and outputting the first to Nth decoded video data;
The first to Nth video display information includes first to Nth layout information respectively indicating layouts of the first to Nth videos in one synthesized video obtained by synthesizing the first to Nth videos. Layout information receiving means for receiving from the apparatus;
Layout information storage means for storing the received first to Nth layout information;
Video synthesizing means for synthesizing first to Nth videos represented by the first to Nth decoded video data according to the first to Nth layout information to generate first to Nth synthesized videos; ,
Video transmission means for transmitting the first to Nth composite video to the first to Nth video display devices;
Determination value calculating means for calculating a determination value for the first video display device based on at least the layout of the first video in the second to Nth layout information of the first to Nth layout information; ,
Video quality control means for determining an encoding parameter to be applied in the first video display device from the calculated determination value;
Video quality control signal transmitting means for transmitting the determined encoding parameter to the first video display device;
A video composition device.   The video composition apparatus according to claim 1, wherein the determination value calculation unit calculates the determination value based on a size of the first video in the second to N-th layout information.   The video composition apparatus according to claim 2, wherein the determination value is calculated based on a largest or smallest size of the first video in the second to Nth layout information.   The first to N-th layout information includes a cut-out size from the first video, and the determination value calculation means sets the cut-out size from the first video in the second to N-th layout information. The video composition apparatus according to claim 1, wherein the determination value is calculated based on the determination value.   The determination value calculation means calculates the determination value based on a smallest or largest one of the cut sizes from the first video in the second to N-th layout information. Item 5. The video composition device according to Item 4.   Each of the first to Nth layout information includes hierarchy information of the first to Nth videos, and the determination value calculation means includes the first video hierarchy in the second to Nth layout information. The video synthesizing apparatus according to claim 1, wherein the determination value is calculated on the basis of the video.   The determination value calculation means calculates the determination value based on the smallest or largest one of the first video layers in the second to Nth layout information. The video composition device described in 1.   The said video quality control means has a table which matched the said determination value and the said encoding parameter, The said encoding parameter is determined based on the said table, The one of Claim 1 thru | or 7 characterized by the above-mentioned. The video composition device described in 1.   9. The video composition apparatus according to claim 1, wherein the encoding parameter includes at least one of a resolution and a frame rate. Video generation means for generating video data;
Encoding means for encoding the video data according to encoding parameters and outputting encoded video data;
Video transmission means for transmitting the encoded video data to a video synthesizer;
Layout information transmitting means for transmitting layout information for video composition to the video composition device;
Video receiving means for receiving composite video data;
Display means for displaying the composite video data;
Layout information receiving means for receiving layout information respectively used by one or more other video display devices;
Determination value calculating means for calculating a determination value for the own device from the layout of the image of the own device in each received layout information;
Video quality control means for determining an encoding parameter to be applied from the calculated determination value, and notifying the determined encoding parameter to the encoding means;
A video display device comprising:   The video display device according to claim 10, wherein the determination value calculation unit calculates the determination value based on a size of the video of the device itself in each received layout information.   The video display device according to claim 11, wherein the determination value is calculated based on a largest or smallest image size of the own device in each received layout information.   Each of the received layout information includes a cut size from a video represented by the video data, and the determination value calculation means calculates the determination value based on the cut size in each layout information. Item 13. The video display device according to Item 10.   14. The video display device according to claim 13, wherein the determination value calculation unit calculates the determination value based on a smallest or largest one of the cutout sizes of the video in each of the layout information.   Each of the layout information includes hierarchical information of a plurality of videos to be combined, and the determination value calculation means calculates the determination value based on the video hierarchy of the own device in each of the layout information. The video display device according to claim 10.   16. The video display according to claim 15, wherein the determination value calculation means calculates the determination value based on the smallest or largest of the video layers of the device itself in each of the layout information. apparatus.   17. The video quality control means includes a table in which the determination value and the encoding parameter are associated with each other, and determines the encoding parameter based on the table. The video display device described in 1.   18. The video display device according to claim 10, wherein the encoding parameter includes at least one of resolution and frame rate. Generating first to Nth video data in first to Nth video display devices (N is an integer of 2 or more);
Encoding the first to Nth video data according to the first to Nth encoding parameters to generate first to Nth encoded video data;
Transmitting the first to Nth encoded video data to a video synthesizer;
Transmitting first to N-th layout information for video composition to the video composition device;
Decoding the first to Nth encoded video data to generate first to Nth decoded video data;
Synthesizing the first to Nth videos represented by the first to Nth decoded video data according to the first to Nth layout information to generate first to Nth synthesized videos;
Transmitting the first to Nth composite images to the first to Nth image display devices;
A determination value for the first video display device is calculated based on at least the layout of the first video in the second to Nth layout information among the first to Nth layout information,
Determining the first encoding parameter to be applied by the first video display device from the calculated determination value;
Video composition method. Receiving first to Nth encoded video data from first to Nth (N is an integer of 2 or more) video display devices;
Decoding the first to Nth encoded video data and outputting the first to Nth decoded video data;
Receiving first to Nth layout information for video composition from the first to Nth video display devices;
Storing the received first to Nth layout information;
Synthesizing the first to Nth videos represented by the first to Nth decoded video data according to the first to Nth layout information to generate first to Nth synthesized videos;
Transmitting the first to Nth composite images to the first to Nth image display devices;
Calculating a determination value for the first video display device based on at least the layout of the first video in the second to N-th layout information among the first to N-th layout information;
Determining an encoding parameter to be applied in the first video display device from the calculated determination value;
Transmitting the determined encoding parameters to the first video display device;
A program that causes a computer to execute. Generating video data; and
Encoding the video data according to encoding parameters and outputting the encoded video data;
Transmitting the encoded video data to a video synthesizer;
Transmitting layout information for video composition to the video composition device;
Receiving composite video data; and
Displaying the composite video data;
Receiving layout information respectively used by one or more other video display devices;
Calculating a determination value for the own device from the layout of the image of the own device in each received layout information;
Determining an encoding parameter to be applied from the calculated determination value;
For causing the video display device to execute the program.

Images