JP7056576B2 - Transmitter, transmit method, receiver and receive method - Google Patents

Description

The present art relates to a transmitting device, a transmitting method, a receiving device, and a receiving method, and more particularly to a transmitting device for transmitting a coded stream composed of coded image data of a plurality of views that can be synthesized.

For example, Patent Document 1 describes that a plurality of images are joined together and combined to obtain a panoramic image that reproduces a wide field of view. When a coded stream consisting of coded image data of multiple views that can be combined is sent, the receiving side is an algorithm that connects the image data of multiple views with high accuracy in order to display the combined image. Is required. Here, the composite position is determined by moving one pixel at a time and taking a pixel correlation value in each case for comparison. As described above, the process of connecting the image data of a plurality of views with high accuracy involves a process of determining the composite position, which makes the processing load heavy.

Japanese Unexamined Patent Publication No. 2000-299804

An object of this technique is to make it possible to reduce the processing load of image composition processing on the receiving side.

The concept of this technology is
A transmitter that sends a coded stream consisting of coded image data from multiple views that can be combined, and a transmitter.
It is in a transmission device including an information insertion unit that inserts information of a synthesis point into the coded image data of each of the above views.

In the present technique, the transmission unit transmits a coded stream composed of coded image data of a plurality of views that can be synthesized. The information insertion unit inserts information on the composite point into the coded image data of each view. For example, the composite point may include a reference point and a non-reference point at the time of synthesis. Further, for example, the information of the composite point may be composed of x-coordinates and y-coordinates having the top left of the image as the origin.

As described above, in the present technique, the information of the synthesis point is inserted into the coded image data of each view and transmitted. Therefore, on the receiving side, it is possible to perform the synthesis processing based on the information of the synthesis point, and it is possible to reduce the processing load.

In the present technique, for example, information for identifying the view may be added to each of the information of the synthesis point inserted in the coded image data of each view. As a result, the receiving side can easily identify each view.

Further, in the present technique, for example, information indicating whether or not the view is a representative view may be added to each of the information of the composite point inserted in the coded image data of each view. This makes it possible for the receiving side to select an area corresponding to the representative view as a part of the composite image to be displayed by default.

Further, in the present technique, for example, information indicating a set of composite views may be added to each of the composite point information inserted in the coded image data of each view. This makes it possible for the receiving side to specify a set of composite views and obtain a desired composite image.

In addition, other concepts of this technology
It has a receiver that receives a coded stream consisting of coded image data of multiple views that can be synthesized.
Information on the composition point is inserted in the coded image data of each of the above views.
The receiving device further includes a compositing unit that synthesizes image data of a predetermined number of views selected from the plurality of views based on the information of the compositing points to obtain composite image data.

In the present technique, the receiving unit receives a coded stream composed of coded image data of a plurality of views that can be synthesized. Information on the composition point is inserted in the coded image data of each view. The compositing unit combines image data of a predetermined number of views selected from a plurality of views based on the information of the compositing points to obtain composite image data.

As described above, in the present technique, the image data of a predetermined number of views selected from a plurality of views is synthesized based on the information of the synthesis point inserted into the coded image data of each view and sent. be. Therefore, the processing load can be reduced.

In the present technique, for example, information indicating a set of composite views is added to each of the information of the composite points inserted in the coded image data of each view, and the composite unit is the set of composite views. The image data of the views belonging to the set specified by the specified information may be combined. This makes it possible to specify a set of composite views and obtain a desired composite image.

Further, in the present technique, for example, a display processing unit for selectively extracting image data of a part of a region from the composite image data to obtain display image data may be further provided. This makes it possible to selectively display an image in a part of the composite image.

In this case, for example, an operation unit in which a user operates a part of the area may be further provided. This allows the user to arbitrarily adjust the display area in the composite image. Then, in this case, for example, the display data for indicating the position in the composite image of the extracted partial region may be superimposed on the display image data. This allows the user to easily recognize the position of the display area in the composite image.

Further, in the present technology, for example, information indicating whether or not the view is a representative view is added to each of the information of the composite point inserted in the coded image data of each view, and the display processing unit defaults. Then, the image data of the area of the representative view may be taken out from the composite image data to obtain the image data for display. As a result, by default, it is possible to display the representative view image among the composite images.

According to this technique, it is possible to reduce the processing load of the image composition processing on the receiving side. The effects described herein are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a block diagram which shows the configuration example of the transmission / reception system as an embodiment. It is a block diagram which shows the configuration example of a transmission device. It is a figure which shows the structural example of a stitching data SEI message. It is a figure which shows the content of the main information in the structural example of a stitching data SEI message. It is a figure which shows an example of the sequence of the coded image data of N views. It is a block diagram which shows the configuration example of a receiving device. It is a figure which shows the identification example of the set of the composite view by the value of information "tier". It is a figure which shows an example of the image of the view 1, the view 2, and the view 3 belonging to the set of the composite view designated by "tier1". It is a figure which shows the example of the result of the synthesis processing. It is a figure which shows an example of the positional relationship between the master point M1 which is a composite point included in views 1 and 2 and the target point T1. It is a figure for demonstrating an example of the procedure of a synthesis process. It is a figure for demonstrating an example of the procedure of a synthesis process. It is a figure which shows the flow of the composition processing in an image composition / display selection part. It is a figure which shows an example of the display when the whole composite image (composite view) is displayed on the display part. It is a figure which shows the example of the display of the composite image by the user interaction in the case of displaying a part area of a synthetic image (composite view).

Hereinafter, embodiments for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The explanation will be given in the following order.
1. 1. Embodiment 2. Modification example

<1. Embodiment>
[Transmission / reception system]
FIG. 1 shows a configuration example of a transmission / reception system 10 as an embodiment. The transmission / reception system 10 has a transmission device 100 and a reception device 200.

The transmission device 100 carries the transport stream TS as a container on a broadcast wave and transmits it. The transport stream TS includes a coded stream (video stream) composed of coded image data of a plurality of views that can be synthesized. In this case, the coded stream may be composed of one stream or a plurality of coded streams.

Information on the composition point is inserted into the coded image data of each view. By inserting the information of the composition point into the coded image data of each view in this way, the receiving side can perform the composition processing based on the information of the composition point, and it is possible to reduce the processing load. Become.

There are two or more composite points in the image of each view. This composite point includes a reference point that serves as a reference at the time of synthesis and other non-reference points. Hereinafter, the reference point is appropriately referred to as a “master point”, and the non-reference point is referred to as a “target point”. Here, the information of the composite point is composed of, for example, the x-coordinate "x_coordinate" and the y-coordinate "y_coordinate" with the top-left of the image as the origin.

Information "view_id" that identifies the view is added to each of the information of the composite point inserted in the coded image data of each view. By adding the information for identifying the views in this way, the receiving side can easily identify each view.

Further, information "masterview_flag" indicating whether or not the view is a representative view is added to each of the information of the composite point inserted in the coded image data of each view. By adding information indicating whether or not the view is a representative view in this way, the receiving side can select an area corresponding to the representative view as a part of the composite image to be displayed by default. ..

Further, information "tier" indicating a set of composite views is added to each of the composite point information inserted in the coded image data of each view. By adding the information indicating the set of composite views in this way, the receiving side can specify the set of composite views and obtain a desired composite image.

The receiving device 200 receives the above-mentioned transport stream TS transmitted on the broadcast wave from the transmitting device 100. As described above, the transport stream TS includes a coded stream (video stream) composed of coded image data of a plurality of views that can be synthesized. Then, information on the composite point is inserted in the coded image data of each view.

The receiving device 200 obtains composite image data by synthesizing image data of a predetermined number of views selected from a plurality of views based on the information of the synthesis point. For example, the receiving device 200 synthesizes image data of a view belonging to the set specified by the specified information of the set of the composite view. The designation of the set of the composite view is performed by, for example, a user operation.

The receiving device 200 selectively extracts, for example, image data for a part of a region, for example, an image size of one view from the composite image data, and obtains image data for display. As a result, the image of a part of the composite image is selectively displayed. For example, by default, the image data in the area of the representative view is fetched based on the information of the representative view, and the image of the representative view is displayed. The adjustment of this part of the area is performed by, for example, a user operation. Then, in this case, the receiving device 200 displays, for example, the position in the composite image of the extracted partial region on the display image and provides it for the user's convenience.

"Configuration of transmitter"
FIG. 2 shows a configuration example of the transmission device 100. The transmission device 100 includes a control unit 101, a user operation unit 102a, a display unit 102b, an uncompressed data buffer (dpb: decoded picture buffer) 103, a video encoder 104, and a compressed data buffer (cpb: coded picture buffer). ) 105, a TS formatter 106, and a transmission unit 107.

The control unit 101 includes a CPU (Central Processing Unit) and controls the operation of each unit of the transmission device 100. A user operation unit 102a and a display unit 102b constituting the user interface are connected to the control unit 101. The user can perform various input operations by the user operation unit 102a. For example, the stitch composition information can be set from the user operation unit 102a. This stitch composition information includes information on composition points (master point, target point), view identification information, representative view information, composition view group information, and the like.

The video encoder 104 inputs video data VD1, VD2, ..., VDN of a plurality of view to be synthesized, here N views, through the uncompressed data buffer 103, encodes them with AVC, HEVC, or the like, and each of them is encoded. Obtain a coded stream (video stream) consisting of the coded image data of the view. In this case, the coded stream is composed of one or more streams.

At this time, the video encoder 104 inserts stitch composition information into the coded image data of each view. Specifically, a newly defined stitching data SEI message (Stitching_data SEI message) having stitch composition information is inserted in the "SEIs" part of each access unit (AU) having encoded image data for each picture. do.

FIG. 3 shows a structural example (Syntax) of a stitching data SEI message. FIG. 4 shows the contents (Semantics) of the main information in the structural example. The 8-bit field of "byte_length" indicates the subsequent size of this SEI message in bytes. The 16-bit field of "number_of_stitching_sets" indicates the number of stitch composition information. The following fields exist repeatedly for this number.

The 16-bit field of "stp_id" indicates the identification information of the point (combination point) for stitch synthesis. The 16-bit field of "view_id" indicates the identification information of the view including the stitch composition point.

The 16-bit field of "master_point_x_coordinate" indicates the x-coordinate of the master point with the top-left of the image as the origin. The 16-bit field of "master_point_y_coordinate" indicates the y-coordinate of the master point with the top left of the image as the origin. The 16-bit field of "target_point_x_coordinate" indicates the x-coordinate of the target point with the top left of the image as the origin. The 16-bit field of "target_point_y_coordinate" indicates the y-coordinate of the target point with the top left of the image as the origin.

The 4-bit field of "tier" indicates a set of composite views. The 1-bit field of "masterview_flag" indicates whether or not the view including the stitch composition point is a representative view. For example, "1" indicates that it is a representative view, and "0" indicates that it is not a representative view.

FIG. 5 shows an example of a sequence of coded image data of N views. The stitching data SEI message described above is inserted into the coded image data (access unit) of each picture. In the illustrated example, regarding the first picture and the second picture, "Masterview = 1" is set in the stitch composition information inserted in the coded image data of view 1, and view 1 becomes the representative view. It is set.

Further, regarding the third and subsequent pictures, "Masterview = 1" is set in the stitch composition information inserted in the coded image data of the view 2, and the view 2 is set as the representative view. As will be described later, the representative view is preferentially displayed by default when a part of the composite image is selectively displayed.

Returning to FIG. 2, the compressed data buffer (cpb) 105 temporarily stores the coded stream generated by the video encoder 104. The TS formatter 106 reads out the coded stream stored in the compressed data buffer 104, converts it into a PES packet, further converts it into a transport packet and multiplexes it, and obtains a transport stream TS as a multiplexed stream. The transmission unit 107 carries the transport stream TS obtained by the TS formatter 106 on a broadcast wave or a net packet and transmits it to the receiving device 200.

The operation of the transmission device 100 shown in FIG. 2 will be briefly described. The moving image data VD1, VD2, ..., VDN of N views that can be combined are supplied to the video encoder 104 via the uncompressed data buffer (dpb) 103. In the video encoder 104, each of the moving image data VD1, VD2, ..., VDN is encoded by AVC, HEVC, or the like to obtain a coded stream (video stream).

Further, in the video encoder 104, stitch composition information including composition point (master point, target point) information, view identification information, representative view information, composition view group information, and the like is inserted into the coded image data of each view. To. Specifically, in the video encoder 104, a stitching data SEI message (see FIG. 3) having stitch synthesis information is inserted into the “SEIs” portion of the access unit (AU).

The coded stream generated by the video encoder 104 is temporarily stored in the compressed data buffer (cpb) 105. In the TS formatter 106, the coded stream stored in the compressed data buffer 105 is read out, converted into a PES packet, further converted into a transport packet, and multiplexed to obtain a transport stream TS as a multiplexed stream.

The transport stream TS obtained by the TS formatter 106 is sent to the transmission unit 107. In the transmission unit 107, the transport stream TS obtained by the TS formatter 106 is carried on a broadcast wave or a net packet and transmitted to the receiving device 200.

"Receiver configuration"
FIG. 6 shows a configuration example of the receiving device 200. The receiving device 200 includes a control unit (Central Processing Unit) 201, a user operation unit 201a, a receiving unit 202, a TS analysis unit 203, a compressed data buffer (cpb: coded picture buffer) 204, and a video decoder 205. It has an uncompressed data buffer (dpb: decoded picture buffer) 206, an image composition / display selection unit 207, a composition unit 208, and a display unit 209.

The control unit 201 includes a CPU (Central Processing Unit) and controls the operation of each unit of the receiving device 200. A user operation unit 201a constituting a user interface is connected to the control unit 201. The user can perform various input operations by the user operation unit 201a. For example, if the receiver can select a part of the composite image to be displayed, the user operation unit 201a can perform the selection operation.

The receiving unit 202 receives the transport stream TS transmitted from the transmitting device 100 on a broadcast wave or a packet on the net. The TS analysis unit 203 takes out the coded stream (coded image data of each view) included in the transport stream TS and sends it to the compressed data buffer (cpb) 204.

The video decoder 205 decodes all the views or the encoded image data of each view belonging to the set of composite views designated by the user from the user operation unit 201a according to the decoding ability of the receiving device 200. The specification of the set of the composite view is performed by setting the value of the information "tier" to a predetermined value. In this case, whether or not the view belongs to the specified set of composite views is identified by whether or not it has a composite point associated with the value of the set information "tier".

FIG. 7 shows an example of identifying a set of composite views by the value of the information “tier”. Note that this example shows a state in which the images of each view have already been combined based on the information of the composite points (master point, target point). For example, when the composition view set is specified by setting the value of the information "tier" to "tier1", each view of view 1 (view1), view 2 (view2), and view 3 (view3) is "view3". Since it has a composite point associated with "tier 1", it is a view that belongs to the specified composite view set. Further, for example, when the set of the composite view is specified by setting the value of the information "tier" to "tier2", each view of view 1 (view1), view 4 (view4), and view 5 (view5) is specified. Has a composite point associated with "tier2", so it is a view that belongs to the specified composite view set.

The video decoder 205 reads and decodes the encoded image data of each picture in each view stored in the compressed data buffer 204 at the decoding timing given by the DTS (Decoding Time stamp) of the picture. The image data of each picture of each view obtained by decoding is temporarily stored in the uncompressed data buffer 206. The image data of each picture is read from the uncompressed data buffer 206 at the timing given by the PTS (Presentation Time Stamp) of the picture, the moving image data VD of each view is obtained, and the image composition / display selection unit 207 is obtained. Will be sent to.

Further, the video decoder 205 extracts the parameter set and the SEI message inserted in the video stream and sends them to the control unit 201. The extracted information also includes a stitching data SEI message (see FIG. 3) inserted in the access unit of the coded image data of each view described above. As a result, the control unit 201 recognizes the stitch composition information of each view. As described above, the stitch composition information includes information on the composition point (master point, target point), view identification information, representative view information, composition view group information, and the like.

The image composition / display selection unit 207 performs composition processing on the image data of each view obtained in the uncompressed data buffer 206 to obtain composite image data. In the compositing process, image data is synthesized by superimposing composite points having the same identification information "stp_id". In this case, starting from the reference master point, the target points are rotated and scaled as necessary so that the target points overlap each other.

FIG. 8 shows an example of images of view 1, view 2 (view 2), and view 3 (view 3) belonging to the set of composite views specified by “tier 1” in FIG. 7 above. The stitch composition information of the view 1 includes various information related to the composition point identified by "stp_id1". Here, "view_id1" indicates that this composite point is included in view 1. It is also shown that the coordinates of the master point are (x11, y11). It is also shown that the coordinates of the target point are (x12, y12).

The stitch composition information of the view 2 includes various information related to the composition point identified by "stp_id1". Here, "view_id2" indicates that this composite point is included in view 2. It is also shown that the coordinates of the master point are (x21, y21). It is also shown that the coordinates of the target point are (x22, y22).

Further, the stitch composition information of the view 2 includes various information related to the composition point identified by "stp_id2". Here, "view_id2" indicates that this composite point is included in view 2. It is also shown that the coordinates of the master point are (x23, y23). It is also shown that the coordinates of the target point are (x24, y24).

The stitch composition information of the view 3 includes various information related to the composition point identified by "stp_id2". Here, "view_id3" indicates that this composite point is included in view 3. It is also shown that the coordinates of the master point are (x31, y31). It is also shown that the coordinates of the target point are (x32, y32).

In the composition process of view 1 and view 2, image data is combined by superimposing the composition points (master point, target point) identified by "stp_id1". Further, in the compositing process of the view 2 and the view 3, the image data is synthesized by superimposing the compositing points (master point, target point) identified by "stp_id2". As a result, a composite image data obtained by synthesizing the image data of views 1, 2 and 3 can be obtained. FIG. 9 shows an example of the result of the synthesis process.

The procedure of the synthesis process will be described. Here, the composition process of view 1 and view 2 will be described as an example. In this case, the composition process is performed using the composition points (master point, target point) identified by "stp_id1" included in the view 1 and the view 2.

FIG. 10A shows an example of the positional relationship between the master point M1 and the target point T1, which are synthetic points included in the view 1. The coordinates of the master point M1 are (x11, y11), and the coordinates of the target point T1 are (x12, y12). With the master point M1 as the base point, the target point T1 is at a position rotated by θ1 from the vertical axis. Further, the distance between the master point M1 and the target point T1 is r1.

FIG. 10B shows an example of the positional relationship between the master point M2 and the target point T2, which are composite points included in the view 2. The coordinates of the master point M2 are (x21, y21), and the coordinates of the target point T2 are (x22, y22). With the master point M2 as the base point, the target point T2 is located at a position rotated by θ2 from the vertical axis. Further, the distance between the master point M2 and the target point T2 is r2. Here, it is assumed that θ2 <θ1, r2> r1.

In step 1, as shown in FIG. 11A, the positions of the two views (images) are set so that the coordinates of the master point M1 of the view 1 and the master point M2 of the view 2 overlap. When the position is set in this way, the coordinates of the master points M1 and M2 overlap, but since θ2 <θ1, r2> r1 as described above, the coordinates of the target point T1 of the view 1 and the target point T2 of the view 2 Do not overlap.

In the subsequent steps, the position setting process is further advanced so that the coordinates of the target points T1 and T2 overlap. When θ2 = θ1 and r2 = r1, the coordinates of the target points T1 and T2 also overlap in the process of this step 1, so that the process of setting the positions of the two views (images) is completed. Become.

In step 2, as shown in FIG. 11B, as the first step for superimposing the coordinates of the target point T1 of the view 1 and the target point T2 of the view 2, the view 2 is set as a fulcrum and the master point M2 is used as a fulcrum. Rotate by θ1-θ2). As a result, the target point T2 of the view 2 is located on the extension of the line connecting the master points M1 and M2 and the target point T1 of the view 1. In this case, conversely, it is conceivable to rotate the view 1 by − (θ1-θ2) with the master point M1 as the fulcrum.

In step 3, as shown in FIG. 11C, the target point T2 of the view 2 is the target point of the view 1 as the second step for superimposing the coordinates of the target point T1 of the view 1 and the target point T2 of the view 2. Scaling is performed starting from the master points M1 to M2 so as to overlap T1. In this case, the distance between the master point M2 of the view 2 and the target point T2 is equal to the distance r1 between the master point M1 of the view 1 and the target point T1. In this case, conversely, it is conceivable to scale the target point T1 of the view 1 so as to overlap the target point T2 of the view 2.

In step 4, the image quality (luminance, color, etc.) of view 1 and view 2 is adjusted to match. In this case, for example, in the case of brightness adjustment, as shown in FIG. 12A, the rectangular area regulated by the master point and the target point is formed in a state where the master points and the target points overlap each other as described above. Set. Then, the luminance histograms of views 1 and 2 in this rectangular area are taken and adjusted so that they are the same. FIG. 12B shows the state after the luminance is adjusted. Although detailed description is omitted, the same applies to color adjustment. In addition to the case where both brightness adjustment and color adjustment are performed, it is also conceivable to perform only one of them.

FIG. 13 shows the flow of the compositing process in the image compositing / display selection unit 207. In step ST1, the process is started. Next, in step ST2, the positions are set so that the master points of the two views (images) having the same “stp_id” value overlap with each other at the same coordinates (see FIG. 11A).

Next, in step ST3, when the two target points (Target Pint) do not overlap on the coordinates even if the master points overlap on the same coordinates, and θ2 and θ1 are different, the master point is set as the center. Rotate either view so that θ2 is equal to θ1 or θ1 is equal to θ2 (see FIG. 11 (b)).

Next, in step ST4, if the two target points do not overlap on the coordinates even if θ2 becomes equal to θ1, scaling is performed to reduce or enlarge one view so that the two target points overlap (Fig.). 11 (c)). Next, in step ST5, the image quality of the two views is adjusted so that the histograms regarding brightness and / or color are the same in the rectangular region formed by the master point and the target point (see FIG. 12).

If there is another view to be combined, the above steps ST2 to ST5 are repeated by that amount, and when the composition of all the views is completed, the composition process is completed in step ST6.

Returning to FIG. 6, the image composition / display selection unit 207 may use the composite image data as a whole area or a part of the image composition / display selection unit 207 according to the display image frame size of the display unit 209 and the user selection operation from the user operation unit 201a. The image data is taken out and output as image data for display. For example, when the display frame size of the display unit 209 is a size capable of displaying the entire composite image, the image composite / display selection unit 207 extracts the image data of the entire area from the composite image data and displays it. Output as image data.

Further, for example, when the display frame size of the display unit 209 is a size capable of displaying a part of the composite image, for example, displaying only one view, the image composition / display selection unit 207 may display the composite image. The image data of a part of the area is taken out from the data and output as the image data for display. At this time, by default, the image composition / display selection unit 207 takes out the image data in the area of the representative view and outputs it as display image data.

Further, when the image composition / display selection unit 207 extracts the image data of a part of the area from the composite image data and outputs the image data as display image data in this way, the image corresponds to the user's selection operation from the user operation unit 201a. Part of the area from which data is to be taken out is changed, and the image data of the changed part of the area is taken out and output as image data for display. This enables scroll display according to the scroll operation of the user. It is also conceivable to sequentially select the area of each view as the display area instead of the scroll operation.

The control unit 201 generates UI display data for indicating which position of the image data in the composite image is in the fetched state. In this case, when the image data of the entire area is extracted from the composite image data, this UI display data indicates that there is no scroll component. On the other hand, when the image data of a part of the area is extracted from the composite image data, the extraction position is displayed by this UI display data, and it is shown that there is a scroll component.

The compositing unit 208 synthesizes the display image data generated by the image compositing / display selection unit 207 and the UI display data generated by the control unit 201, and sends them to the display unit 209. The display unit 209 is composed of, for example, an LCD (Liquid Crystal Display), an organic EL (Organic Electro-Luminescence) panel, or the like. The display unit 209 may be an external device connected to the receiving device 200.

FIG. 14 shows an example of display when the entire composite image (composite view) is displayed on the display unit 209. On the display screen, in addition to the composite image, the UI display data indicates that the composite image is the entire display and that there is no scroll component. FIG. 15 shows a case where a part of the adult image (composite view) is displayed, and shows an example of displaying the composite image by user interaction. In this case, the display area is scrolled by the scroll operation of the user. On the display screen, in addition to the image of a partial area, the UI display data indicates that the composite image is partially displayed together with its position, and that there is a scroll component with an arrow indicating the direction. Has been done.

FIG. 15A shows an example of the UI screen when the display area is the leftmost representative view area of the composite image. In this case, the arrow pointing to the right indicates that the display image position is at the left end and only scrolling to the right is possible. FIG. 15B shows an example of the UI screen when the display area is scrolled by P1 on the right side. In this case, the vertical bar indicates the relative position of the display area with respect to the entire composite image, and the arrows pointing to both the right and left indicate that scrolling to both the right and left is possible.

FIG. 15C shows an example of the UI screen when the display area is further scrolled to the right by P2. In this case, the vertical bar indicates the relative position of the display area with respect to the entire composite image, and the arrows pointing to both the right and left indicate that scrolling to both the right and left is possible. FIG. 15D shows an example of the UI screen when the display area is further scrolled to the right by P3. In this case, the left-pointing arrow indicates that the display image position is at the right end and only scrolling to the left is possible.

The operation of the receiving device 200 shown in FIG. 6 will be briefly described. The receiving unit 202 receives the transport stream TS transmitted from the transmitting device 100 on a broadcast wave or a packet on the net. This transport stream TS is supplied to the TS analysis unit 203. The TS analysis unit 203 takes out the coded stream (coded image data of each view) included in the transport stream TS and sends it to the compressed data buffer 204.

In the video decoder 205, the coded image data of all the views or each view belonging to the set of composite views designated by the user is decoded from the user operation unit 201a according to the decoding ability of the receiving device 200. The specification of the set of the composite view is performed by setting the value of the information "tier" to a predetermined value. In this case, in the video decoder 205, the encoded image data of each picture in each view stored in the compressed data buffer 204 is read out at the decoding timing given by the DTS (Decoding Time stamp) of the picture. It is decoded.

The image data of each picture of each view obtained by the video decoder 205 is temporarily stored in the uncompressed data buffer 206. The image data of each picture is read from the uncompressed data buffer 206 at the timing given by the PTS (Presentation Time Stamp) of the picture, the moving image data VD of each view is obtained, and the image composition / display selection unit 207 is obtained. Will be sent to.

Further, the video decoder 205 extracts the parameter set and the SEI message inserted in the video stream and sends them to the control unit 201. This extracted information also includes stitching data SEI messages (see FIG. 3) inserted in the access unit of the coded image data of each view. As a result, the control unit 201 recognizes the stitch composition information of each view. This stitch composition information includes information on composition points (master point, target point), view identification information, representative view information, composition view group information, and the like.

In the image composition / display selection unit 207, under the control of the control unit 201, the image data of each view obtained in the uncompressed data buffer 206 is subjected to a composition process to obtain the composite image data. In the compositing process, image data is synthesized by superimposing composite points having the same identification information "stp_id". In this case, starting from the reference master point, rotation and scaling are performed as necessary so that the target points overlap each other, and the composition is performed.

Further, in the image composition / display selection unit 207, image data of all or a part of the composite image data is extracted from the composite image data according to the display frame size of the display unit 209 and the user's selection operation from the user operation unit 201a. Is output as image data for display. The display image data is supplied to the display unit 209 via the compositing unit 208, and the display unit 209 displays the image based on the display image data. In this way, scalable display is performed according to the display image frame size of the display unit 209.

For example, when the display frame size of the display unit 209 is a size capable of displaying the entire composite image, the image composite / display selection unit 207 extracts and displays the image data of the entire area from the composite image data. It is output as image data. Therefore, in this case, the entire composite image is displayed on the display unit 209.

Further, for example, when the display frame size of the display unit 209 is a size capable of displaying a part of the composite image, for example, displaying only one view, the image composition / display selection unit 207 displays the composite image. The image data of a part of the area is extracted from the data and output as display image data. Therefore, in this case, a part of the composite image is displayed on the display unit 209. At this time, in the image composition / display selection unit 207, since the image data in the area of the representative view is taken out by default, the image of the representative view is displayed on the display unit 209.

Further, at this time, when the partial area for extracting the image data is changed according to the user's selection operation from the user operation unit 201a, the image composition / display selection unit 207 changes the image data of the partial area after the change. Is taken out and output as image data for display. Therefore, scroll display is also performed according to the scroll operation of the user.

Further, the control unit 201 generates UI display data for indicating which position of the image data in the composite image is in the fetched state. This UI display data is combined with the display image data in the composition unit 208. In this case, when the image data of the entire area is taken out from the composite image data, this UI display data indicates that there is no scroll component. On the other hand, when the image data of a part of the area is extracted from the composite image data, the extraction position is displayed by this UI display data, and it is shown that there is a scroll component. The user can easily perform the scroll operation by using this UI display data.

As described above, in the transmission / reception system 10 shown in FIG. 1, the transmission device 100 inserts the information of the synthesis point (master point, target point) into the coded image data of each view and transmits the data. Therefore, on the receiving side, it is possible to perform the composition processing of each view based on the information of the composition point, and it is possible to reduce the processing load.

Further, in the transmission / reception system 10 shown in FIG. 1, in the transmission device 100, information for identifying the view is added to each of the information of the synthesis points inserted in the coded image data of each view. Therefore, the receiving side can easily identify each view.

Further, in the transmission / reception system 10 shown in FIG. 1, in the transmission device 100, information indicating whether or not the view is a representative view is added to each of the information of the synthesis points inserted in the coded image data of each view. be. Therefore, on the receiving side, it is possible to select an area corresponding to the representative view as a part area to be displayed by default in the composite image.

Further, in the transmission / reception system 10 shown in FIG. 1, the transmission device 100 adds information indicating a set of composite views to each of the information of the composite points inserted in the coded image data of each view. Therefore, on the receiving side, it is possible to specify a set of composite views and obtain a desired composite image.

Further, in the transmission / reception system 10 shown in FIG. 1, the receiving device 200 is provided with an operation unit in which the user operates a part of the area to be displayed in the composite image. Therefore, the user can arbitrarily adjust the display area in the composite image. Then, in this case, the receiving device 200 superimposes the display data for indicating the position in the composite image of the extracted partial region on the display image data. Therefore, the user can easily recognize the position of the display area in the composite image and can smoothly perform the scroll operation.

<2. Modification example>
In the above-described embodiment, an example is shown in which a set of synthesis points is composed of one master point and one target point. However, it is also conceivable to configure a set of composite points with one master point and two or more target points. In that case, in the synthesis process, rotation or scaling (including non-linear rotation and scaling) is performed as necessary so that two or more target points overlap each other starting from the reference master point.

Further, in the above-described embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 is shown, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the portion of the receiving device 200 may be, for example, a configuration of a set-top box and a monitor connected by a digital interface such as (HDMI (High-Definition Multimedia Interface)). Note that “HDMI” is registered. It is a trademark.

Further, in the above-described embodiment, an example in which the container is a transport stream (MPEG-2 TS) is shown. However, this technique can be similarly applied to a system configured to be delivered to a receiving terminal using a network such as the Internet. In Internet distribution, it is often distributed in containers of ISOBMFF (or MP4) or other formats. That is, the container corresponds to a container of various formats such as the transport stream (MPEG-2 TS) adopted in the digital broadcasting standard and ISOBMFF (or MP4) used in the Internet distribution.

In addition, the present technology can also have the following configurations.
(1) A transmitter for transmitting a coded stream composed of coded image data of a plurality of views that can be synthesized, and a transmission unit.
A transmission device including an information insertion unit that inserts information of a synthesis point into the coded image data of each of the above views.
(2) The transmission device according to (1) above, wherein the composite point includes a reference point and a non-reference point at the time of synthesis.
(3) The transmission device according to (1) or (2) above, wherein the information of the composite point is composed of x-coordinates and y-coordinates whose origin is the top left of the image.
(4) The transmission device according to any one of (1) to (3) above, wherein information for identifying the view is added to each of the information of the synthesis point inserted in the coded image data of each view. ..
(5) Any of the above (1) to (4), in which information indicating whether or not the view is a representative view is added to each of the information of the composite points inserted in the coded image data of each of the views. The transmitter described in.
(6) The description according to any one of (1) to (5) above, wherein information indicating a set of composite views is added to each of the information of the composite points inserted in the coded image data of each view. Transmitter.
(7) A transmission step in which the transmission unit transmits a coded stream composed of coded image data of a plurality of views that can be synthesized.
A transmission method in which the information insertion unit includes an information insertion step for inserting information of a synthesis point into the coded image data of each of the above views.
(8) A receiver for receiving a coded stream composed of coded image data of a plurality of views that can be synthesized is provided.
Information on the composition point is inserted in the coded image data of each of the above views.
A receiving device further comprising a compositing unit for synthesizing image data of a predetermined number of views selected from the plurality of views based on the information of the compositing points to obtain composite image data.
(9) Information indicating a set of composite views is added to each of the information of the composite points inserted in the coded image data of each view.
The receiving device according to (8) above, wherein the compositing unit synthesizes image data of a view belonging to the set specified by the designated information of the set of the composite view.
(10) The receiving device according to (8) or (9) above, further comprising a display processing unit that selectively extracts image data of a part of the area from the composite image data to obtain display image data.
(11) Information indicating whether or not the view is a representative view is added to each of the information of the composite points inserted in the coded image data of each of the views.
By default, the display processing unit extracts image data in a region of the representative view from the composite image data to obtain display image data. The receiving device according to (10).
(12) The receiving device according to (10) or (11), further comprising an operation unit in which a user operates a part of the above area.
(13) The receiving device according to (12), wherein the display processing unit superimposes display data on the display image data in order to indicate a position in the composite image of the extracted partial region.
(14) The receiving unit has a receiving step of receiving a coded stream composed of coded image data of a plurality of views that can be synthesized.
Information on the composition point is inserted in the coded image data of each of the above views.
A receiving method in which the compositing unit further includes a compositing step of composing image data of a predetermined number of views selected from the plurality of views based on the information of the compositing points.

The main feature of this technique is that the information of the composite point (master point, target point) is inserted into the coded image data of each view and transmitted, and the receiving side receives the information of each view based on the information of the composite point. It is possible to perform the synthesis processing and reduce the processing load (see FIGS. 2 and 3).

10 ... Transmission / reception system 100 ... Transmission device 101 ... Control unit 102a ... User operation unit 102b ... Display unit 103 ... Uncompressed data buffer 104 ... Video encoder 105 ... Compression Data buffer 106 ... TS formatter 107 ... Transmitter 200 ... Receiver 201 ... Control unit 201a ... User operation unit 202 ... Receiver 203 ... TS analysis unit 204 ... Compressed data buffer 205 ・ ・ ・ Video decoder 206 ・ ・ ・ Uncompressed data buffer 207 ・ ・ ・ Image composition / display selection unit 208 ・ ・ ・ Composition unit 209 ・ ・ ・ Display unit

Claims (13)

A transmitter that sends a coded stream consisting of coded image data from multiple views that can be combined, and a transmitter.
It is provided with an information insertion unit for inserting information on a composite point including information on two points, a reference point and a non-reference point at the time of synthesis, into the coded image data of each of the above views by adding identification information of the composite point. Transmitter. The transmission device according to claim 1, wherein the information of the synthesis point is composed of x-coordinates and y-coordinates whose origin is the top left of the image. The transmission device according to claim 1, wherein information for identifying the view is added to each of the information of the synthesis point inserted in the coded image data of each view. The transmission device according to claim 1, wherein information indicating whether or not the view is a representative view is added to each of the information of the composite points inserted in the coded image data of each of the views. The transmission device according to claim 1, wherein information indicating a set of composite views is added to each of the information of the composite points inserted in the coded image data of each of the views. A transmission step in which the transmitter transmits a coded stream consisting of coded image data of a plurality of views that can be synthesized.
The information insertion unit inserts the information of the composite point including the information of the two points of the reference point and the non-reference point at the time of synthesis into the coded image data of each of the above views by adding the identification information of the composite point. A transmission method with an information insertion step. It has a receiver that receives a coded stream consisting of coded image data of multiple views that can be synthesized.
In the coded image data of each of the above views, the information of the composite point including the information of the two points of the reference point and the non-reference point at the time of synthesis is inserted with the identification information of the composite point added .
A receiving device further comprising a compositing unit for synthesizing image data of a predetermined number of views selected from the plurality of views based on the information of the compositing points to obtain composite image data. Information indicating a set of composite views is added to each of the information of the composite points inserted in the coded image data of each of the views.
The receiving device according to claim 7 , wherein the compositing unit synthesizes image data of a view belonging to the set designated by the designated information of the set of the composite view. The receiving device according to claim 7 , further comprising a display processing unit for selectively extracting image data of a part of the area from the composite image data to obtain image data for display. Information indicating whether or not the view is a representative view is added to each of the information of the composite points inserted in the coded image data of each of the views.
The receiving device according to claim 9 , wherein the display processing unit extracts image data in a region of the representative view from the composite image data to obtain image data for display by default. The receiving device according to claim 9 , further comprising an operation unit in which the user operates a part of the above area. The receiving device according to claim 11 , wherein the display processing unit superimposes display data on the display image data in order to indicate a position in the composite image of the extracted partial region. The receiver has a receive step of receiving a coded stream consisting of coded image data of a plurality of views that can be synthesized.
In the coded image data of each of the above views, the information of the composite point including the information of the two points of the reference point and the non-reference point at the time of synthesis is inserted with the identification information of the composite point added .
A receiving method in which the compositing unit further includes a compositing step of composing image data of a predetermined number of views selected from the plurality of views based on the information of the compositing points.

Images