JP7553872B2 - TRANSMISSION SYSTEM, TRANSMISSION METHOD, AND TRANSMISSION PROGRAM - Google Patents

Description

The present invention relates to a transmission system, a transmission method, and a transmission program.

SMPTE ST2110 is a method for transmitting uncompressed video and audio over a network. SMPTE ST2110-20 describes a method for storing video essence in the payload of RTP (Real-time Transport Protocol) packets for transmission (Non-Patent Document 1). In uncompressed video transmission, the number of packets per frame is constant as long as the video parameters and the pixel mapping method to the RTP packets do not change.

The payload portion of an RTP packet contains the pixel values of the pixels that make up the image, as well as the RTP payload header required for reconstruction on the receiving side. In particular, the RTP payload header contains the coordinates of the first pixel contained in the packet, so the image can be reconstructed on the receiving side by arranging the pixel values of the pixels accordingly.

SMPTE ST 2110-20:2017, Professional Media Over Managed IP Networks: Uncompressed Active Video

As it is now possible to transmit uncompressed video over networks, it is expected that many more streams will be transmitted within networks than before. Furthermore, in order to utilize technologies such as redundant transmission, it has become necessary to make changes to the network route configuration in real time. When making such configuration changes, it is essential that they do not adversely affect current stream transmission, and to do so, it is necessary to know which streams are flowing on which routes. Furthermore, if the system does not function properly after a route configuration change, it is desirable to know to what extent the streams are flowing correctly on the new route.

However, in next-generation transmission, which is expected to send uncompressed high-resolution video, extremely high processing performance is required to acquire all stream information within the operating network as is and process it in real time, and this can only be achieved in extremely limited environments or devices.

The present invention has been made in consideration of the above circumstances, and an object of the present invention is to enable understanding of the streams flowing through a network while reducing the load on the network.

To achieve the above objective, one aspect of the present invention is a transmission system comprising a duplication unit that duplicates a plurality of packets obtained by packetizing an uncompressed video stream, a transmission unit that transmits the duplicated packets via a narrowband path, and a generation unit that generates a low-resolution image from packets sampled due to packet loss occurring on the narrowband path.

One aspect of the present invention is a transmission method performed by a transmission system, which includes the steps of duplicating a plurality of packets obtained by packetizing an uncompressed video stream, transmitting the duplicated packets via a narrowband path, and generating a low-resolution image from packets sampled due to packet losses occurring on the narrowband path.

One aspect of the present invention is a transmission program that causes a computer to function as the above-mentioned transmission system.

According to the present invention, it is possible to understand the streams flowing through the network while reducing the load on the network.

FIG. 1 illustrates an example of a configuration of a transmission system according to a first embodiment; 13 is a flowchart showing a transmission process. FIG. 11 illustrates an example of a configuration of a transmission system according to a second embodiment. FIG. 13 illustrates an example of a configuration of a transmission system according to a third embodiment. 2 is an example of a hardware configuration.

Below, an embodiment of the present invention is described with reference to the drawings.

First Embodiment
FIG. 1 is a system configuration diagram showing an example of the configuration of a transmission system according to the first embodiment.

The transmission system of this embodiment includes at least one relay device 2 arranged on a network, and a management device 1 that aggregates information. The management device 1 and the relay device 2 are connected by a wireless or wired narrowband path 3 (communication path).

The relay device 2 is connected to the network devices 4 and 5 via a wired or wireless connection. The network devices 4 and 5 include, for example, transmitting and receiving devices. The relay device 2 relays a stream transmitted from at least one network device 4 to at least one network device 5. In the example shown in the figure, the relay device 2 relays multiple streams transmitted from multiple network devices 4 to the corresponding network devices 5, respectively.

The stream in this embodiment is uncompressed video that contains pixel value data and coordinate information, such as RTP packets packetized according to SMPTE ST 2110.

The transmission system of this embodiment simplifies management and operation when changing the route of an uncompressed video stream or adding or removing equipment, such as understanding the route of each stream in the current network and verifying that the system after the change is operating as expected.

In addition, the stream in this embodiment is described below as an example of an uncompressed video stream (hereinafter referred to as a "video stream"), but it may also be an uncompressed video and audio stream.

The illustrated relay device 2 has at least one receiving port 21, a duplication unit 22, at least one first transmitting port 23, and a second transmitting port 24.

Each receiving port 21 receives uncompressed video packets (hereinafter referred to as "video packets") in which a video stream has been packetized from the network device 4.

The duplicating unit 22 duplicates (mirrores) the video packets received at each receiving port 21 and sends them to the second transmitting port 24. In the illustrated example, the duplicating unit 22 duplicates video packets of multiple video streams. The duplicating unit 22 also outputs the received video packets to the corresponding first transmitting port 23. The first transmitting port 23 transmits the video packets to the next network device 5.

In this embodiment, RTP packets are used as video packets, but this is not limited to RTP packets and packets other than RTP packets may also be used.

The second transmitting port 24 transmits the video packet copied by the copying unit 22 to the management device 1 via the narrowband path 3.

The video packets output from the second transmission port 24 are transmitted to the management device 1 while causing packet loss by passing through the narrowband path 3. In the narrowband path 3, the video packets are discarded if the amount of video packets exceeds the amount that can be transmitted. Specifically, the second transmission port 24 discards video packets that do not fit within the bandwidth (transmission capacity) of the narrowband path 3, and outputs to the narrowband path 3 an amount of video packets that fits within the bandwidth. In this embodiment, the narrowband path 3 is used as the communication path between the relay device 2 and the management device 1, causing packet loss in the narrowband path 3.

The narrowband path 3 may be a path with a transmission capacity smaller than the transmission capacity of the video stream input to the relay device 2. In the example shown in FIG. 1, the narrowband path 3 may be a path with a transmission capacity smaller than the total transmission capacity of the video packets of the n (n=3) video streams input to the relay device 2.

For example, when an uncompressed video stream with an image size of HD (1280 x 720) and a frame rate of 60 fps flows from one network device 4 to relay device 2, the HD-sized video stream conforming to SMPTE2110 is approximately 1.3 Gbps. In this case, 1 GbE (Gigabit Ethernet), which has a transmission capacity smaller than 1.3 Gbps, may be used as the narrowband path 3 connecting relay device 2 and management device 1. In this case, even with one video stream, the transmission capacity is insufficient by approximately 0.3 Gbps, and packet loss occurs. Also, as shown in the figure, when video streams flow from n network devices 4 to relay device 2, narrowband path 3 may be a path with a transmission capacity smaller than n x 1.3 Gbps.

The transmission capacity of narrowband path 3 may be determined according to the processing performance of management device 1. For example, the processing volume for generating a low-resolution image is determined by the number of known points (the number of packets remaining after packet loss), the resolution (reduction ratio) of the low-resolution image to be output, etc. Therefore, if the processing performance of management device 1 is low, it is better to reduce the transmission capacity of narrowband path 3.

On the other hand, if the number of remaining packets decreases too much, the quality of the low-resolution image is likely to deteriorate, and even with the same transmission capacity of the narrowband path 3, the higher the resolution of the video stream, the greater the amount of packet loss, and the further the image quality of the low-resolution image deteriorates. Therefore, the transmission capacity of the narrowband path 3 may be determined not only based on the processing performance of the management device 1, but also based on a balance between the resolution of the target video stream, the number of streams, and the resolution of the low-resolution image to be output.

Note that since the number n of video streams flowing through the network (network device 4, relay device 2, network device 5) changes, narrowband path 3 may have a transmission capacity that causes packet loss when n becomes large to a certain extent. For example, for a group of HD-size video streams, if narrowband path 3 is set to 2 Gbps, packet loss will occur from n=2.

Alternatively, a path with a relatively large transmission capacity such as 10 GbE may be used as the narrowband path 3, and packet loss may occur when the number of video streams or bit rate input to the relay device 2 exceeds a predetermined threshold, and the low-resolution image generation process performed by the management device 1 may be stopped depending on whether or not there is packet loss.

Specifically, if no packet loss occurs, the management device 1 can obtain an image without any loss, and so may not generate a low-resolution image and may simply display the high-quality video stream, or may generate and display an image of sufficient quality using a low-load method. On the other hand, if packet loss occurs, a low-resolution image cannot be displayed unless it is generated, and so the management device 1 generates and displays a low-resolution image. In this way, the management device 1 may switch between generating a low-resolution image or not depending on whether or not there is packet loss.

Alternatively, the relay device 2 may be connected to the management device 1 via multiple narrowband paths 3 having different transmission capacities, and the packet loss rate may be changed by switching the narrowband path 3 to be used depending on the number of video streams and the bit rate. In this case, the relay device 2 has a second transmission port 24 for each narrowband path 3.

Specifically, the relay device 2 has at least one receiving port 21 and at least one first transmitting port 23, and a path is set depending on which receiving port 21 the video packet that arrived at is to be output from which first transmitting port 23. Therefore, a path selection unit (not shown) of the relay device 2 acquires the number of packets flowing into each receiving port 21, bit rate information, etc., and selects the first transmitting port 23 that will be the path. In addition, the path selection unit uses the number of packets flowing into each receiving port 21, bit rate information, etc., to determine which of the multiple second transmitting ports 24 to allocate the duplicated packet group to, thereby switching the narrowband path 3. The path selection unit is connected to the receiving port 21, the first transmitting port 23, and the second transmitting port 24.

The remaining packets sampled due to packet loss on narrowband path 3 are sent to management device 1.

The illustrated relay device 2 may be a device that relays a video stream transmitted from one network device 4 to one network device 5. In this case, the relay device 2 has one receiving port 21, a copying unit 22, one first transmitting port 23, and a second transmitting port 24.

Video packets transmitted from one or more relay devices 2 via narrowband path 3 are input to the management device 1.

The management device 1 has a receiving port 11, a processing unit 12, and a display unit 13. The processing unit 12 has a packet accumulation unit 121 and a generation unit 122. Note that the management device 1 does not have the display unit 13, and may have a display device (display) outside the management device 1.

The receiving port 11 receives video packets from the relay device 2 and sends them to the packet storage unit 121 of the processing unit 12.

Video packets are accumulated in the packet accumulation unit 121. The packet accumulation unit 121 classifies and accumulates the video packets by stream using information such as the source address of the video packets, and sends the accumulated video packets to the generation unit 12 at regular time intervals or when a certain amount of packets have been accumulated for each stream. In other words, the packet accumulation unit 121 sends those video packets to the generation unit 122 at the timing when the amount of packets necessary to generate a low-resolution image such as a thumbnail image has been accumulated for each stream.

The generation unit 122 generates a low-resolution image from the video packets sampled due to packet loss occurring on the narrowband path 3, and outputs the image to the display unit 13. The generation unit 122 generates a low-resolution image for each video stream. The generation unit 122 acquires pixel value data of the uncompressed video from the video packets and the coordinates (position information) of the pixel value data, and uses these to generate a low-resolution image. The low-resolution image shows an overview of the video stream, and serves as, for example, a thumbnail image.

Specifically, the generation unit 122 obtains the coordinates of the pixel value data held in the media payload from the payload header of an RTP packet (video packet) defined in SMPTE 2110. The RTP packet has an RTP header, a payload header that contains information about the pixel value data (such as coordinates), and a media payload that contains the pixel value data.

As a method for creating a low-resolution image such as a thumbnail image, the generation unit 122 may perform image reduction using a classical resampling method (resolution conversion method) such as Bilinear or Bicubic on the acquired coordinates.

When processing from pixel value data in multiple frames, the generation unit 122 may take into account object movement between frames, and rather than treating the frames equally, may employ techniques such as processing a certain frame as the main information, selecting frames for each local region, or reducing the effect of small amounts of movement by increasing the reduction ratio.

As another reduction method, the generation unit 122 may utilize machine learning (learning model). In this case, the generation unit 122 may generate a low-resolution image using a learning model that is machine-learned using the pixel value data and coordinates of the RTP packet, or the output image by the above-mentioned resampling method as input.

The generation unit 122 may also reconstruct an incomplete image in which only known pixels of the acquired pixel value data are arranged based on coordinates, and solve a regression problem to generate a low-resolution image from the incomplete image.

In addition, the generation unit 122 may be notified in advance of secondary information such as coordinates of important parts that represent the outline of the image, and the generation unit 122 may then employ trimming or seam carving that includes this information to reduce the image size.

In addition, the generation unit 122 may switch the generation method based on the required resolution of the display unit 13, the amount of streams to be processed in parallel, etc., rather than using a single method. For example, the generation unit 122 may use a resampling method when displaying multiple streams side by side, and may utilize machine learning when a request is input to check the overview of one of the streams.

The display unit 13 displays low-resolution images such as thumbnail images. As a result, the low-resolution images generated by the generation unit 122 are displayed on the display unit 13 of the management device 1, or on an external display device via the network. Thus, the video stream on the network is visualized and displayed in a form that can be seen by the user.

The display unit 13 may display information indicating what kind of video stream is flowing between which devices on the network by using a low-resolution image. That is, the display unit 13 may display the low-resolution image as the video stream flowing between the network devices 4 and 5. For example, the display unit 13 may display a line representing the connection between the devices and a low-resolution image superimposed on the line.

In addition, since the video packet contains a source address and a destination address assigned to each port of network devices 4 and 5, by analyzing the video packet it is possible to determine which network device 4 the video stream is being sent from to which network device 5.

In the display unit 13, a simple display such as a block diagram using a GUI is assumed. The lines representing the connections between the devices may be displayed by acquiring the route in advance by confirming that the management device 1 or the relay device 2 can actually communicate using ping or the like. In addition, the lines representing the connections between the devices may be displayed by acquiring the route by the management device 1 acquiring information from NMOS (which manages information about terminals on the network), which will be described later, when actually transmitting. In addition, the generation unit 122 may acquire the source address and destination address included in the video packet to acquire the lines representing the connections between the devices.

In addition, when a user views a fiber (transmission path) through which a video stream flows on a display device outside the management device 1, such as a head-mounted display, a thumbnail image or the like may be displayed superimposed on the fiber. Specifically, when a user views or gazes at a real-world fiber that is actually connected through a transparent display such as AR glasses, a thumbnail image or the like of the video stream flowing through the fiber is displayed superimposed on the fiber.

Figure 2 is a flowchart showing the transmission processing of this embodiment.

The relay device 2 receives video packets (RTP packets) in which an uncompressed video stream has been packetized from each network device 4, and copies the received video packets (S11). The relay device 2 also transmits the received video packets to the corresponding network device 5.

The relay device 2 transmits the copied video packets to the management device 1 via the narrowband path 3 (S12). The management device 1 receives the sampled video packets via the narrowband path 3 and generates a low-resolution image such as a thumbnail image using the video packets (S13). The management device 1 displays the low-resolution image (S14).

Second Embodiment
FIG. 3 is a system configuration diagram showing an example of the configuration of a transmission system according to the second embodiment.

The transmission system of this embodiment includes a management device 1A. The management device 1A acquires network device information and stream information from an NMOS (Networked Media Open Specifications) RDS (Registration & Discovery System) server 8, and acquires the multicast address of a desired multicast terminal 9 from the device information. The management device 1A then uses the multicast address to send a join to the multicast terminal 9, and participates in the multicast. This allows the management device 1A to acquire the same video stream (video packets) as the video stream flowing from the multicast terminal 9 to the network device 5.

The multicast terminal 9 shown in the figure includes a source terminal that transmits a video stream and a multicast network. The multicast terminal 9 and the receiving port 11 of the management device 1A are connected by a narrowband path 3.

The illustrated management device 1A has a receiving port 11, a processing unit 12, a display unit 13, another receiving port 14, and a transmitting port 15. The management device 1A differs from the management device 1 of the first embodiment shown in Figure 1 in that it has another receiving port 14 and a transmitting port 15. However, the receiving port 11 and the transmitting port 15 may be a single port that has both transmitting and receiving functions.

The other receiving port 14 obtains network device information (NMOS information) and stream information from the RDS server 8. The RDS server 8 is a server that detects and registers devices in the network. The other receiving port 14 obtains the multicast address of the multicast terminal 9 that distributes the desired stream from the device information and stream information, and sends the multicast address to the transmitting port 15.

The transmission port 15 sends a join to the multicast terminal 9 using the multicast address. As a result, the multicast terminal 9 sends the same video stream (video packets) as the video stream sent to the network devices 5, etc., to the management device 1A via the narrowband path 3. The narrowband path 3 in this embodiment is a path with a smaller transmission capacity than the transmission capacity of the video packets of the video stream distributed by the multicast terminal 9, similar to the narrowband path 3 in the first embodiment. As a result, the video packets output from the multicast terminal 9 are transmitted to the management device 1A while causing packet loss by passing through the narrowband path 3.

The receiving port 11 of the management device 1A receives the video packets sampled due to packet loss on the narrowband path 3, and outputs the received packets to the processing unit 12. The processing unit 12 generates a low-resolution image such as a thumbnail image from the sampled video packets, as in the first embodiment. The display unit 13 displays the generated low-resolution image.

Third Embodiment
FIG. 4 is a system configuration diagram showing an example of the configuration of a transmission system according to the third embodiment.

The transmission system of this embodiment includes a management device 1B. Management device 1B differs from management device 1A of the second embodiment in that management device 1B acquires video packets of multiple video streams from multicast network 7, but is otherwise similar to management device 1A.

Multiple source terminals 6 are connected to the multicast network 7. The source terminals 6 multicast (transmit) video packets of a video stream (multicast source) via the multicast network 7. The multicast network 7 and the receiving port 11 of the management device 1B are connected by a narrowband path 3.

As in the second embodiment, the other receiving port 14 obtains network device information (NMOS information) and stream information from the RDS server 8, obtains multiple multicast addresses of the desired video streams from this information, and sends the multicast addresses to the transmitting port 15.

The transmitting port 15 uses multiple multicast addresses to send a join to the multicast network 7. As a result, the multicast network 7 sends multiple video streams that are the same as the video streams (video packets) sent to the network devices 5, etc., to the management device 1B via the narrowband path 3.

In the embodiment, the narrowband path 3 is a path with a transmission capacity smaller than the total transmission capacity of the video packets of the multiple video streams that the multicast network 7 distributes to the management device 1. As a result, the video packets output from the multicast network 7 are transmitted to the management device 1B while causing packet loss by passing through the narrowband path 3.

The receiving port 11 of the management device 1B receives the video packets sampled due to packet loss on the narrowband path 3, and outputs the received packets to the processing unit 12. The processing unit 12 generates a low-resolution image for each video stream from the sampled video packets, as in the first embodiment. The display unit 13 displays the generated low-resolution image.

Effects of the embodiment
The transmission systems of the first to third embodiments described above include a duplicating unit 22 that copies a plurality of packets obtained by packetizing an uncompressed video stream, a transmitting unit 24 that transmits the copied packets via a narrowband path 3, and a generating unit 122 that generates a low-resolution image from packets sampled due to packet loss occurring on the narrowband path 3. The duplicating unit may copy packets by using multicast participation.

In this way, in this embodiment, packet sampling is performed through a simple environmental preparation of a narrowband path 3, and a low-resolution image showing an overview of the video stream is generated from the remaining packets after sampling.

In this way, in this embodiment, it is possible to grasp the stream flowing through the network while reducing the load on the network. Specifically, packet loss is intentionally caused by video packets passing through narrowband path 3. As a result, in this embodiment, the number of video packets transmitted is reduced without the need for special devices or functions, and a low-resolution image is generated from the reduced video packets and transmitted to management device 1 (maintenance base). Therefore, in a wide range of usage environments, it is possible to realize visualization of video streams in parallel and in real time.

In addition, by reducing the number of video packets due to packet loss on narrowband path 3 and generating low-resolution images from limited information, processing costs can be reduced, and multiple video streams can be visualized in parallel in real time.

In this way, this embodiment can suppress the increase in processing load caused by the expansion of the network size and the amount of video streams flowing through the network without the need for special equipment, and can realize visualization of video streams that can be widely used on networks in the future.

<Hardware Configuration>
The management devices 1, 1A, 1B and relay device 2 described above may be implemented, for example, by a general-purpose computer system as shown in FIG. 5. The illustrated computer system includes a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. The memory 902 and the storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of each device. For example, the management devices 1, 1A, 1B and relay device 2 are implemented by the CPU of the management device executing a program for the management device, and the CPU of the relay device executing a program for the relay device, respectively.

The management device and relay device may be implemented in one computer or in multiple computers. The management device and relay device may be virtual machines implemented in a computer. The programs for the management device and relay device may be stored on a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or may be distributed via a network.

Note that the present invention is not limited to the above embodiment, and various modifications are possible within the scope of the gist of the present invention. For example, a plurality of management devices 1 as shown in FIG. 1 may be provided, and the plurality of management devices 1 may be connected to a central management server via a network. This allows the central management server to grasp the video streams flowing through a large-scale network. The same applies to the management devices 1A and 1B in FIGS. 3 and 4.

1, 1A, 1B: Management device 11, 14: Receiving port 12: Processing unit 121: Packet storage unit 122: Generation unit 13: Display unit 15: Transmission port 2: Relay device 21: Receiving port 22: Duplicating unit 23, 24: Transmission port 4, 5: Network device 6: Source terminal 7: Multicast network 8: RDS server 9: Multicast terminal

Claims (7)

a duplication unit for duplication of a plurality of packets obtained by packetizing an uncompressed video stream;
a transmission unit that transmits the duplicated packet via a narrowband path;
A generation unit that generates a low-resolution image from packets sampled due to packet loss occurring in the narrowband path. The transmission system according to claim 1 , wherein the generating unit acquires pixel value data of the uncompressed video and position information of the pixel value data from the packets, and generates the low-resolution image. The transmission system according to claim 1 , further comprising a display unit that displays the low-resolution image as the stream flowing between network devices. The transmission system according to claim 1 , wherein the narrowband path is a path having a transmission capacity smaller than a transmission capacity of the uncompressed video stream. The duplication unit duplicates packets of a plurality of uncompressed video streams;
The transmission system according to claim 1 , wherein the generation unit generates the low resolution image for each of the streams. A transmission method performed by a transmission system, comprising:
replicating a plurality of packets obtained by packetizing an uncompressed video stream;
transmitting the duplicated packets over a narrowband path;
generating a low-resolution image from packets sampled due to packet losses occurring in the narrowband path. A transmission program that causes a computer to function as a transmission system according to any one of claims 1 to 5.

Images