JP7739209B2 - Video providing device and video providing system - Google Patents

Description

One aspect of the present invention relates to a video providing device and a video providing system.

Patent Document 1 discloses a communications system that transmits video data from a distribution server to a terminal device. This communications system is designed to enable the flexible display of low-resolution and high-resolution video depending on the network bandwidth availability or the performance of the terminal. The distribution server acquires video data contained in content uploaded from a client device, and, based on that video data, generates video data for each of multiple layers ranging from low resolution to high resolution, to be transmitted to the terminal device.

JP 2014-11622 A

In the system described in Patent Document 1, high-resolution video data is generated for the entire video area, which places a heavy computational load on the distribution server. Furthermore, because high-resolution video data for the entire video area must be transmitted from the client device to the distribution server, a large amount of network bandwidth is required to transmit the video data to the distribution server.

Therefore, one aspect of the present invention aims to provide a video providing device and a video providing system that can reduce the network bandwidth required for video distribution and reduce the computational load on the video distribution device.

A video providing device according to one aspect of the present invention provides video data that serves as the basis for the low-resolution video data and high-resolution video data to a video distribution device configured to be able to distribute low-resolution video data with a first resolution for a first region and high-resolution video data with a second resolution higher than the first resolution for a second region that is part of the first region to a user terminal having a display unit. The video providing device includes a video acquisition unit that acquires the original video data with the second resolution for the first region, a video processing unit that generates first video data by reducing the resolution of a portion of the original video data that corresponds to a third region, which is the first region minus the second region, and generates second video data by extracting a portion of the original video data that corresponds to the second region, and a transmission unit that transmits the first video data and the second video data to the video distribution device as the video data that serves as the basis for the low-resolution video data and the high-resolution video data.

According to one aspect of the present invention, a video providing device transmits high-resolution (second resolution) original video data to a video distribution device only for a portion of the original video data corresponding to a second region to be distributed with high-resolution video data (second video data). Meanwhile, for a portion of the original video data corresponding to a third region not to be distributed with high-resolution video data, data (first video data) that has been pre-reduced in resolution by the video providing device is transmitted to the video distribution device. This configuration reduces the network bandwidth required to transmit video data from the video providing device to the video distribution device. Furthermore, if high-resolution data corresponding to the entire first region (e.g., the original video data itself) is transmitted to the video distribution device, the video distribution device must reduce the resolution of the entire first region of the video data, thereby increasing the computational load on the video distribution device. Meanwhile, the video providing device reduces the amount of data received by the video distribution device and reduces the region that needs to be reduced in resolution by the video distribution device, thereby reducing the computational load on the video distribution device. Therefore, the video providing device reduces the network bandwidth required for video distribution and the computational load on the video distribution device.

One aspect of the present invention provides a video providing device and a video providing system that can reduce the network bandwidth required for video distribution and the computational load on the video distribution device.

FIG. 1 is a diagram illustrating an example of a functional configuration of a video providing system according to an embodiment. 10A and 10B are diagrams illustrating an example of resolution reduction processing by a moving image processing unit. 10A and 10B are diagrams illustrating an example of a process of determining a second region by a moving image processing unit. 10A and 10B are diagrams illustrating an example of a process for generating low-resolution tiles and high-resolution tiles by a moving image generating unit. FIG. 10 is a diagram illustrating an example of a video distribution process from a video distribution device to an HMD. 10A and 10B are diagrams illustrating an example of a process for generating high-resolution tiles by a moving image generating unit; FIG. 10 is a sequence diagram illustrating an example of an operation of the video providing system. FIG. 2 is a diagram illustrating an example of a hardware configuration common to a video providing device and a video distribution device.

One embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Note that in the description of the drawings, identical or equivalent elements will be designated by the same reference numerals, and duplicate explanations will be omitted.

Figure 1 is a diagram showing an example of a video providing system 1 according to one embodiment. The video providing system 1 is a computer system that distributes video that can be viewed by users to user terminals.

As an example, the video providing system 1 includes a camera 10, a video providing device 20, a video distribution device 30, and an HMD (Head Mounted Display) 40, which is one or more user terminals used by one or more users. Note that while only one HMD 40 is shown in Figure 1, if there are multiple users to whom video is distributed by the video distribution device 30, the video providing system 1 will include multiple HMDs 40 used by each of the multiple users.

The camera 10 and video providing device 20 are communicatively connected via any wired or wireless communication network. Similarly, the video providing device 20 and video distribution device 30 are communicatively connected via any communication network. Furthermore, each of the video providing device 20 and video distribution device 30 is communicatively connected to the HMD 40 via any communication network.

Camera 10 is a device that captures the video to be distributed (original video). The video to be distributed may be, for example, video of an event such as a live performance, concert, play, or sports match. In this case, camera 10 is installed at the event venue (e.g., a concert hall, stadium, etc.). Camera 10 is configured to be able to capture high-resolution video of such events (e.g., video with a resolution of 8K or higher).

Camera 10 may be made up of multiple cameras. For example, camera 10 may be made up of multiple VR cameras for capturing 8K VR video (e.g., 360-degree 8K high-resolution video). Alternatively, camera 10 may be made up of multiple cameras installed to enable multi-angle shooting, or multiple cameras (e.g., multiple 4K cameras) configured to enable 8K wide-angle shooting.

The video distribution device 30 is configured to be able to distribute to the HMD 40 low-resolution tiles LT (low-resolution video data) (see Figure 4) with a first resolution (low resolution) in the first area A1, and high-resolution tiles HT (high-resolution video data) (see Figure 4) with a second resolution (high resolution) higher than the first resolution in the second area A2, which is part of the first area A1.

The first area A1 is the entire area that can be viewed by the user. As will be described in more detail below, as shown in FIG. 5, the user U can switch the display area 51 displayed on the display unit 41 of the HMD 40 within the first area A1 by changing the orientation of their head to change the orientation of the HMD 40 worn on the user U's head, or by setting the desired viewing angle by operating a controller or the like (not shown). The first area A1 is the entire area that can be viewed by switching the display area 51 in this way.

Also, as an example, the first resolution is 2K (1920 pixels x 1080 pixels), and the second resolution is 8K (7680 pixels x 4320 pixels). Note that the resolutions (first resolution and second resolution) here are based on the entire image corresponding to the first area A1. In other words, the low-resolution tile LT has the image quality when the entire image corresponding to the first area A1 is expressed in 2K. On the other hand, the high-resolution tile HT has the image quality when the entire image corresponding to the first area A1 is expressed in 8K.

The video distribution device 30 may be configured as a single computer device, or as multiple computers that can communicate with each other. For example, the video distribution device 30 may be configured as a server (video processing server) that executes the processing of the video generation unit 32 (described below), and a server (distribution server) that executes the processing of the distribution unit 34 (described below). The video distribution device 30 is deployed, for example, on the cloud.

The video providing device 20 is a device that provides the video data that forms the basis of the low-resolution tiles LT and high-resolution tiles HT to the video distribution device 30. For example, the video providing device 20 is installed at an event venue together with the camera 10, processes (image processes) the video data acquired from the camera 10, and transmits the processed video data to the video distribution device 30 located on the cloud. Like the video distribution device 30, the video providing device 20 may be configured as a single computer device, or may be configured as multiple computers that can communicate with each other.

The HMD 40 is an example of a user terminal used by a user. The HMD 40 is a device worn on the user's head. The form of the HMD 40 is not limited to a specific form. The HMD 40 can take various forms, such as goggles, glasses (eyeglasses), or a hat. In this embodiment, the HMD 40 is a VR device that has the function of providing the user with virtual reality (VR).

The HMD 40 has a display unit 41 and a sensor 42.

The display unit 41 is a display placed in front of both of the user's eyes. The display unit 41 is composed of, for example, a liquid crystal display element, an organic EL (Electro Luminescence) element, etc. The display unit 41 may have separate right-eye display panels placed in front of the user's right eye, and left-eye display panels placed in front of the user's left eye. The display unit 41 displays video (low-resolution tiles LT or high-resolution tiles HT) distributed from the video distribution device 30.

The sensor 42 includes, for example, a position sensor that detects the orientation and position of the HMD 40. The position sensor may be, for example, an acceleration sensor, a gyro sensor, a GPS sensor, or the like. The display area 51 (see FIG. 5) displayed on the display unit 41 changes depending on the orientation of the HMD 40 detected by the sensor 42. This mechanism provides the user with a rich VR experience. In other words, by turning their head sideways, the user can enjoy a VR experience in which they feel as if they are actually present at the event venue, with their field of view (display area 51) naturally changing as their head orientation changes. Display area information indicating the display area 51 identified by the orientation of the HMD 40 detected by the sensor 42 (i.e., information for identifying the area of the first area A1 that the user is looking at) is periodically transmitted to the video providing device 20 and the video distributing device 30.

The video providing device 20 has a video acquisition unit 21, a video processing unit 22, a transmission unit 23, and a display area information acquisition unit 24 (second display area information acquisition unit).

The video acquisition unit 21 acquires 8K (second resolution) original video data IM (see Figure 2) of the first area A1 from the camera 10.

Figure 2 is a schematic diagram showing an example of resolution reduction processing by the video processing unit 22. As shown in Figure 2, the video processing unit 22 generates first video data IM1 by reducing the resolution of a portion of the original video data IM corresponding to a third area A3, which is obtained by excluding the first area A1 and the second area A2. In the example of Figure 2, the second area A2 is set approximately in the center of the first area A1. Therefore, the third area A3 is formed in the shape of a square ring, with the central area (second area A2) of the first area A1 hollowed out. As an example, the video processing unit 22 reduces the resolution of the portion corresponding to the third area A3 to 1/4 of the original resolution. As a result, first video data IM1 of 2K (first resolution) is obtained.

The video processing unit 22 also generates second video data IM2 by extracting a portion of the original video data IM that corresponds to the second area A2. In the example of Figure 2, rectangular second video data IM2 that is smaller than the original size (the size of the first area A1) is obtained. In other words, the video processing unit 22 can obtain the above-mentioned first video data IM1 and second video data IM2 by lowering the resolution of only the portion of the original video data IM that corresponds to the third area A3.

The second area A2 may be any area smaller than the first area A1, and the position and size of the second area A2 may be set arbitrarily. The size of the second area A2 may be a preset fixed value, or may be a variable value that can be changed arbitrarily. The shape of the second area A2 is not limited to the rectangular shape of this embodiment. The second area A2 does not necessarily have to be a single continuous area, but may be composed of two or more separate areas.

The second area A2 may be set manually by an operator of the video providing device 20, or may be set autonomously (automatically) without human intervention. For example, if the video data acquired by the camera 10 is data from a concert, it is estimated that many users (i.e., users of HMDs 40 to which videos are distributed from the video distribution device 30) will prefer to view areas where the lead singer of the concert is visible. In this case, the operator may manually set the portion of the first area A1 that is estimated to be viewed by many users as the second area A2. Alternatively, the video processing unit 22 may use image recognition to extract a subject (the singer in the above example) that has been set in advance as an object likely to attract users' attention, and set the portion including the extracted subject as the second area A2.

Note that if the second area A2 is made smaller, the portion (third area A3) where resolution reduction processing is performed in the video processing unit 22 will become larger. Therefore, if the second area A2 is made too small, the processing time required for resolution reduction processing by the video processing unit 22 will increase, which may delay the transmission of video data from the video providing device 20 to the video distribution device 30, potentially impairing smooth video distribution to the user (HMD 40). Therefore, the video processing unit 22 may determine the size of the second area A2 so that the processing time required for resolution reduction processing falls within an acceptable range.

The video processing unit 22 may also set the second area A2 based on display area information acquired by the display area information acquisition unit 24, which will be described later. Details of this processing will be described after explaining the display area information acquisition unit 24. The frequency and timing of setting (updating) the second area A2 may be set arbitrarily. For example, once the second area A2 is set, it may be fixed for a certain period of time, or it may be updated as appropriate in response to changes in the situation.

The transmission unit 23 transmits the first video data IM1 and second video data IM2 generated by the video processing unit 22 to the video distribution device 30 as video data that will serve as the basis for the low-resolution tiles LT and high-resolution tiles HT.

The display area information acquisition unit 24 acquires display area information (second display area information) related to the display area 51 (see FIG. 5) of the first area A1 that is displayed on the display unit 41 of the HMD 40. The display area information acquisition unit 24 acquires display area information for one or more HMDs 40 that are the target of video distribution by the video distribution device 30.

In this embodiment, the display area information acquisition unit 24 acquires display area information indicating the display area 51 identified by the orientation of the HMD 40 acquired by the sensor 42 of the HMD 40 directly from the HMD 40. However, the route by which the display area information acquisition unit 24 acquires the display area information is not limited to the above. The display area information acquisition unit 24 may acquire the display area information via a device other than the HMD 40. Furthermore, the display area information may be information that can identify the area displayed on the display unit 41 of the HMD 40 (display area 51), and does not necessarily have to be information that directly indicates the display area 51. For example, if the video providing device 20 is configured to be able to calculate the display area 51 based on the orientation of the HMD 40, the display area information may be information that indicates the orientation of the HMD 40.

Next, we will explain an example of processing when the video processing unit 22 determines the second area A2 using display area information. For example, when the video distribution device 30 is distributing video to a single HMD 40, or when high-resolution video data is being distributed to a specific HMD 40, the video processing unit 22 may determine the second area A2 to include the display area 51 of that HMD 40.

On the other hand, if there are multiple HMDs 40 that are the target of video distribution by the video distribution device 30, the video processing unit 22 may set the second area A2 to include the display areas 51 of a predetermined number or more of HMDs 40 based on the display area information of each of the multiple HMDs 40 acquired by the display area information acquisition unit 24. Here, the "predetermined number" is a number that is arbitrarily set in advance, and may be, for example, a predetermined threshold value, or the maximum number of display areas 51 that can be covered (included) by a second area A2 of a predetermined size at a given point in time.

Figure 3 is a diagram showing a schematic example of the above-mentioned processing by the video processing unit 22 (processing of determining the second area A2 based on display area information). In the example of Figure 3, the display areas 51a-51g of multiple (seven in this example) users (HMDs 40) are acquired by the display area information acquisition unit 24. In this example, the size of the second area A2 (the size of the second area A2 relative to the first area A1) is assumed to be predetermined.

If the above "predetermined number" is set to the maximum number of display areas 51 that can be covered by the second area A2, the video processing unit 22 may set the second area A21 to include the largest number of display areas 51 (in this example, four display areas 51a to 51d).

On the other hand, in the example of Figure 3, the second area A2 may be composed of multiple separated areas, and if the above-mentioned "predetermined number" is set to a threshold value of "2," the video processing unit 22 may set the second area A22, which can include two adjacent display areas 51f, 51g, in addition to the second area A21. In this case, the second area A2 is composed of two separated areas (second areas A21, A22).

The above-described configuration for setting the second area A2 based on the display area information provides the following advantages. Even if the video distribution device 30 creates a high-resolution tile HT corresponding to the second area A2, if the area corresponding to the high-resolution tile HT is not being viewed on the HMD 40 (i.e., if the display area 51 does not include the area corresponding to the high-resolution tile HT), the process of creating the high-resolution tile HT is wasted. Furthermore, the process of transmitting the high-resolution second video data IM2 corresponding to the second area A2 from the video providing device 20 to the video distribution device 30 is itself wasted. On the other hand, the above-described configuration allows the second area A2 to be set to the portion that the user desires to view based on the display area information, thereby avoiding the above-described unnecessary process and enabling the high-resolution video data (high-resolution tile HT) of the area the user wants to view to be appropriately distributed to the user.

The video distribution device 30 has a receiving unit 31, a video generating unit 32, a display area information acquiring unit 33 (first display area information acquiring unit), and a distribution unit 34.

The receiving unit 31 receives the first video data IM1 and the second video data IM2 from the video providing device 20 (transmitting unit 23).

Figure 4 is a diagram illustrating an example of the process of generating low-resolution tiles LT and high-resolution tiles HT by the video generation unit 32. As shown in Figure 4, the video generation unit 32 generates low-resolution tiles LT for the entire first area A1 and high-resolution tiles HT corresponding to at least a portion of the second area A2 based on the first video data IM1 and the second video data IM2.

The video generation unit 32 generates low-resolution video data for the second area A2 (i.e., video data with the same resolution as the second video data IM2) by, for example, performing the same low-resolution processing as the video processing unit 22 on the second video data IM2. The video generation unit 32 can generate low-resolution tiles LT for the entire first area A1 by combining the video data generated in this manner (data for the portion corresponding to the second area A2) with the first video data IM1 (data for the portion corresponding to the third area A3). In this embodiment, the low-resolution tiles LT are video data with a resolution (2K) that is 1/4 the resolution (8K) of the original video data IM.

Here, the video generation unit 32 may use the second video data IM2 as the high-resolution tile as is, but if the size of the second video data IM2 (the size of the second area A2) is large, the amount of data transmitted when the high-resolution tile is transmitted to the HMD 40 by the distribution unit 34 (described below) may become large, which may constrict the network bandwidth between the video distribution device 30 and the HMD 40. Therefore, as shown in FIG. 4, the video generation unit 32 may generate multiple high-resolution tiles smaller than the size of the second video data IM2 (i.e., video data corresponding to an area smaller than the second area A2).

In the example of Figure 4, the video generation unit 32 generates four high-resolution tiles HT1 to HT4 by dividing the second video data IM2 into upper left, upper right, lower left, and lower right sections. However, the method of generating multiple high-resolution tiles HT from the second video data IM2 is not limited to the example of Figure 4. Multiple high-resolution tiles HT may have overlapping areas, or there may be portions of the second area A2 where no high-resolution tiles HT are generated. In other words, one or more high-resolution tiles HT do not necessarily have to be generated to cover the entire second area A2.

The position, number, and size of the high-resolution tiles HT may be set manually by an operator of the video distribution device 30, or may be set autonomously (automatically) without human intervention. For example, the video generation unit 32 may set the high-resolution tiles HT based on display area information acquired by the display area information acquisition unit 33, which will be described later. Details of this process will be described after explaining the display area information acquisition unit 33. The frequency and timing of setting (updating) the high-resolution tiles HT can be set arbitrarily. For example, the high-resolution tiles HT may be fixed for a certain period of time after being set, or may be updated as appropriate in response to changes in the situation.

Increasing the number of high-resolution tiles HT generated or the frequency with which the positions and sizes of the high-resolution tiles HT are updated will increase the processing load (processing time) on the video generation unit 32 accordingly. As a result, the transmission process of video data from the video distribution device 30 to the HMD 40 may be delayed, which may impair smooth video distribution to the user (HMD 40). Therefore, the video generation unit 32 may determine the update frequency and number of high-resolution tiles HT to be generated so that the processing time falls within an acceptable range.

The display area information acquisition unit 33 acquires display area information (first display area information) related to the display area 51 (see FIG. 5) of the first area A1 that is displayed on the display unit 41 of the HMD 40. The processing performed by the display area information acquisition unit 33 is similar to that performed by the display area information acquisition unit 24 of the video providing device 20. Furthermore, the display area information acquired by the display area information acquisition unit 33 is similar to the display area information acquired by the display area information acquisition unit 24.

When a high-resolution tile HT corresponding to the display area 51 indicated in the display area information acquired by the display area information acquisition unit 33 has been generated by the video generation unit 32, the distribution unit 34 distributes the high-resolution tile HT corresponding to the display area 51 to the HMD 40. For example, when a single high-resolution tile HT covering the entire display area 51 has already been generated by the video generation unit 32, the distribution unit 34 distributes the single high-resolution tile HT to the HMD 40. On the other hand, when a single high-resolution tile HT alone cannot cover the display area 51 and the entire display area 51 can be covered by combining multiple generated high-resolution tiles HT, the distribution unit 34 distributes the multiple high-resolution tiles HT to the HMD 40.

On the other hand, if the video generation unit 32 has not generated a high-resolution tile HT corresponding to the display area 51 indicated in the display area information acquired by the display area information acquisition unit 33, the distribution unit 34 distributes a low-resolution tile LT corresponding to the display area 51 to the HMD 40. It may be possible for one or more high-resolution tiles HT to cover only a portion of the display area 51. In such a case, the distribution unit 34 may distribute the one or more high-resolution tiles HT to the HMD 40, and also distribute a low-resolution tile LT corresponding to the display area 51 to the HMD 40. In that case, the display area 51 displayed on the display unit 41 of the HMD 40 will include a mixture of portions displayed in high resolution (i.e., portions where high-resolution tiles HT are present) and portions displayed in low resolution (i.e., portions where high-resolution tiles HT are not present and low-resolution tiles LT are displayed).

The distribution unit 34 may also distribute to the HMD 40 video data corresponding to the area of the first area A1 excluding the display area 51 (i.e., the non-display area 52 that is not displayed on the display unit 41 of the HMD 40 and is not viewed by the user). This is to prevent a situation in which the video data displayed on the display unit 41 is not present on the HMD 40 immediately after the orientation of the HMD 40 changes due to the user moving their head, changing the display area 51. For example, the distribution unit 34 may transmit video data (tiles) corresponding to the display area 51, and also always distribute to the HMD 40 a low-resolution tile LT corresponding to the entire first area A1. In other words, if a high-resolution tile HT corresponding to the display area 51 exists, the distribution unit 34 may distribute to the HMD 40 both the high-resolution tile HT and the low-resolution tile LT corresponding to the entire first area A1. On the other hand, if there is no high-resolution tile HT corresponding to the display area 51, the distribution unit 34 only needs to distribute to the HMD 40 the low-resolution tile LT corresponding to the entire first area A1.

Next, we will explain an example of processing when the video generation unit 32 determines the position or size of the high-resolution tile HT using display area information. For example, when the video distribution device 30 is distributing video to one HMD 40, or when distributing high-resolution video data to a specific HMD 40, the video generation unit 32 may generate the high-resolution tile HT so that it includes the display area 51 of that HMD 40.

On the other hand, if there are multiple HMDs 40 that are the targets of video distribution by the video distribution device 30, the video generation unit 32 may generate the high-resolution tile HT so as to include the display areas 51 of a predetermined number or more of HMDs 40, based on the display area information of each of the multiple HMDs 40 acquired by the display area information acquisition unit 33. Here, the "predetermined number" is a number that is arbitrarily set in advance, and may be, for example, a predetermined threshold value, or the maximum number of display areas 51 that can be covered (included) by a high-resolution tile HT of a predetermined size at a given point in time.

Figure 6 is a schematic diagram illustrating an example of the above-described process (process of generating high-resolution tiles HT based on display area information) performed by the video generation unit 32. In the example of Figure 6, high-resolution tiles HT1 to HT4 have been generated in advance. Consider a case where a predetermined number or more users' display areas 51 are concentrated in region R in the center of the second area A2. In this case, in order to deliver high-resolution video data corresponding to the display areas 51 to each user's HMD 40, all four high-resolution tiles HT1 to HT4 must be delivered. Transmitting multiple high-resolution tiles HT in this manner increases the amount of data communication between the video distribution device 30 and each HMD 40, potentially constraining network bandwidth. Furthermore, because each of the high-resolution tiles HT1 to HT4 only partially overlaps region R, a large amount of unnecessary data (i.e., the portions of each of the high-resolution tiles HT1 to HT4 outside region R) is transmitted. In such a case, the video generation unit 32 may generate a high-resolution tile HT5 to include region R. With this configuration, it is only necessary to transmit one high-resolution tile HT5 to each HMD 40, effectively reducing the amount of data communication between the video distribution device 30 and each HMD 40. Note that if there is only one HMD 40 to which video distribution is targeted, the same effect as above can be achieved by generating a high-resolution tile HT that includes the display area 51 of that HMD 40.

The video generation unit 32 may also determine the size of the high-resolution tile HT to be distributed to the HMD 40 based on the network bandwidth between the video distribution device 30 and the HMD 40. For example, when the network bandwidth margin (e.g., available network bandwidth) between the video distribution device 30 and the HMD 40 is equal to or greater than a predetermined threshold, the video generation unit 32 may generate a higher-resolution tile HT of a larger size than when the network bandwidth margin is less than the threshold. In the example of Figure 6, consider a case where the display area 51 of a certain HMD 40 coincides with area R. In this case, for example, when the network bandwidth margin between the video distribution device 30 and the HMD 40 is equal to or greater than a threshold, the video generation unit 32 may generate a high-resolution tile HT5 as the high-resolution tile HT to be distributed to the HMD 40. When the network bandwidth margin is less than the threshold, the video generation unit 32 may generate a high-resolution tile HT6 smaller than the high-resolution tile HT5 as the high-resolution tile HT to be distributed to the HMD 40. In this example, the high-resolution tile HT6 is smaller than the region R, so when the high-resolution tile HT6 is distributed to the HMD 40, low-resolution video data (low-resolution tile LT) is displayed on the edges of the display region 51. By changing the size of the high-resolution tile HT sent to the HMD 40 according to the network bandwidth as described above, it is possible to distribute high-quality (high-resolution) video data to the user (HMD 40) as long as the network bandwidth between the video distribution device 30 and the HMD 40 is not congested.

Next, an example of the operation of the video providing system 1 will be described with reference to Figure 7.

In step S1, the video acquisition unit acquires original video data IM (see Figure 2) from the camera 10.

In step S2, the display area information acquisition unit 24 acquires display area information (i.e., information for identifying the display area 51 of each HMD 40) of one or more HMDs 40 that are the target of video distribution by the video distribution device 30.

In step S3, the video processing unit 22 generates, from the original video data IM, first video data IM1 with reduced resolution corresponding to the third area A3, and second video data IM2 with high resolution (i.e., the same resolution as the original video data IM) corresponding to the second area A2. Here, the video processing unit 22 may perform a process of setting the second area A2 before the above process. Furthermore, the process of setting the second area A2 may be performed based on the display area information acquired by the display area information acquisition unit 24, as described above. Note that the frequency of the process of setting (updating) the second area A2 may differ from the frequency of the processes of generating the first video data IM1 and the second video data IM2. For example, the process of generating the first video data IM1 and the second video data IM2 is executed each time original video data IM is acquired from the camera 10, but in order to reduce the processing load on the video providing device 20 (video processing unit 22), the process of setting (updating) the second area A2 may be executed less frequently than the frequency of the process of generating the first video data IM1 and the second video data IM2.

In step S4, the transmission unit 23 transmits the first video data IM1 and second video data IM2 generated by the video processing unit 22 to the video distribution device 30.

In step S5, the receiving unit 31 receives the first video data IM1 and the second video data IM2 from the video providing device 20 (transmitting unit 23).

In step S6, the display area information acquisition unit 33 acquires display area information (i.e., information for identifying the display area 51 of each HMD 40) of one or more HMDs 40 that are the target of video distribution by the video distribution device 30.

In step S7, the video generation unit 32 generates low-resolution tiles LT for the entire first area A1 based on the first video data IM1 and the second video data IM2 (see Figure 4).

In step S8, the video generation unit 32 generates high-resolution tiles HT corresponding to at least a portion of the second area A2 based on the second video data IM2 (see FIG. 4). Here, the video generation unit 32 may determine the position, number, and size of the high-resolution tiles HT as described above.

In step S9, the distribution unit 34 distributes the video data (low-resolution tiles LT and high-resolution tiles HT) generated by the video generation unit 32 to each HMD 40 according to the display area 51 of each HMD 40. In this embodiment, the distribution unit 34 transmits at least the low-resolution tiles LT corresponding to the entire first area A1 to each HMD 40. Furthermore, if the display area 51 of a certain HMD 40 is included in the second area A2 and the high-resolution tiles HT corresponding to the display area 51 have already been generated by the video generation unit 32, the distribution unit 34 distributes the high-resolution tiles HT to that HMD 40.

In step S10, the HMD 40 receives video data from the video distribution device 30 and displays the video data on the display unit 41. As a result, the user wearing the HMD 40 can watch the video via the display unit 41.

According to the video providing system 1 and video providing device 20 described above, high-resolution data (second video data IM2) is transmitted to the video distributing device 30 for only the portion of the high-resolution (second resolution) original video data IM corresponding to the second area A2, which is the distribution target of the high-resolution tile HT. On the other hand, for the portion corresponding to the third area A3, which is not the distribution target of the high-resolution tile HT, data (first video data IM1) that has been pre-lowered in resolution by the video providing device 20 is transmitted to the video distributing device 30. This configuration reduces the network bandwidth required to transmit video data from the video providing device 20 to the video distributing device 30. Furthermore, if high-resolution video data (e.g., the original video data IM itself) corresponding to the entire first area A1 is transmitted to the video distributing device 30, the video distributing device 30 must lower the resolution of the entire first area A1 of the video data, thereby increasing the computational load on the video distributing device 30. On the other hand, the video providing device 20 can reduce the amount of data received by the video distribution device 30 and also reduce the area that needs to be converted to lower resolution in the video distribution device 30, thereby reducing the computational load on the video distribution device 30. Therefore, the video providing device 20 and video providing system 1 can reduce the network bandwidth required for video distribution and also reduce the computational load on the video distribution device 30.

In the above embodiment, an HMD worn on the user's head was given as an example of a user terminal, but the user terminal to which videos are distributed by the video distribution device 30 is not limited to an HMD. The user terminal only needs to have a display unit (display) capable of displaying video data distributed from the video distribution device 30, and may be, for example, a smartphone, tablet terminal, desktop PC, laptop PC, etc.

The video data displayed on the display unit 41 may also be configured to allow zooming in and out. For example, when a zoom-out operation is performed, the entire first area A1 may be displayed on the display unit 41. More specifically, the display area 51 displayed on the display unit 41 may coincide with the first area A1, and low-resolution tiles LT corresponding to the entire first area A1 may be displayed in the display area 51. On the other hand, when a zoom-in operation is performed, an area smaller than the first area A1 may be set as the display area 51, as in the above embodiment.

The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and/or software. Furthermore, there are no particular limitations on how each functional block is realized. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, via wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or multiple devices.

Functions include, but are not limited to, judging, determining, calculating, computing, processing, deriving, investigating, searching, verifying, receiving, transmitting, outputting, accessing, resolving, selecting, choosing, establishing, comparing, assuming, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.

For example, the video providing device 20 in one embodiment of the present disclosure may function as a computer that performs the video providing method of the present disclosure. Figure 8 is a diagram showing an example of a hardware configuration common to the video providing device 20 and video distribution device 30 in one embodiment of the present disclosure. The video providing device 20 and video distribution device 30 may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the term "apparatus" can be interpreted as a circuit, device, unit, etc. The hardware configuration of the video providing apparatus 20 and the video distributing apparatus 30 may be configured to include one or more of the devices shown in FIG. 8, or may be configured to exclude some of the devices. The HMD 40 may also have a hardware configuration similar to the hardware configuration shown in FIG. 8.

The functions of the video providing device 20 and the video distribution device 30 are realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communications via the communication device 1004, and control at least one of reading and writing data from and to the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) that includes an interface with peripheral devices, a control unit, an arithmetic unit, registers, etc.

The processor 1001 also reads programs (program code), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The program used is a program that causes a computer to execute at least some of the operations described in the above-described embodiments. For example, each functional unit of the video providing device 20 (e.g., the video processing unit 22, etc.) may be stored in the memory 1002 and implemented by a control program that runs on the processor 1001, and similar implementations may be used for other functional blocks. While the various processes described above have been described as being executed by a single processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The program may also be transmitted from a network via a telecommunications line.

Memory 1002 is a computer-readable recording medium and may be composed of, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory (primary storage device), etc. Memory 1002 can store executable programs (program code), software modules, etc. for implementing the video providing method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be composed of at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray® disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium that includes at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmission/reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, network controller, network card, or communication module.

The input device 1005 is an input device (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, speaker, LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

Furthermore, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

In addition, the video providing device 20 and the video distribution device 30 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by this hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

Although the present embodiment has been described in detail above, it will be clear to those skilled in the art that the present embodiment is not limited to the embodiments described herein. The present embodiment can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

The order of the procedures, sequences, flowcharts, etc. of each aspect/embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

The determination may be made based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, notification of specified information (e.g., notification that "X is true") is not limited to being done explicitly, but may also be done implicitly (e.g., not notifying the specified information).

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or Digital Subscriber Line (DSL)) and/or wireless technologies (such as infrared or microwave), these wired and/or wireless technologies are included within the definition of transmission media.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or other corresponding information.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the mathematical formulas and the like that use these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not intended to be limiting in any way.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the noun following these articles being plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." It should be noted that the term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

1...Video providing system, 20...Video providing device, 21...Video acquisition unit, 22...Video processing unit, 23...Transmission unit, 24...Display area information acquisition unit, 30...Video distribution device, 31...Reception unit, 32...Video generation unit, 33...Display area information acquisition unit, 34...Distribution unit, 40...HMD (user terminal), 41...Display unit, 42...Sensor, 51, 51a-51g...Display area, A1...First region, A2, A21, A22...Second region, A3...Third region, IM...Original video data, HT, HT1-HT5...High-resolution tiles (high-resolution video data), IM1...First video data, IM2...Second video data, LT...Low-resolution tiles (low-resolution video data).

Claims (9)

A video providing device configured to be able to distribute low-resolution video data of a first resolution in a first region and high-resolution video data of a second region that is a part of the first region and a second resolution higher than the first resolution to a user terminal having a display unit, the video providing device providing video data that is the source of the low-resolution video data and the high-resolution video data,
a video acquisition unit that acquires original video data of the first region at the second resolution;
a video processing unit that generates first video data by reducing the resolution of a portion of the original video data that corresponds to a third area obtained by excluding the second area from the first area, and generates second video data by extracting a portion of the original video data that corresponds to the second area;
a transmitting unit that transmits the first video data and the second video data to the video distribution device as video data that will become the low-resolution video data and the high-resolution video data;
A video providing device comprising: a display area information acquisition unit that acquires display area information regarding a display area of the first area that is displayed on the display unit of the user terminal;
the moving image processing unit determines the second area so as to include the display area.
The video providing device according to claim 1 . the display area information acquisition unit acquires the display area information of a plurality of user terminals to which the video distribution device is to distribute video;
the video processing unit determines the second area so as to include the display areas of a predetermined number or more of the user terminals.
The video providing device according to claim 2 . a video distribution device configured to be able to distribute low-resolution video data having a first resolution of a first region and high-resolution video data having a second resolution higher than the first resolution of a second region that is a part of the first region to a user terminal having a display unit;
a video providing device that provides the video distribution device with video data that is the basis of the low-resolution video data and the high-resolution video data,
The video providing device includes:
a video acquisition unit that acquires original video data of the first region at the second resolution;
a video processing unit that generates first video data by reducing the resolution of a portion of the original video data that corresponds to a third area obtained by excluding the second area from the first area, and generates second video data by extracting a portion of the original video data that corresponds to the second area;
a transmission unit that transmits the first video data and the second video data to the video distribution device as video data that are the basis of the low-resolution video data and the high-resolution video data,
The video distribution device includes:
a video generation unit that generates the low-resolution video data of the entire first area and generates the high-resolution video data corresponding to at least a part of the second area based on the first video data and the second video data;
a first display area information acquisition unit that acquires first display area information regarding a display area of the first area that is displayed on the display unit of the user terminal;
a distribution unit that distributes the high-resolution video data corresponding to the display area indicated in the first display area information to the user terminal when the high-resolution video data corresponding to the display area indicated in the first display area information has been generated by the video generation unit, and distributes the low-resolution video data corresponding to the display area indicated in the first display area information to the user terminal when the high-resolution video data corresponding to the display area indicated in the first display area information has not been generated by the video generation unit.
Video delivery system. the video providing device further includes a second display area information acquiring unit that acquires second display area information relating to a display area of the first area that is displayed on the display unit of the user terminal;
the moving image processing unit determines the second area so as to include the display area indicated in the second display area information.
The video providing system according to claim 4 . the second display area information acquisition unit acquires the second display area information of a plurality of user terminals to which the video distribution device is to distribute the video;
the video processing unit determines the second area so as to include the display areas indicated in the second display area information of a predetermined number or more of the plurality of user terminals.
The video providing system according to claim 5 . the video generation unit generates the high-resolution video data so as to include the display area indicated by the first display area information.
The video providing system according to any one of claims 4 to 6. the first display area information acquisition unit acquires the first display area information of a plurality of user terminals to which the video distribution device is to distribute the video;
the video generation unit generates the high-resolution video data so as to include the display areas indicated in the first display area information of a predetermined number or more of the user terminals.
The video providing system according to claim 7 . the video generation unit determines a size of the high-resolution video data to be distributed to the user terminal based on a network bandwidth between the video distribution device and the user terminal.
The video providing system according to any one of claims 4 to 8.