NZ709336B2 - Crowd-sourced video rendering system - Google Patents

Abstract

Disclosed is a method for graphic rendering over multiple clients (82, 84). The method includes distributing rendering tasks to connected client nodes (82, 84) having capable graphics processing units by transmitting viewport state data objects and a unique spatial location to each of the clients (82, 84); performing path tracing at each of the clients (82, 84) from the starting point of their unique spatial locations, and transmitting their rendered output back to the server (22). The server (22) obtains the samples by determining whether a required number samples to be composited exceeds a number of participated clients (82, 84). The server (22) generates a composite rendered output from the individual rendered outputs received by the participating clients (82, 84), and then transmits the composite to all connected clients (82, 84) for display. Thus, as the number of client nodes (82, 84) increases, the scene is rendered more rapidly and at higher quality. 2, 84); performing path tracing at each of the clients (82, 84) from the starting point of their unique spatial locations, and transmitting their rendered output back to the server (22). The server (22) obtains the samples by determining whether a required number samples to be composited exceeds a number of participated clients (82, 84). The server (22) generates a composite rendered output from the individual rendered outputs received by the participating clients (82, 84), and then transmits the composite to all connected clients (82, 84) for display. Thus, as the number of client nodes (82, 84) increases, the scene is rendered more rapidly and at higher quality.

Description

CROWD-SOURCED VIDEO RENDERING SYSTEM TECHNICAL FIELD [l] The present disclosure generally relates to distributed, crowd-sourced graphics rendering of applications, and more specifically relates to a server utilizing the graphics processing units of one or more connected clients to perform diffuse light rendering, receiving the rendered scenes from the one or more connected clients, generating a composite scene from the received scenes, and transmitting the composite scene to all connected clients.

BACKGROUND A client-server architecture, in general, is a distributed computing architecture that partitions tasks or work loads between servers, which may be considered as "service providers", and clients, which may be considered as "service requesters" or "service consumers".

Often, the servers and the clients are connected via a computer network and various types of data may be transmitted between individual servers and individual clients bi-directionally over the computer network. Traditionally, servers usually have more resources and greater performance capabilities than the clients. A server may share its resources with one or more clients, such as performing certain tasks for the clients (i.e., providing services to the clients). Because a server typically has more resources than a client, the server may complete a task, especially a resource­ demanding task, much faster than the client is able to.

Data exchanged between a server and a client may be represented using any suitable data format and transmitted using any suitable communications protocol. For example, when an application is executed on a server for a client, the output of the application may be represented using a structured document, such as a HyperText Markup Language (HTML) document or an Extensible Markup Language (XML) document. The server may transmit the HTML or XML document, which includes the data that represent the output of the application, to the client over a HyperText Transfer Protocol (HTTP) connection between the server and the client. The client, upon receiving the HTML or XML document, may consume the document and render the output of the application locally using the HTML or XML document, such as in a web browser executed on the client.

SUMMARY The present disclosure generally relates to a crowd-sourced graphics rendering system that distributes rendering tasks among one or more connected clients that transmit rendered output back to a server, which composites the client output into a single rendered output for transmission and display to all connected clients.

In the context of server-side graphics rendering, a single server may include an array of graphics processing units (GPUs) to render graphics objects and stream the output to clients. However, rendering complex 3D environments is expensive in terms of processor cycles, and additional GPUs or servers must be added to adequately provide rendered frames or scenes to attached clients. Thus, traditional server-side graphics rendering encounters significant scalability issues. It may be prohibitively expensive for the provider to add servers or GPUs to meet the demands of an increasing number of clients; conversely, overinvestment in server or GPU infrastructure results in underutilization when demand falls below capacity.

Additionally, in recent years, the number of consumer-level devices having GPUs capable of quickly performing relatively complex 3D rendering has increased dramatically. For example, whereas the number of personal computers possessing GPUs capable of performing of a 3D scene in under a minute may have been limited to a handful of computers in path tracing the early 2000's, currently over forty-million personal computers possess GPUs with this capability. Therefore it is desirable to leverage these client-side GPUs to offload graphics rendering tasks from the server to the clients, in effect crowd-sourcing the rendering of a given frame or environment.

In one embodiment, a computer system hosts a plurality of application instances, each application instance corresponding to a remote client application. The computer system maintains a network connection to each of the remote client applications for which an application instance is hosted. In particular embodiments, the application instance is a software stub, as further discussed below. In particular embodiments, the computer system transmits render assets to the client applications for computers having sufficiently powerful GPUs to contribute to rendering. In particular embodiments, the computer system transmits render assets in the form of viewport state data objects (VSDOs) as further discussed below. In particular embodiments, the computer system also transmits or assigns a unique spatial position to each client participating in the crowd-sourced rendering process. In particular embodiments, the client devices utilize the received render assets and the umque viewpoint to render the frame or environment. In particular embodiments, the client devices perform path tracing to render the diffuse lighting of a scene or environment. After completing its assigned rendering task, each participating client device transmits its rendered output back to the computer system. In particular embodiments, the rendered output is a frame from a particular viewpoint. In particular embodiments, the rendered of the environment. In particular embodiments, the output is a lightmap for the diffuse lighting rendered output is a modified VSDO. Upon receiving the rendered outputs, the computer system generates a composite rendered output for transmission back to all connected clients, even those that did not participate in the crowd-sourced rendering process.

In particular embodiments, each connected client views the scene from the exact same viewpoint, and the computer system transmits a composite frame from the specific viewpoint to each connected client for display. In particular embodiments, each connected client views the scene from a different perspective, and the computer system transmits a composite render asset for the individual clients to use in their own rendering engine. In particular environments, the render asset is a light map or VSDO containing light map data representing the diffuse lighting of the environment. Thus, clients may render the scene from different viewpoints using the render assets while retaining high-quality radiosity lighting. In particular embodiments, the clients render the fast-changing specular lighting locally and in conjunction with the received render assets containing the crowd-sourced diffuse lighting for the environment. Thus, as the number of client nodes possessing capable GPUs increases, any particular scene may be rendered more quickly or at higher graphics quality.

In an embodiment, a method for generating a composite render is disclosed. The method comprises transmitting to one or more clients, by a processor, common render assets, a unique spatial position different from other transmitted spatial positions and one or more instructions to generate a render using the common render assets and its unique spatial position or using the common render assets and a respective unique spatial position that was transmitted to each of the one or more clients. In an embodiment, each unique spatial position is generated randomly.

In an embodiment, the instructions to generate a render comprise instructions to perform path tracing on the render assets from the unique spatial position wherein the render is a light map for a portion of or an entire environment containing the render assets. In an embodiment the render is a frame, and the one or more instructions further comprise a common viewpoint from which to render the frame. In an embodiment, the render assets are viewport state data objects (VSDOs) and the render is the set of rendering assets enhanced by diffuse lighting data calculated as a result of the path tracing.

The method further comprises receiving, by the processor, from each of the one or by the processor, a composite render from the received more clients, a render and generating, renders. The composite render thus generated is transmitted by the processor to the one or more clients. In an embodiment, generating, by the processor, the composite render comprises adding, by the processor, the received light maps to generate a composite light map of a portion of or the entire environment containing the render assets. In an embodiment, generating, by the processor, the composite render further comprises adding the received frames to generate a composite frame.

A method of receiving a composite render is disclosed. The method comprises receiving, by a processor from a server, render assets, a unique spatial position and one or more instructions to generate a render using the received render assets and the unique spatial position.

The method further comprises generating, by the processor, the render in accordance with the one or more instructions, transmitting, by the processor, the generated render to the server and receiving, by the processor from the server, a composite render. In an embodiment, the method further comprises rendering, by the processor, specular lighting from a server-independent viewpoint, combining, by the processor, the rendered specular lighting with the received composite render and outputting, by the processor, the combined rendered specular lighting and received composite render to a display.

In an embodiment, a system for generating a composite render is disclosed. The system comprises, a graphics processing unit, a processor and a storage medium for tangibly storing thereon program logic for execution by the processor. In an embodiment, the program logic comprises common render asset transmission logic, executed by the processor, to transmit common render assets to one or more clients, spatial position transmission logic, executed by the processor, to transmit, to each of the one or more clients, a unique spatial position different from other transmitted spatial positions and instruction transmission logic executed by the processor, to transmit, to each of the one or more clients, one or more instructions to generate a render using the common render assets and its unique spatial position. In an embodiment, the program logic further comprises receiving logic, executed by the processor, to receive, from each of the one or more clients, a render, composite render generating logic, executed by the processor, to generate a composite render from the received renders and composite render transmitting logic, executed by the processor, to transmit the composite render to the one or more clients.

In an embodiment, a system for receiving a composite render is disclosed. The system comprises, a graphics processing unit, a processor and a storage medium for tangibly storing thereon program logic for execution by the processor. In an embodiment, the program logic comprises receiving logic, executed by a processor, for receiving from a server, render assets, a unique spatial position and one or more instructions to generate a render using the received render assets and the unique spatial position. The programming logic further comprises render generating logic, executed by the processor, for generating the render in accordance with the one or more instructions, transmitting logic, executed by the processor, for transmitting the generated render to the server and composite render receiving logic, executed by the processor for receiving from the server, a composite render. In an embodiment, the system further comprises lighting rendering logic, executed by the processor, for rendering specular lighting from a server-independent viewpoint. The system also comprises combining logic, executed by the processor, for combining the rendered specular lighting with the received composite render and outputting logic, executed by the processor, for outputting the combined rendered specular lighting and received composite render to a display.

In an embodiment, the instruction transmission logic to generate a render further comprises instructions to perform path tracing on the render assets from the unique spatial position wherein the render is a light map for a portion of or an entire environment containing the render assets. In an embodiment, the composite render generating logic further comprises adding logic to add the received light maps to generate a composite light map of a portion of or the entire environment containing the render assets. In an embodiment, the composite render generating logic further comprises adding logic, executed by the processor, to add the received frames to generate a composite frame.

These and other embodiments whose features can be combined will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS FIGURE 1 illustrates an example client-server system implementing a crowd- sourced graphics rendering system.

FIGURE 2 illustrates a method of distributing the rendering of a frame from a fixed viewpoint to a plurality of clients.

FIGURE 3 illustrates example received client frames for a given scene.