AU773934B2 - Display manager - Google Patents

Description

S&FRef: 581280

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan Zhenya Alexander Yourlo, Cathryn Anne Chamley Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Display Manager ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PR2124 *9 [32] Application Date 18 Dec 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:nvcd Ue OC 20U1 2i Batch tAQ: 5815c -1-

DISPLAYMANAGER

Field of the Invention The present invention relates generally to video display systems and, in particular, to an efficient approach to the display management of video information derived from an application source operating from a computer network and intended for delivery to a viewer via the network.

Background Australian Patent Publication No. AU-A-53527/99 discloses a customisable user interface system, the salient components of which are illustrated in Fig. 1A. Fig. 1A shows a hardware architecture of an interface system 100 where a smart card 102 incorporating a memory arrangement is pre-programmed to facilitate user access to resources available via a computer network 105, such as the Internet. The smart card 102, ooooo °is provided with a number of icons 104 or the like that are typically each representative of a particular function or access. The smart card 102 is insertable into a smart card reader 106, the later being provided with electrical connectors 108 configured to couple to complementary connectors (not seen in Fig. 1 A) of the smart card 102 to enable a reading

C

of data stored in the memory arrangement thereof. The reader 106 is provided with a transparent touch panel 110 arranged so that when the smart card 102 is inserted into the 000003 reader 106 and electrical connection is made, each of the icons 104 are able to be viewed through the touch panel 110 whereby a user can depress the touch panel 110 at a location overlying a particular icon 104 and the reader 106 operates to associate a position output from the panel 110 with a mapping stored within the memory arrangement of the smartcard 102. The reader 106 outputs a signal 112 associated with a function or some other predetermined event related to the selected icon 104. Typically, the reader 106 is a hand-held device and communicates with a computing arrangement, generally formed within a so-called "set-top" box 114, that couples to a user output interface, in this 581280.doc -2example an audio-visual output device 116, such as a television set. The set-top box 114 operates to interpret the signals 112 received from the reader 106, which may be electrical, radio frequency, or infra-red, and according to a specific, possibly proprietary, protocol. The set-top box 114 converts those signals to a form suitable for communication via the network 105 to cause appropriate transmission to a functional destination, which may for example be a server computer 118. The server computer 118 performs the selected function, which in this case and according to the icons 104 of the particular card 102 illustrated, is the retrieval of on-line music, and provides data to the set-top box 114, which permits reproduction on the output device 116.

The system 100 is customisable by virtue of the user being able to utilize a number of different smart cards 102 to perform corresponding different operations. For example, whereas the illustrated smart card 102 is used to retrieve and cause reproduction ooooo of on-line music video by way of the television set, other functions may be performed such as electronic banking, home shopping, ordering home delivery fast food such as pizza, and the like. In each instance, insertion of an appropriate smart card 102 into the reader 106 causes a corresponding computer application to commence operation, either within the set-top box 114 or the server computer 118, in order to service user commands S" entered via the reader 106, and to return appropriate information for audio-visual •ooeo feedback to the user. For example, associated with each of the above noted functions would typically be one or more menu displays which, in concert with the reader 106, form a graphical user interface on the output device 116 by which the user can check selections being made (eg. pizza style to be ordered, toppings, payment methods) prior to actually confirming each or any function.

An example of this is illustrated in Figs. 1B to 1D where, having inserted the smart card 102 into the reader 106, the application 210 commences, for example on the server computer 118, and returns to the set-top box 114 for display on the output device a 581280.doc -3first menu screen 120 relating to the function to be performed, in this case a selection of Blues Guitar Masters. Using the reader interface device 106 and by selecting appropriate icons 104, the user can scroll through the various offerings to make a desired selection, in this case for an artist called Young Dead Guy. A further menu screen 122 is then 4 displayed as seen in Fig. 1C advising the user of the possible selections that may be made.

The user again scrolls, makes a selection and the application 210 then retrieves the music video which is streamed, perhaps from another location on the network 105, to the settop box 114 for appropriate output 124 as seen in Fig. 1D.

Problems arise with the above arrangement in the provision, and in particular display, of data from the operating application 210 to the set-top box 114. These problems include delayed response where complicated images are required to be i delivered, this sometimes being evidenced by incremental uploading of the display image.

,oooo Display delays can occur as a result of the application and/or network, particularly where a number of applications may be operating simultaneously. Further, where multiple applications operate, the precedence of any one application 210 on the display must also be asserted in an unambiguous fashion.

Summary of the Invention S" It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

According to a first aspect of the invention, there is provided a method of managing the provision of information obtained from a plurality of sources for display at an output, said method being performed at a location remote from at least said output, and comprising the steps of: receiving source data from each of said sources, said source data comprising at least components of individual screen displays of said information; 581280.doc -4buffering each of said received source data in a screen buffer corresponding to each said source as said source data is received; identifying one of said sources for output selection; and checking the screen buffer for said identified source and delivering the corresponding said source data for display at said output when each component of the corresponding screen display is wholly available in the corresponding said screen buffer.

In accordance with another aspect of the present disclosure there is provided a display manager comprising: input means for receiving, from a plurality of sources, display data comprising components of an individual screen display corresponding to each said source; buffer means for buffering in a corresponding buffer portion each of said screen displays by accumulating said corresponding components; enablement means for identifying when, for each said buffer portion, an entirety of said screen display has been accumulated in said buffer means; and multiplexing means, responsive to at least said enablement means, for selecting a one of said sources for output to a remote display destination when said enablement means identifies the corresponding buffer portion contains the entire corresponding o• S"screen display.

According to another aspect of the invention there is provided a computer program and a computer readable medium having recorded thereon a computer program for implementing any one of the methods described above.

Other aspects of the invention are also disclosed.

Brief Description of the Drawings One or more embodiments of the present invention will now be described with reference to the drawings, in which: 581280.doc Fig. 1A is a schematic representation of a prior art user customisable interface system; Figs. 1B to 1D show a number of examples of display output available from the system of Fig. 1A; Fig. 2 schematically illustrates an architecture within which the system of Fig. 1A may be operated; Fig. 3 is a simplified representation of the salient features of Fig. 2; Fig. 4 represents data flow within the system 200; Fig. 5 illustrates a macroblock format transmission; Fig. 6 shows an example of the formation of transition video; Fig. 7 shows further examples of transition frames; Figs. 8A to 8C depict a translation transition; Fig. 9 is a schematic block diagram of a general purpose computer upon which the arrangements described can be practiced; and Fig. 10 is a flowchart illustrating the general sequence of operations of the display manager of Fig. 2.

Detailed Description S"Traditional arrangements for delivering multimedia data over computer networks generally use some form of data compression or encoding to minimise transmission 20 bandwidth. Such is particularly the case with Internet communications where static •o••e images are often transmitted as JPEG images compressed using discrete cosine transforms or in other forms that use other transform methods and associated transform matrices.

Audio information can also be compressed. There exist some forms of compression that accommodate both audio and visual information and this includes the various MPEG standards. As a consequence, the arrangement of Figs. 1A to 1D may provide for at least the menus displays 120 and 122 to be delivered in a compressed form using any one of 581280.doc -6these known or similar standards. Typically, the video stream as seen in Fig. 1D may be delivered as pure video for example via a cable network, or alternatively in a compressed MPEG form, such a delivery method becoming popular over recent years.

Fig. 2 shows an arrangement operable within the architecture of Fig. 1A which provides for optimisations to be made in the manner in which display data may be communicated via the network 105 in order to provide visual outputs to the user using the output device 116.

As seen in Fig. 2, the set-top box 114 incorporates a card interface module 202 which interacts with the card reader 106 to interpret commands arising user operation of the card reader 106 and for forwarding those commands to an appropriate computer application, such as one of the applications 210 coupled to the network 105. In some arrangements, it may be possible for more than one of the applications 210 to operate ooooo simultaneously. A display manager 206 is provided to manage the various outputs from the operating applications 210 so that an application 210 of precedence, for example corresponding to the smart card 102 currently inserted in the card reader 106, has precedence for display on the output device 116. In this fashion, the display manager 206 operates with the knowledge of the various operating applications 210 to ensure that the application 210 currently being used by the user has precedence on the output device 116.

The display manager 206 communicates output information to the output device 116 via an output driver 204 formed within set-top box 114. The output driver 204 may include a display driver for driving a video or television display as well as an audio driver for driving an audio output device examples of both of which are found in traditional television receivers. Where data compression/encoding is used over the network 105, the set-top box 114 incorporates a complementary decompressor/decoder for retrieving output (display) data.

581280.doc -7- In a preferred implementation, each of the applications 210 generate display information and output that information in a compressed/encoded form, for example according to an MPEG standard.

As better seen in Fig. 3, the display manager 206 acts as a multiplexer of screen updates for a number of applications 210, that may be running on one or more separate server computers distributed across the network 105.

The display manager 206 uses a focus signal 220 to perform a multiplexing selection of a desired output from one of the applications 210. As is illustrated, the focus signal 220 may be derived 238 from a controlling one of the applications 210, used to launch and/or manage operation of the other (processing) applications 210. Alternatively, the focus signal 220 may be derived 242 from the set-top box 114, particularly where the settop box 114 incorporates significant computing capacity. The selected application 210 then has its output further encoded by the display manager 206, and forwarded to a user site. As also seen in Fig. 3, inputs from the various applications 210 operating for the time being are supplied to respective screen buffers 230 within the display manger 206 which in turn output to a multiplexer 236. A logic arrangement is provided having a module 232 to assess the state of the buffers 230 to determine when a screen image is fully load thereto. The focus signal 220 is examined to determine 240 which of the applications 210 has precedence for display on the output device 116. An AND module 234 then uses the an output of the module 232 and the determined application to switch the multiplexer 236 to the corresponding screen buffer 230, thereby permitting output of the contents thereof to the output device 116. Such output can thus occurs when a full screen image is available for display in the relevant buffer 230 and when the corresponding application 210 has focus, or precedence on the output.

Whilst Fig. 3 illustrates a schematic hardware-style configuration of the display manager 206, it will be appreciated that such functionality may be implemented in 581280.doc -8software operating within a general-purpose computing system. A flowchart representing a method 1000 performed by such software is shown in Fig. 10. The method 1000 operates continually whilst any one or more application 210 is functioning and at step 1002 acts to receive a display component (ie. part of a screen display) from an application 210. At step 1004, the display component is stored in a buffer 230 corresponding to the application 210 from which it was received, thereby accumulating the received display component with display components previously received from the same application 210. Step 1006 then checks the state of the buffer 230 at which storage just took place to determine if that buffer 230 contains of full screen display (ie. all components thereof). If not, control returns to step 1002 for receipt of the next display component from any one of the applications 210. If so, control passes to step 1008 which then determines if the application 210 corresponding to the full buffer 230 has focus, or precedence on the output. If not, control again returns to step 1002. If so, the entire content of the relevant buffer 230 is output. This step may involve certain encoding processes, such as those to be described. Control then returns to step 1002.

The output produced by the display manager 206 is preferably an MPEG-2 compliant system stream, which is comprised of a number of layers including an MPEG-2 Layer-3 audio stream, and a MPEG-2 video stream. A specific role of the display manager 206 is to produce constant frame-rate output that is delivered to the set-top box 114 on demand.

When an application 210 produces a screen update, the application 210 forwards a set of image macroblocks to the display manager 206 (with a yet-to-be-defined protocol) that represents the screen update in its entirety. The application 210 need not supply the completed screen update with any specific time period, in so much as the screen update should be supplied in a timely fashion to ensure an acceptable end-user experience.

When an application 210 requires audio output in addition to display updates, the application 210 sends packets taken from a pre-encoded MPEG-2 Layer-3 audio 581280.doc -9stream 404 to the display manager 206 as also seen in Fig. 4. The display manager 206 inserts the audio packets 404 into a system stream 406 at the designated locations before forwarding the system stream 406 to the receiving set-top box 114.

The display manager 206 also may internally contain memory in which an MPEG-2 video stream is stored. Such a stream may be inserted into the output stream 406 in instances where a change of application 210 is occurring, and the new application 210 has not yet provided any screen output, or if a standard system message needs to be displayed for a certain time period.

Transitions between screens may be synthesized by the display manager 206 by using macroblock manipulation techniques, and may be generated on receipt of a complete update from an application 210, or may make use of the update order of macroblocks from an application 210 to perform a transition whilst the macroblock updates are being received.

•Protocol between Display Manager and Applications The manner in which the display manager 206 interacts with various applications may be governed by a protocol such as that described below as a series of interaction steps.

o(i) Applications register with a display manager, specifying a name and a desired .ooo.i output format. The display manager 206 acknowledges the registration, which amounts 0 00 20 to a request to deliver services, and specifies the output format (including resolution) that 00**0* 0 0 the application 210 will be required to produce, as well as an ID (identification) code for that application. The ID code is used to identify the application 210 to the display manager 206 when the application 210 makes a request.

(ii) When an update occurs at the application level, the application 210 encodes the changes, and notifies the display manager, transmitting the updated and encoded frame to the display manager.

581280.doc (iii) The display manager 206 then incorporates the frame into the output stream.

(iv) When the application 210 wishes to incorporate audio into its output, the application 210 signals the display manager 206 with this intention, including the application 210 ID and the audio source in the audio request. The audio source may be a reference either to a file or to a bitstream.

The display manager 206 is responsible for incorporating the bitstream into a transport stream. The display manager 206 returns an ID for this audio resource to the application.

(vi) When the application 210 wishes to cease an audio stream, the application 210 signals this intention to the display manager, including the audio stream ID in this request.

When an application 210 no longer wishes to produce video, it signals this oi S• intention to the display manager.

(viii) The display manager 206 then ceases to produce output from the input source.

Any audio streams related to this application 210 are also ceased being output. Any further request from this application 210 will be ignored, excluding any further registration requests. Any memory internal to the display manager 206 that is associated with this application 210 may also be freed. The display manager 206 may switch to oogo• another source for output.

Form of transmissions between Display Manager and Applications Applications must provide input to the display manager 206 in an agreed format.

Two formats that may be used are a macroblock format and a frame format, these being discussed below.

Frame format.

In this format, an application 210 sends its visual output to the display manager 206 as a frame. This or each frame will contain the entire pixel buffer for the application, 581280.doc S -11even though only a small amount of the buffer may have changed. To improve efficiency, an application 210 may also transmit to the display manager 206 a region which describes a series of rectangles which enclose the areas of the frame that have changed since the last frame was updated.

The display manager, upon receiving these updates, may decide how best to encode the changes. The display manager 206 may decide not to encode the changes at all. This may be appropriate for example if the output device is no longer active, or the active output stream has been switched, or the output device no longer corresponds to the application.

The display manager 206 may also make adjustments to the encoding process depending on the attributes of the target device. For example, if the target device is very small, and therefore likely to downsample the output, the display manager 206 may •choose a lower quality encoding method whose output could be more highly compressed.

another example, the display manager 206 may be aware that part of the display is S*o obscured (for example as experienced with a picture-in-picture on a television, or a window on top of the application window on a personal computing device). In this case, 5555 the display manager 206 may choose to encode that area in a graphically incorrect, but computationally efficient, manner (eg: all one colour).

555555 By transmitting frames instead of macroblocks, the display manager 206 has more 20 freedom in the encoding process. However, since the encoding stage is moved further away from the application level, there is less object level information available to the encoder. This may lead to sub-optimal performance of the encoder, since some of the graphical object level encoding optimisations discussed above will not be available to the encoder. However, the processing time saved by the device level optimisations described above may compensate for this, under certain circumstances. When both methods are 581280.doc -12available, the system architect can choose the most appropriate method for a given architecture.

(ii) Macroblock Format.

An alternative to the frame transmission method is the macroblock transmission method. Using this method, applications forward each update to the display manager 206 as a series ofmacroblocks. These macroblocks may be organised as an MPEG compliant bitstream, or as some other agreed data format.

A number of applications may provide input at any given time. Therefore, it is the role of the display manager 206 to multiplex the output of all of the applications to produce one output video, which can be transmitted to the client. In many cases, the final output will be exactly the same as the output from one of the input applications. For example, if the output device 116 is only showing the most recent application, then the °ooeo S°output stream from the display manager 206 will consist entirely of the output from the °application 210 which has received user input most recently. Alternately, the output from the display manager 206 may be a picture-in-picture arrangement, where one application "5 210 occupies most of the screen, and a second application 210 is displayed in a small SbS portion of the screen. This same technology also allows applications to use the display .i manager 206 to composite live video into a static frame, from two different sources of ooooo encoded video.

20 Considering the first case, where only one application 210 is providing input for the screen, the display manager 206 has a relatively simple task. The display manager 206 simply includes the new video stream into the stream that is currently being sent to the client. However, in the second case, where one application 210 is shown as a picture-inpicture with another application, the task is more complicated. One way of performing this task is to completely decode the (two) bitstreams 402 sent to the display manager 206, reconstruct the original pixel-based frames, composite the two frames using 581280.doc -13conventional techniques, and re-encode the result. However, decoding and re-encoding the frames is typically computationally expensive.

A second, less computationally expensive method, is to only partially decode the frames, to the macroblock stage. In order to insert some video into a frame, the background video is copied, macroblock by macroblock, into the new bitstream 406, until the position of the inserted video, at which point the macroblocks of the second video would be copied to the outgoing bitstream 406. Of course, the DC values of the border macroblock require adjustment to select the previous block, thereby taking into account the value of the macroblock which was the originally the block preceding the current block, such that: x=m DCn DCA(n-l) DCBx x=n o: where DCn the DC component of the current macroblock in the composited image; S S DCA the DC component of the specified macroblock in the leftmost video; *Se DCB the DC component of the specified macroblock in the second video (ie.

15 to the right of DCA in the composited image); m the index of the first macroblock in the slice of the second image of which the current macroblock is a part; and Sn the index of the current macroblock.

This situation is illustrated in Fig. 20 Efficient Transitions between Applications In addition to simple picture-in-picture type compositing, it is possible to provide many efficient transitions based on macroblocks. If there exist two encoded video streams (or a series of macroblocks describing such video streams), there may be circumstances where it is desired to change or switch between these two streams. One example of this situation lies in a remote application environment, such as that illustrates 581280.doc -14in Figs. 1A-1D. In such a situation, the user will, at some stage, cease interacting with a given application, and begin interacting with a second application. One option in this situation is ignore any output from the first application, and begin taking output from the second application. This produces a sudden switch from one application 210 to another.

Another option is to produce a transition to switch from the first application 210 to the second application. This can be efficiently performed at the macroblock level. By selectively compositing macroblocks from each of the two streams, it is possible to effectively produce transitions such as tile transitions and slide transitions.

Macroblock based transitions work on 16x16 pixel blocks from each image. These blocks are selected from each image to form each frame of the transition. For example, in a tiled transition, Fig. 6 shows an example of how two video streams may be blended to perform a tiled transition, using a checkerboard approach. A tiled transition need not follow a simple checkerboard pattern. Lines, or any arbitrary shape may be composed using these macroblocks. In the example of an arbitrary shape, the shape is mapped to ooooo macroblocks, and macroblocks deemed to be "within" the shape are taken from the first video source, while macroblocks deemed to be "outside" the shape would be taken from the second video source. The shape may also take the form of an animated shape. Two examples of alternative tiled patterns are represented in Fig. 7.

Another transition which can be efficiently produced at macroblock level is a translation transition (a slide transition). This may be performed by copying macroblocks to a predetermined position, or by making use of MPEG motion vectors. To produce a translation transition using motion vectors, the motion vectors are simply encoded into the final image. The display manager 206 need not encode macroblocks which are obscured either by the macroblocks of the second video stream, or by the limits of the display device.

581280.doc An example of such a translation is illustrated in Figs. 8A to 8C, where Fig. 8A shows two images for which a transition over time from image B to image A is desired.

Fig. 8B shows how image B is progressively wiped over image A over the course of a number of image frames. Fig. 8C illustrates how this is achieved using motion vectors acting upon macroblocks within a frame. The motion vectors are encoded for each of the macroblocks which move during the translation. The motion vectors need not be calculated using motion estimation, since the direction and quantum of motion that has occurred is known, this being supplied by the application 210 which generated the translation.

The transition may therefore be defined by an predetermined algorithm that implements a desired change between images, for example of a predetermined number of display frames. The algorithm may be a checkerboard, horizontally banded, vertically banded, a slide, or animated, to name just a few.

Buffering Arrangement between Application and Display Manager *eooo7 S 15 In order to ensure a constant rate of output to the client, the display manager 206 may make use of internal buffering of preceding output from an application. One such buffering arrangement is described below.

4 When a frame has been updated, and the display manager 206 receives this updated *e.

Sframe from an application 210 and a copy of this encoded frame can be stored by the display manager. When the display manager 206 is to produce another frame, it may choose to transmit a "no change" P-frame, or an I-frame containing the buffered frame.

This acts to limit the amount of data that must pass between the application 210 and the display manager 206, since the application 210 need only send a new frame when the underlying image has changed.

In addition, the application 210 may register a number of pre-encoded frames with the display manager. These may then be used as reference frames, to which differences 581280.doc -16can be added. For example, if a GUI of an application 210 consisted of a small number of background images, to which details were added as needed, the application 210 may choose to register the frames representing the images with the display manager 206 at some point in time, for example, at registration time. When the application 210 provides updates to the display manager 206, these take the form of a reference to the buffered frame, and a description of the differences between that frame and the updated frame.

These differences may be transmitted in the form of a P-frame. This allows the application 210 to further reduce the amount of data to be transferred when transmitting an updated frame to the display manager.

Inserting audio into the system stream If an application 210 wishes to include audio in the output stream, the display manager 206 must provide some facility to synchronise audio and video. Thus, the oooo display manager 206 must cause the audio and video to be multiplexed onto a transport stream 406, which will be used to transport the output of the display manager 206 to the ooooo client device (eg. 116). There are a number of ways that this may be achieved, which are outlined below.

go The simplest kind of transport stream is the program stream. This is used to transport a single video stream, and a single audio stream (for example, a single television signal).

•However, a program stream can only be used when the audio and video streams have been encoded using a common locked block, and are therefore already synchronised.

This form of transmission would be suitable when the application 210 consisted of a preencoded video and audio stream (for example, a pre-recorded video). Alternatively, the two streams may be artificially synchronised dynamically. In order to do this, the audio stream is partially decoded, so that the timing data therein may be located. The timing data is then be modified to match that of the video stream, such that both appear to have 581280.doc -17been encoded together. The audio stream is then be reconstructed, and encoded into a program stream.

An alternative method is to maintain the two streams as separate streams, then multiplex the video and audio streams into a single transport stream which is more complex than a single program stream. According to another method, a number of potential pre-encoded audio streams may be multiplexed onto a single transport stream, along with a dynamically generated video stream. This allows, for example, different language versions of spoken audio to accompany the video. The appropriate language stream may then be selected from the available language streams by the client receiving the audio streams, according to the locality of the client, and the preferences of the user,.

O.

as indicated to the client.

.:000: 0 In order to synchronise the video and audio streams, the application 210 must signal to the display manager 206 that an audio stream should be transmitted with a given video stream. The application 210 must also signal when the audio should cease to be #oooo *o 15 transmitted.

The methods of encoding, decoding and communication described above with 0: respect to either or both of the applications 210 and display manager 206, are preferably practiced using a general-purpose computer system 900, such as that shown in Fig. 9 wherein the processes of Figs. 2 to 4 may be implemented as software, such as one or more application programs executing within the computer system 900. In particular, the steps encoding and decoding as appropriate are effected by instructions in the software that are carried out by the computer. The software may be stored in a computer readable medium, including the storage devices described below, for example. The software may be loaded into the computer from the computer readable medium, and then executed by the computer. A computer readable medium having such software or computer program 581280.doc -18recorded on it is a computer program product. The use of the computer program product in the computer preferably effects an advantageous apparatus for encoding and decoding.

The computer system 900 comprises a computer module 901, input devices such as a keyboard 902 and mouse 903, output devices including a printer 915 and a display device 914. A Modulator-Demodulator (Modem) transceiver device 916 is used by the computer module 901 for communicating to and from a communications network 920, for example connectable via a telephone line 921 or other functional medium. The modem 916 can be used to obtain access to the Intemrnet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 901 typically includes at least one processor unit 905, a memory unit 906, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface 907, and an I/O interface 913 for the keyboard 902 and mouse 903 and optionally a joystick (not illustrated), and an interface 908 for the modem 916. A storage eo i o o 15 device 909 is provided and typically includes a hard disk drive 910 and a floppy disk drive 911. A magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 912 is typically provided as a non-volatile source of data. The components 905 to 913 of the computer module 901, typically communicate via an interconnected bus 904 °and in a manner which results in a conventional mode of operation of the computer system 900 known to those in the relevant art. Examples of computers on which the described arrangements can be practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

Typically, the application program is resident on the hard disk drive 910 and read and controlled in its execution by the processor 905. Intermediate storage of the program and any data fetched from the network 920 may be accomplished using the semiconductor memory 906, possibly in concert with the hard disk drive 910. In some 581280.doc -19instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk and read via the corresponding drive 912 or 911, or alternatively may be read by the user from the network 920 via the modem device 916. Still further, the software can also be loaded into the computer system 900 from other computer readable medium including magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the computer module 901 and another device, a computer readable card such as a PCMCIA card, and the Internet and Intranets including e-mail transmissions and information recorded on Websites and the like. The foregoing is merely exemplary of relevant computer readable media. Other computer readable media 10 may alternately be used.

0 The various applications 210 and the display manager 206 as discussed above 0* may operate within one or more server computers 118 remote from the set-top box 114 and output device 116 and which may include many of the traditional features found in devices such as the computer system 900 shown in Fig. 9. Further, the set-top box 114 may include much of the arrangement of the computer module 901 of Fig. 9 noting that in such arrangements, typically a floppy disk drive, hard disk drive, or CD ROM drive o• would not need be required. In particular, where the computer system 900 of Fig. 9 is configured to operate that the applications 210, various encoding steps may be performed •within the computer module 901. Similarly, where such arrangement is formed within the set-top box 114, corresponding decoding arrangements may be performed thereby.

Where the arrangement 900 operates as the display manager 206 buffering of the streams output from the different applications 210 may occur within the memory 906, possibly in concert with the hard disk drive 910. "Pre-recorded" streams as described above may be stored within the hard disk drive 910 or the CD-ROM 912.

Encoding and decoding processes for any application 210 may be performed by software operating from the processor 905. The encoding and corresponding decoding 581280.doc methods described may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions described above.

Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories and operate in concert with software components.

The arrangements described provide significant advantages for display management. Specifically, by removing the mixing and selection of separate display sources away from the display location (ie. the set-top box 114 or display device 116) such enable devices at the display location to be of traditional (simple) configuration and thereby able to operate efficiently and quickly to changing display requirements. Such also makes the overall architecture more robust to upgrade and change since any alterations are made remote from the user and thus can be user/destination transparent.

Such arrangements are to be distinguished from traditional display management arrangements where such operations are performed at the display location, often using o i S 15 special purpose hardware. Such is the case with picture-in-picture in traditional television receivers where usually part of the RF receiver is duplicated to enable two broadcast channels to be received simultaneously and decoded. The two signals are then mixed locally to form the picture-in-picture display as seen by the user.

Industrial Applicability It is apparent from the above that the arrangements described are applicable to the computer and data processing industries and particularly where data from a variety of data sources is being encoded or compressed.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

581280.doc -21 In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

4 4 4 4 .440 4* 4*4 c 0 4 1e 4 *O 4 44404e 581280.doc

Claims (4)

1. A method of managing the provision of information obtained from a plurality of sources for display at an output, said method being performed at a location remote from at least said output, and comprising the steps of: receiving source data from each of said sources, said source data comprising at least components of individual screen displays of said information; buffering each of said received source data in a screen buffer corresponding to each said source as said source data is received; identifying one of said sources for output selection; and checking the screen buffer for said identified source and delivering the corresponding said source data for display at said output when each component of the .:ooei corresponding screen display is wholly available in the corresponding said screen buffer.

2. A method according to claim 1 wherein said source data comprises encoded data relating to said corresponding screen display and said delivering comprises forming a data stream of said encoded data for real-time display via said output.

3. A method according to claim 1 or 2 wherein, having identified said one source for selection, said method comprises forming a transition between a current output screen display and the screen display of said identified source.

4. A method according to claim 4 wherein data representing said screen displays is block encoded and said transition is formed by generating a predetermined plurality of transition frames, each said transition frame comprising encoded blocks of said current output screen display and encoded blocks of said screen display of said identified source.

581280.doc -23- A method according to claim 4 wherein said transition is defined by an algorithm operable over said plurality of said frames. 6. A method according to claim 5 wherein said algorithm is responsive to transformation matrices associated with said encoded blocks to produce motion vectors. 7. A method according to any one of claims 1 to 3 wherein said source data comprises video data in a frame format and said delivering comprises encoding said video data into a block format. A method according to any one of the preceding claims wherein said delivering •••oo 0 further comprises combining audio data with said screen display data. 0.00 0 15 9. A method according to claim 8 wherein said combining comprises synchronising said audio and screen display data. o o o* 0 10. A method according to any one of the preceding claims wherein at least one of -:000: said sources is remote from said location at which said method is performed. 9 11. A display manager comprising: input means for receiving, from a plurality of sources, display data comprising components of an individual screen display corresponding to each said source; buffer means for buffering in a corresponding buffer portion each of said screen displays by accumulating said corresponding components 581280.doc -24- enablement means for identifying when, for each said buffer portion, an entirety of said screen display has been accumulated in said buffer means; and multiplexing means, responsive to at least said enablement means, for selecting a one of said sources for output to a remote display destination when said enablement means identifies the corresponding buffer portion contains the entire corresponding screen display. 12. A display manager according to claim 11 further comprising focus means for determining from at least one of said sources or said display destination which one of said sources has precedence for display at said display destination, said multiplexing means being responsive to both said enablement means and said focus means. 13. A display manager according to claim 12 wherein said multiplexing means outputs a screen display of one said source from the corresponding said buffer portion when said focus means determines said one source has precedence and said enablement means determines said corresponding buffer portion contains and entire screen display. 14. A display manager according to any one of claims 11 to 13 wherein said display data comprises encoded data relating to said corresponding screen display, said display manager further comprising means for forming a data stream of said encoded data for real-time display at said display destination. A display manager according to any one of claims 11 to 14 further comprising means for forming a transition between a currently output screen display of one previously selected said source and a screen display of a currently selected said source. 581280.doc 16. A display manager according to claim 15 wherein data representing said screen displays is block encoded and said means for forming a transition generates a predetermined plurality of transition frames, each said transition frame comprising encoded blocks of said currently output screen display and encoded blocks of said screen display of said currently selected source. 17. A display manager according to claim 16 wherein said transition is defined by an algorithm operable over said plurality of said frames. 18. A method according to claim 17 wherein said algorithm is responsive to transformation matrices associated with said encoded blocks to form motion vectors. 19. A display manager according to any one of claims 11 to 18 wherein said source data comprises video data in a frame format and said display manager further comprises means for encoding said video data into a block format. .oeo°i A display manager according to any one of claims 11 to 19, further comprising S° means for combining audio data with said screen display data. 21. A display manager according to claim 20 wherein said means for combining synchronises said audio and said screen display data. 22. A display manager according to any one of claims 11 to 21 wherein at least one of said sources is remote from said location at which said display manager is configured. 581280.doc -26- 23. A computer program operable upon a server computer for managing the provision of information obtained from a plurality of sources for display at an output, said server computer being configured at a location remote from at least said output, said program comprising: code for receiving source data from each of said sources, said source data comprising at least components of individual screen displays of said information; code for buffering each of said received source data in a screen buffer corresponding to each said source as said source data is received; code for identifying one of said sources for output selection; and code for checking the screen buffer for said identified source and delivering the corresponding said source data for display at said output when each component of the corresponding screen display is wholly available in the corresponding said screen buffer. ooooi 24. A computer program according to claim 23 wherein said source data comprises encoded data relating to said corresponding screen display and said delivering comprises i forming a data stream of said encoded data for real-time display via said output. 25. A computer program according to claim 23 or 24 further comprising code for oooo° forming a transition between a current output screen display and the screen display of said identified source. 26. A computer program according to claim 25 wherein data representing said screen displays is block encoded and said transition is formed by generating a predetermined plurality of transition frames, each said transition frame comprising encoded blocks of said current output screen display and encoded blocks of said screen display of said identified source. 581280.doc -27- 27. A computer program according to claim 26 wherein said transition is defined by an algorithm operable over said plurality of said frames. 28. A computer program according to claim 27 wherein said algorithm is responsive to transformation matrices associated with said encoded blocks to form motion vectors. 29. A computer program according to any one of claims 23 to 28 wherein said source data comprises video data in a frame format and said delivering comprises encoding said video data into a block format. 30. A computer program according to any one of claims 23 to 29 further comprising code for combining audio data with said screen display data. V0006 90.0 31. A computer program according to claim 30 wherein said combining comprises :0 synchronising said audio and said screen display data. S- 32. A computer program according to any one of claims 23 to 31 wherein at least one of said sources is remote from said server computer. S" 33. A computer readable medium comprising a computer program according to any one of claims 23 to 32. 34. A method of managing the display of information obtained from a plurality of sources at an output, said method being substantially as described herein with reference to Figs. 2 to 10 of the drawings. 581280.doc -28- A display manager substantially as described herein with reference to any one of the embodiments as that embodiment is illustrated in the drawings. 36. A data stream comprising display information formed according the method of any one of claims 1 to 10 or 34. DATED this EIGHTEENTH Day of DECEMBER 2001 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant SPRUSON&FERGUSON S o*55S5 S 581280.doc