AU727553B2 - Improvements in Image Filtering - Google Patents

Description

Editorial Note Specification 88375/98 comprises of: Description (Pages 1-6) Appendix A (Pages 1-25) Claims (Pages 7-8) Abstract

VI

S F Ref: 436644

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Canon Kabushiki Kalsha 30-2, Shimomaruko 3-chome Ohta-ku Tokyo 146-%501

JAPAN

Canon inffrmatien Svstem: Researfe Austfraia Pty LtZI 1 TI~~ Ic~ltDriv I Actual Inventor(s): Address for Service: Invention Title: NUi l Kyue new Soutr nales 2113- AU STCTRALA Bernard John Glannetti Spruson Ferguson, Patent Attorneys Level 33 St Martins Tower, 31 Market Street Sydney, New South Wales, 2000, Australia Improvements in Image Filtering ASSOCIATED PROVISIONAL APPLICATION DETAILS [311 Application No(s) [331 Country P09655 AU [321 Application Date 8 October 1997 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815 -1- IMPROVEMENTS IN IMAGE FILTERING Field The present invention relates to digital image processing and, in particular, to the field of applying an unsharp mask filter to an image.

Background In recent years, computer output devices including display screens, colour printers and output devices capable of displaying a quality picture have become increasingly popular. Unfortunately, there is associated with outputting a quality picture a certain amount of image filtering or image processing. Image filtering may be used to enhance image detail following a deterioration of image data resulting from inputting an image using an input device and subsequently displaying the input image on an output device. Typically, blurring or defocusing of an image occurs when a picture (image) is scanned using an image scanner, or when a geometrically transformed image is constructed by means of interpolation between image picture elements.

A technique commonly adopted in photography for overcoming the problems with such defocusing or blurring of the scanned image is to sharpen the image using 20 an "unsharp mask" filter. The unsharp mask filter takes its name from traditional photographic darkroom techniques of enhancing the edges of a graphical object of an image by removing or subtracting an "unsharp" or low-pass filtered (smoothed) form of the image from the original image.

The general principle of unsharp mask filtering is to add to an original image 25 signal, representing an image, a constant amount of high-pass filtered signal (or "edge" sharpened signal) derived from the original image signal. The high-pass filtered signal is determined, for example, by removing from the original image signal a low-pass (unsharpened) filtered signal derived from the original image (CFP0641AU)(OPEN29)436644) [0:\CISPA\OPEN\OPEN29394754:MXL -2signal. Unfortunately, this traditional form of unsharp filtering is disadvantageous in that it results in an abrupt transition between the original signal and the high-pass filtered signal, especially where an abrupt edge is present in the original image.

Further, conventional unsharp filtering is disadvantageous in that it also enhances high-frequency noise contained in the' input image.

Summary In accordance with the first aspect of the present invention there is provided a method of filtering an input image to produce a filtered output image, said method including, for substantially each candidate pixel in the input image, the step of: if the gradient of the candidate pixel and neighbours of the candidate pixel is greater then a predetermined noise threshold, applying a sharpening function to said candidate pixel to produce a filtered output pixel; and otherwise utilising said candidate pixel as the filtered output image pixel.

Preferably, the sharpening function includes the steps of applying a sharpening filter to the candidate pixel to produce a corresponding sharpened pixel; S and compositing the corresponding sharpened pixel with the candidate pixel so as to produce the filtered output pixel. The compositing process preferably utilises a factor proportional to the image gradient at the candidate pixel.

**In a second aspect, the present invention provides an apparatus for filtering an input image to produce a filtered output image, said apparatus including processing means for processing substantially each candidate pixel in the input 25 image, said processing means comprising: means for applying a sharpening function to said candidate pixel to produce a filtered output pixel if the gradient of the candidate pixel and its neighbours is greater then a predetermined noise threshold; and (CFP0641 AU)(OPEN29)436644) [O:\CISRA\OPEN\OPEN29]394754:MXL means for utilising said candidate pixel as the filtered output image pixel if said gradient is less than or equal to said noise threshold.

Brief Description of Drawings Notwithstanding any other forms which may fall within the scope of the present invention, preferred forms of the invention are be described hereinafter, by way of example only, with reference to the accompanying drawings in which: Fig. 1 illustrates a form of providing an unsharp mask filtering effect; and Figs. 2 and 3 are flow diagrams illustrating a filtering process according to the preferred embodiment.

Detailed Description Appendix A contains the disclosure of Australian Patent Application No.

26164/97 filed on 20 June 1997. Appendix A discloses an improved form of image filtering that overcomes a substantial number of the disadvantages of conventional image "sharpening" techniques. The disclosed technique includes combining of filtered signals in a predefined manner to reduce or eliminate noise in the input signal. The technique also disclosed a soft transition providing a smooth change between an original image and a high pass filtered or sharpened image, the transition factor being derived from a predetermined function of the input image such as a gradient factor.

It would be advantageous to provide a quicker form of processing an image so as to apply an unsharp mask filtering effect.

Fig. 1 illustrates aspects of the unsharp marking process as set out in i 25 Appendix A. An original image 10 having pixels 20 located by coordinates in the usual manner is subjected to sharpening to form sharpened image 12 having suitable sharpened pixels 21. The sharpened image 12 is created utilising techniques such as the subtraction of a low pass filtered version from the original; A gradient (CFP0641AU)(OPEN29)436644) [O:\CISRA\OPEN\OPEN29]394754:MXL ~f.1 -4map 14 of the original image 10 is also produced. Additionally, in accordance with the requirements of a particular application, the original image can be subject to processing such as blurring via low pass filtering etc. as disclosed in Appendix A Importantly, the two images 10, 12 are composited together 15 utilising an alpha channel technique that relies on the alpha or opacity value being proportional to the gradient value 14 for a particular pixel. The compositing process is capable of implementation using a number of standard techniques. A general discussion of compositing is available in a number of standard texts. For a classical discussion, reference is made to the article "Compositing Digital Images" by Porter and Duff published in SIGGRAPH 84 at page 253 259. The final composited pixel is output 23 which forms part of the final image 16.

However, the arrangement of Fig. 1 requires substantial computational resources to execute. In particular, the sharpened image 12 is formed for each pixel.

The gradient 14 is also determined for each pixel in addition to the possible optional processing 13 of the original image. Further, the pixels of each image are composited together so as to form the final image. An implementation in accordance with Fig. 1 would therefore likely require significant processing time to process an image. This may result in a significant slowing down of the arrangement of Fig. 1 in processing an overall image. This is also affected by increases in pixel resolutions 20 and the processing of full colour rather than single colour images.

The unsharp mask operation is an attempt to apply a sharpening convolution to an image's edges whilst avoiding the enhancement of low level noise. The S"arrangement of Fig. 1 applies the unsharp mask operation to the entire image creating a separate new image 12. The sharpening convolution requires each pixel of 25 a new image to be compared to its neighbours. If there is a relatively large difference, the sharpened pixel is utilised. If there is relatively little difference, the corresponding pixel in the original image is used. In between values result in a (CFP0641 AU)(OPEN29)436644) [O:\CISRA\OPEN\0PEN29]394754:MXL blend of the original pixel and the sharpened pixel generally through some form of interpolation between the two values.

The calculation of the sharpened convolution for every pixel is less efficient as a substantial portion of these pixels may have a gradient less than a predetermined noise threshold and may never be utilised.

The improved image filtering technique according to a preferred embodiment of the present invention advantageously provides a quicker form of unsharp filtering of objects. The process according to the various embodiments of the invention determines, for each pixel in an image, if the difference of the pixel between each of its neighbours is significant, and only to apply a sharpening convolution if this is the case. Table 1 contains pseudo code detailing the process: Table 1 for (each input_pixel) if (gradient (input_pixel) noise_threshold) output_pixel sharpened(input_pixel) else output_pixel input_pixel Fig. 2 illustrates a basic flow chart of the unsharp masking process in accordance with the preferred embodiment. In step 30, for each input pixel, the gradient with respect to adjacent pixels is first determined. In decision block 31, a check is made to determine if this gradient is less than a predetermined noise threshold. If decision block 31 returns false no change to the input pixel is made. However, if decision block 31 returns true (yes) indicating the gradient is determined to be greater than the predetermined threshold, a new sharpened output pixel value is calculated in step 32.

(CFP0641A U)(OPEN29)436644) [O:\CISRA\OPEN\OPEN29]394754:MXL r 3 l -6- Fig. 3 illustrates step 32 of Fig. 2 in greater detail. Pixels adjacent the pixel of interest are examined in step 35 to determine a gradient. From the gradient, a sharpened pixel value is then calculated in step 36. In step 37, the sharpened pixel value is composited with the original pixel to form an output pixel which is then output.

The modifications provided by the preferred embodiment results in a substantial lessening of the computational requirements in the creation of a final output image. In this respect, it is no longer necessary to sharpen the whole image for combining with an original. As most images consist of substantial regions having flat intensity and colour values, substantially improved processing results.

The preferred embodiment can readily be implemented in an apparatus, whereby the method described is implemented in a software algorithm for image processing running on a computer system. Hardware acceleration can also be employed to improve the performance of the computer system as a whole when implementing the invention.

It will be appreciated by the person skilled in the art that numerous variations and/or modifications may be made to the present invention as shown in the specific 0. embodiment without departing from the spirit or scope of the invention as broadly described. The present embodiment is, therefore, to be considered in all respects to 0 20 be illustrative and not restrictive.

0.

••0 *00•0 (CFPo641AU)(OPEN29)436644) [O0 \CISRA\OPEN\OPEN291 394754: MXL Editorial Note Pages 7 8 are the Claims pages and appear after Appendix A.

Appendix A is Patent Application No. 26164/97 complete with its own set of claims on pages 19-25. These claims are not the claims for 88375/98.

APPENDIX A Australian Patent Applicaton No. 26164/97 Filed on 20 June 1997 *o *o ooe ooo -1- IMAGE FILTERING METHOD AND APPARATUS The present invention relates to image processing involving filtering of an image, and in particular, to a method and apparatus for enhancing an image.

BACKGROUND

In recent years, computer output devices including display screens, color printers and output devices capable of displaying a quality picture have become increasingly popular. Unfortunately, there is associated with outputting a quality picture a certain amount of image filtering or image processing. Image filtering may be used to enhance image detail following a deterioration of image data resulting from inputting an image using an input device and subsequently displaying the input image on an output device, for example. Typically, blurring or defocusing of an image occurs when a picture (image) is scanned using an image scanner, or when a geometrically transformed image is constructed by means of interpolation between image picture elements.

A technique commonly adopted in photography for overcoming the problems with such defocusing or blurring of the scanned image is to sharpen the image using an "unsharp mask" filter. The unsharp mask filter takes its name from traditional photographic darkroom techniques of enhancing the edges of a graphical object of an o 20 image by removing or subtracting an "unsharp" or low-pass filtered (smoothed) form of the image from the original image.

The general principle of unsharp mask filtering is to add to an original image signal, representing an image, a constant amount of high-pass filtered signal (or "edge" sharpened signal) derived from the original image signal. The high-pass filtered signal is determined, for example, by removing from the original image signal a low-pass (unsharpened) filtered signal derived from the original image signal. Unfortunately, this traditional form of unsharp filtering is disadvantageous in that it results in an abrupt transition between the original signal and the high-pass filtered signal, especially where an abrupt edge is present in the original image. Further, conventional unsharp filtering -2filtering is disadvantageous in that it also enhances high-frequency noise contained in the input image.

Thus, a need clearly exists for an improved image filtering technique that overcomes one or more of the disadvantages ofconventional image "sharpening" techniques. A need exists for an image filtering technique in which an image signal and one or more filtered signals, or two or more differently filtered signals are combined in a predefined manner to reduce or eliminate noise in the output signal. A need also exists for an image sharpening technique that has a soft transition providing a smooth change between an original image signal and a high-pass filtered or sharpened signal.

Preferably, the transition between the original image signal and the high-pass filtered signal can be determined by a predetermined function of the original image signal, rather than filtering by a constant amount. Further, it is desirable to have an image filtering method that can be easily implemented with image compositing.

SUMMARY

In accordance with a first aspect of the present invention, there is provided a method of filtering an image signal, the image signal representing an image, said ooeo method comprising the steps of: filtering the image signal to provide a filtered image signal representing a 20 filtered image; determining a mapping function from a predetermined continuous arbitrary function of said image signal; and combining the filtered image signal with the image signal in accordance with said mapping function to produce a final image signal representing a final image.

In accordance with a second aspect of the present invention there is provided a method of filtering an image signal, the image signal representing an image, said method comprising the steps of: applying a first filter to the image signal to provide a first filtered image signal; -3applying a second filter to the image signal to provide a second filtered image signal; generating a mapping function from a predetermined continuous arbitrary function of said image signal; and combining the first and second filtered image signals in accordance with said mapping function to produce a final image signal.

In accordance with a third aspect of the present invention, there is provided an apparatus for filtering an image signal representing an image, said apparatus comprising: input means for receiving said image signal; at least one filtering means, coupled to the input means, for filtering the image signal to provide a corresponding filtered signal; mapping means, coupled to the input means and capable of receiving the image signal, for providing a mapping function of the image signal in accordance with a predetermined continuous or arbitrary function; compositing means, coupled to the input means, the at least one filtering means and the mapping means, for combining said image signal with said each filtered signal in accordance the mapping function to provide a final imagesignal having a smoothened transition between said image signal and said each filtered signal.

S 20 In accordance with a fourth aspect of the invention, there is provided a computer program product having a computer readable medium having a computer program recorded thereon for filtering an image signal, the image signal representing an image, said computer program product comprising: means for filtering the image signal to provide a filtered image signal representing a filtered image; means for determining a mapping function from a predetermined continuous arbitrary function of said image signal; and means for combining the filtered image signal with the image signal in accordance with said mapping function to produce a final image signal representing a final image.

In accordance with a fifth aspect of the invention, there is provided a computer program product having a computer readable medium having a computer program recorded thereon for filtering an image signal, said method comprising the steps of: first filtering means for filtering the image signal to provide a first filtered image signal; second filtering means for filtering the image signal to provide a second filtered image signal; means for generating a mapping function from a predetermined continuous arbitrary function of said image signal; and means for combining the first and second filtered image signals in accordance with said mapping function to produce a final image signal BRIEF DESCRIPTION OF THE DRAWINGS The embodiments of the present invention are described hereinafter with reference to the accompanying drawings, in which: Figure 1 is a flow diagram illustrating a process of image filtering according to a first embodiment of the invention; Figure 2 shows a schematic diagram of the image-filtering process steps of Fig. 1; Figure 3 illustrates an apparatus according to the first embodiment; Figure 4 is a flow diagram illustrating a process of image filtering according to a second embodiment of the invention; Figure 5 illustrates an example of the intermediate images for each step according to the second embodiment of Figure 4; Fig 6 is a flow diagram illustrating a process of image filtering according to a third embodiment of the invention; Fig. 7 is a flow diagram illustrating a modification of the third embodiment of Fig. 6; Fig. 8 is a generalised overview of the image filtering techniques according to the embodiments of the invention; and Fig. 9 is a block diagram illustrating an exemplary computer platform for implementing the embodiment of the invention.

DETAILED DESCRIPTION Overview The embodiments of the present invention relate to methods, apparatuses and systems for image filtering in which an image signal and one or more filtered signals, or at least two differently filtered image signals are combined using a predetermined arbitrary function of the image signal. This can provide a smooth change between an original image signal and a filtered signal. The predetermined function of the original image signal is used to determine the transition between the original image signal and the filtered signal, and is preferably continuous or piece-wise continuous. If the **function is piece-wise continuous, it is characterised by the domain of the function having more than two corresponding range values.

Fig. 8 is a flow diagram providing a generalised overview of image enhancing °according to the embodiments of the invention. An image 90 is provided as input to the image filtering steps. The process comprises a (first) filtering step 96 for producing a filtered or sharpened image 102 from the input image 90. Preferably, the step 96 involves high-pass filtering the image 90. A further step 94 determines a mapping function 100 using the input image 90, where the mapping function is dependent on a predetermined function g of the image 90. The function is determined from the input -6image and is used to control the transition in pixel values between the original image signal and the filtered signal. The input image 90 may be provided directly to step 104 for interpolating or combining two image sources. Optionally, a second filtering step 92 (optional filter 2 in the flow diagram) may be applied to the input image 90 to produce a filtered original image 98 that can be provided to the interpolating step 104 instead. This may involve low-pass filtering the image As shown in Fig. 8, dashed arrows between the first filter 96 and step 94 and between the optional filter 92 and step 94 indicate that the output of the filters 92, 96 may optionally be provided as inputs to step 94 as well, dependent on the particular continuous or piece-wise continuous mapping function being practiced. This aspect of the invention is described in greater detail hereinafter with reference to the third and fourth embodiments.

In step 104, either the input image 90 or the filtered original image 98 is interpolated with the preferably sharpened image 102 using the mapping function 100 to provide an output image 106. The output image 106 is a filtered version of the input image, but is also one that has reduced noise between the input image 90, 98 and the filtered image 102. Specific embodiments of this generalised image enhancing technique are described hereinafter with respect to Figs. 1 to 7. Numerous specific details, such as ranges of gradient map values, RGB color space, an exemplary 20 computer architecture for implementing the technique, etc. are described in detail to provide a more thorough description. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In .ther instances, well-known features have not been described in detail so as not to unnecessarily obscure the present invention.

Process of First Embodiment Fig. 1 is a flow diagram illustrating the image filtering or enhancing process according to the first embodiment of the invention. An input image 10 (referred to below as the original image) is processed by a sharpen filter 11 to provide a sharpened image 12. The sharpen filter 11 accentuates sharp or abrupt changes in the image and enhances the detail of the image. However, sharpening an image tends to exaggerate features of the image that may be considered undesirable by a viewer. For example, unwanted noise in the input image 10 is also enhanced by the sharpen filter 11 so that the sharpened image 12 may include a "sharpening" of the unwanted noise.

Preferably, the original image 10 is also processed through a differential image filter 13 which produces a "gradient map" 14 of the original image 10. The gradient map 14 is produced by spatially differentiating the original image 10, and taking the absolute value magnitude) of the resultant values. Typically, the gradient map 14 is normalised to have values ranging between zero and one In a discrete pixel (or digital) representation of the original image 10, a point in the gradient map 14 having a value of one represents a maximum difference between two adjacent picture element values of the original image In an analog signal representation of an original image, a gradient map of the image represents a normalised direct differentiation of the spatial frequencies of the analog signal. Therefore, a value of one at any point in the gradient map represents a maximum gradient, while a value of zero represent substantially no gradient in the analog signal.

In the first embodiment, the original image 10 is directly input to an image compositing step 15. However, it will be understood by a person skilled in the art that an "identity filter" can also be applied to the original image 10 before providing the filter output to the compositing step 15. Mathematically, this is equivalent to not applying any filter at all to the original image 10, since the "identity filter" provides a resultant output signal that is identical to an input signal applied to the identity filter.

25 The image compositing step 15 takes as inputs the original image 10, the sharpened image 12 and the gradient map 14 of the original image In the first embodiment, the gradient map 14 is a function of the original image and is used to determine the opacity of each picture element of the original image -8and each corresponding picture element of the sharpened image 12, which are composited to produce the final image 16.

Referring now to Figure 2, and for the purpose of the examples described herein, the original image 10 and the sharpened image 12 are assumed to each comprise a plurality of picture elements. The location of these picture elements in each image are denoted by coordinates and The coordinates are measured from the upper left corner of each image 10, 12 with indicating the number of picture elements- to the right and indicating the number of picture elements down, respectively, in relation to the upper left corner. The equivalent in an analog signal of each image is a point in the signal corresponding to an x and y coordinate position measured to the right and down from an upper-left corner of the image. Each picture element of the original image 10 can.be denoted by I(x,y) as a picture element value 20 at location x,y of the image 10. Similarly, S(x,y) denotes a picture element value 21 at location x,y of the sharpened image 12.

The value of the gradient map 14 corresponding to the picture element value of the original image 10 is represented by a normalised function which can take on values between zero and one In the image compositing step 15, the value g(x,y) of gradient map 14 is used to determine the opacity of the corresponding picture elements 20,21 of the original image 10 and the sharpened image 12, respectively.

This is done in a manner so that, when the picture element values 20,21 of the original image 10 and sharpened image 12 are added, the result F(x,y) is a picture element value 23 for the final image 16 in accordance with value g(x,y) of the gradient map 14.

For example, if the gradient map 14 of the original image 10 has a value of one (i.e.

1) implying a sharp gradient at the location x,y of the image 10, the opacity of 25 the picture element 21 of sharpened image 12 is set to fully opaque, the opacity of the picture element 20 of original image 10 is set to transparent, and these picture element values 20,21 are added together to result in the picture element 23 of the final image 16. The resultant final image picture element 23 therefore takes on the value of the picture element 21 of the sharpened image 12.

In general, the gradient map 14 value takes on values between zero and one so the resultant final image 16 is a proportional combination of I(x,y) and S(x,y) in accordance with the value of An example of this proportional combination can be described by a mathematical relationship as follows: (1 I(x,y) g(x,y) (1) where F(x,y) is the picture element 23 of the final image 16. The final image 16 is often referred to as the "unsharp mask filtered" image.

In some color images, in particular those intended for computer graphics, each picture element of an image can be represented by a plurality of color components Red, Green, Blue and an opacity or alpha channel representing the opacity of each picture element. Therefore, each picture element can typically be represented by a color quadruple where "ac" is an opacity value between zero (totally transparent) and one (fully opaque). Often when compositing an image, it is advantageous to "premultiply" the opacity value with the color components (i.e.

aR, aG, a Assuming that each picture element I(x,y) of the original image and each picture element S(x,y) of the sharpened image 11 comprise a plurality of premultiplied color quadruples (aR, aG, a then Equation 1 applies to each premultiplied color component and ca-channel of I(x,y) and S(x,y) to result in corresponding premultiplied color components of F(x,y).

It will be apparent to those skilled in the art that an arbitrary function of the original image may be used as g(x,y) in Equation 1. Preferably, the arbitrary function is continuous or piece-wise continuous. For example, g(x,y) in Equation 1 may be gradient map values that are modified by a lookup table to achieve a desired effect like enhancing certain edges of an image to obtain non-linear effects, or to correct undesired 25 effects as a result of sharp unwanted gradients of the original image Pseudocode for implementing the process of Figs. 1 to 3 is set forth in detail in Table 1.

TABLE 1 Let x image.

Convolve image x with a sharpen filter, giving sharpened image s.

Take gradient of image x and store result in mapping function g.

Apply color map to g, such that opacity channel is set to average of color channels.

Output (s in g) plus (x out g).

The image x, sharpened images, mapping function g, and output correspond to the image 10, sharpened image 12, gradient map 14 and final image 16, respectively, shown in Fig. 1. This pseudocode may be implemented as software executing on a computer having a graphics accelerator, as described in further detail below.

Advantageously, the process can be carried out utilizing convolution and compositing operations carried out by the graphics accelerator. It will be apparent to one skilled in the art that the operators shown in the pseudocode expression of the output image in Table 1 are conventional compositing operators and binary operation ("plus").

*r 20 Apparatus of First Embodiment Referring now to Figure 3, an apparatus in accordance with the first embodiment is illustrated wherein a data bus 30 communicates data to a multi-line store 31. The multi-line store 31 stores a plurality of data representing at least a portion of an original image 10 received from the data bus 30. A sharpen filter32, a differential S 25 filter 33 and a compositing circuit 34 are all coupled to the multi-line store 31 and extract data information from the multi-line store 31. The outputs of the differential filter 33 and the sharpen filter 32 are each coupled to the compositing circuit 34. The in multi-line store 31 is preferred since the sharpen filter 32 and the differential filter 33 require information about at least one neighbouring picture element to process a current picture element. For example, the differential filter 33 requires at least two picture -11 element values to determine a gradient between the picture elements. Not illustrated in Figure 3, but included as part of the differential filter 33 is a normalising circuit that normalises the result output by the differential filter 33 to provide values between zero and one s Timing control lines 35,36 and 37 are provided to the sharpen filter 32, the differential filter 33 and the compositing circuit 34, respectively, to synchronise the corresponding processing so as to composite corresponding picture elements received by the compositing circuit 34 from the multi-line store 31, the differential filter 33 and the sharpen filter 32. The compositing circuit 34 outputs a composite picture element F(x,y) in accordance with Equation 1, and stores this information in random access memory (RAM) 38 and/or outputs this information to a display device.

The embodiments of the invention can preferably be practiced using a conventional general-purpose computer, including IBM-PC/ATs and compatibles, Macintosh computers and Sun SparcStations, for example. Fig. 9 illustrates an exemplary computer architecture on which the method and apparatus of the first embodiment shown in Figs. 1 to 3 are implemented assoftware executing on the computer. Likewise, the embodiments of the invention described hereinafter and shown in Figs. 4 to 7 can practiced with such a computer. In particular, the steps of the image enhancing technique are effected by instructions in the software that are carried out by the computer, and in particular are preferably carried out using a graphics accelerator, image processor, or the like. In the embodiment shown in Fig. 9, a raster image co-processor is used to implement the image enhancing technique.

Fig. 9 simply illustrates the computer 120 connected to a printer 146.

*However, it will be well understood to a person skilled in the art that such a computer 25 may also comprise a video display, input devices, and a number of internal peripheral devices generally represented by block 129. The printer 146 may be any of a. number of output devices including line printers, laser printers, plotters, and other reproduction devices.

-12- The computer 120 comprises a host central processing unit (CPU) 122, host random access memory (RAM) 123, a Peripheral Component Interconnect (PCI) bus 126, a bridge circuit 124, a graphics hardware accelerator 140, and one or more peripheral devices 129. In the drawing, physical memory regions are represented by s grey-filled blocks. The PCI bus 126 is connected to the host CPU 122, .the peripheral device 129 and the graphics accelerator 140 by respective PCI-bus interfaces 127, 120 and 141. The host CPU 122 and the host memory 123, which may be interconnected by a local bus, are connected to the PCI-bus interface 127 by bridging circuitry 124 well known to those skilled in the art. The peripheral device 129 includes another PCIbus device 121 and may be connected to local memory 122. The graphics hardware accelerator 140 comprises a raster image processor (RIP) 144 connected to the PCI-bus interface 141. In turn, the RIP 144 is connected to a generic external interface 147, local memory 143 and the printer 146 via a port 145. The generic external interface 127 permits connection to external devices.

The host CPU 122 sets up various commands for unsharp masking in the host memory 123. The various commands are directed to the RIP 144, which is capable of executing the commands. The commands are preferably a series of coded instructions i that instruct the RIP 144 of the graphics accelerator 140 to perform operations, .9 9including convolutions and compositing operations, on image data (pixels). The image data is supplied to the RIP 144 by the host CPU 122 via the PCI bus 126. Using the 9RIP 144, a convolution operation having a predetermined kernel can be applied to Simage data to produce a blurred image. Alternatively, the convolution operation could produce a sharpening of the image dependent upon the kernel used. Compositing operations are known to those skilled in the art and include operations such as those set 25 forth in the article "Compositing Digital Images" written by Thomas Porter and Tom Duff and published in SIGGRAPH 84 at pp. 253-259 and Computer Graphics, Vol. 18, No. 3, July 1984. The compositing operations provide a technique for combining the image data with the blurred image data and/or the sharpened image data produced by the convolution operation to achieve a desired result. The compositing operations are 13performed by tihe RIP. 144 hiaving received cornpositing instructions froi tie ]lost CPU 102 via the PCI-bus interface 141.

In processing an image to achieve a desired result, pixel data relatingt -intermediate stages are stored in local mnemory 143. For example, sharpened image data is stored in the local memory 143 after a convolution operation for sharpeningI im-age-pixel data is performed. The sharpened image pixel data stored in local memory 143 can then be reused by the RIP 144. For example, the RIP 144 can use the stored data to calculate a difference value between the original image data and the sharpened image data. The difference value can be stored in the local memory 143 and later retrieved by the RIP 144 to calculate an absolute value (modulus) of the difference value resulting in a value for the mapping or gradient function, in accordance with the third embodiment described hereinafter.

While the first embodiment has been described in relation to a specific PC platform, the embodiment may be practiced in numerous other ways without departing from the scope and spirit of tihe present invention. For example, the linage technique can be embodied in an application specific integrated circuit (ASIC). It will apparent to one skilled in the art that the further embodiments of the Irnvention :.°'":described hereinafter can equally be practiced in these manners.

-eoa 20 Process of Second embodiment 4 is a flow diagram illustrating the linage filtering or enhancing technique of the second embodiment. Like elements of the first embodiment of. Fig. I contained '60400 in the •second embodiment are shown in Fig. 4 with the samne reference numerals.

•However, the original imnage 10 is filtered using a blur filter 17 iln Fig. 4 before being 25 directed to tihe linage compositing step 15. The blur filter 17 replaces tihe "identity ::::.filter" (or no filter), if implemented, of the embodiments discussed above. The blur filter 17 averages values over a plurality of picture elements of tile original linage for each picture element produced ina blurred linage 18, whichi is provided to the compositing step 15. The blur filter 17 smooths out abrupt chianges in an iniage 10 and -14consequently tends to disguise sharp transitions resulting from undesired noise in the image Turning now to Figure.5, an example of image filtering is illustrated according to the second embodiment of Figure 4. An original image 50 (depicting a banksia shrub) is provided to the blur filter 17, the sharpen filter 11 and the differential filter 13 (each depicted by a blackened rectangle in Fig. 5) which output a blurred image B(I) 52, a sharpened image S(I) 53 and a gradient map image H(I) 54, respectively. The gradient map image 54 is an "edge" representation image showing substantially more pronounced edges of the graphical object of the original image 50. The gradient map image 54 is denoted in Figure 5 as the function where I is a picture element value at position x,y of the original image Further, the function H(I) is normalised to values between zero and one such that a picture element of the gradient map image 54 at the most pronounced edge has a value of one and a picture element at the least pronounced edge no edges) has a value of zero Corresponding picture elements B(I) and S(I) of the blurred image 52 and sharpened image 53, respectively, are composited in the compositing step 15 using the gradient map image 54 to produce a final image 55 having a picture element value at i e location x,y denoted by F(I).

20 The final image 55 has the combined filtered features of the blurred image 52 and the sharpened image 53 in accordance with values determined from the gradient map image 54. In combining the blurred image 52 with the sharpened image 53, the opacity value of each picture element of the blurred image 52 has a complementary value to that of the opacity value of each corresponding picture element of the 25 sharpened image 53. Thus, the sum of the opacity value of each picture element of the blurred image 53 and the opacity value of each corresponding picture element of the sharpened image 53 totals to a value equivalent to fully opaque. For example, a picture element of the gradient map image 54 taking on a value of 0.75 results, when composited in steps 15, in a final picture element value comprising twenty-five percent opacity of the corresponding blurred image 52 picture element and seventy-five percent opacity of the corresponding sharpened image 53 picture value.

Applying a gradient mapping function 13 to the opacity of picture elements of filtered images being combined is advantageous because it lends itself to image s compositing using image compositing operators. However, those skilled in the art will recognise that other techniques for combining picture elements other than relying upon the opacity of the picture elements may be practiced without departing from the scope and spirit of the invention. For example, a color blend between a first color of a picture element of one image and a second color of another image can be obtained by a simple (or complex) interpolation.

Pseudocode for implementing the process of Figs. 4 and 5 is set forth in detail in Table 2.

TABLE 2 15 Let x =image.

Convolve image x with a blur filter, giving blurred image b.

Convolve image x with a sharpen filter, giving sharpened image s.

Take gradient of image x and store result in mapping function g.

Apply color map to g, such that opacity channel is set to average of color 20 channels.

Output (s in g) plus (b out g).

The blurred and sharpened images correspond with images B(I) and S(I) of Fig. 5. Further, the mapping function g and the output correspond to the map H(I) and the final image F(I).

In a variation of the second embodiment shown in Fig. 4, the sharpen filtering step 11 could be omitted, and instead of providing sharpened image 12 to step original image 10 is provided to step 15. All other features of Fig. 4 being -16substantially the same, this variation of the second embodiment provides an image noise reduction technique.

Process of Third Embodiment Fig. 6 is a flow diagram illustrating the image filtering process according to the third embodiment of the invention. An input image 60 is provided as input to an interpolation step 68, a step for determining the absolute value of a difference 62, and first and second sharpening filtering steps 64 and 66. The output of the first sharpening filtering step 64 is provided as a second input to the step of determining an absolute value of a difference 62 between the sharpened image and the original image 60. In particular, step 62 computes the difference between the two images and then determines the absolute value of that difference, which is provided as the mapping function provided to the interpolation step 68. This form of mapping function is preferable in that it is capable of more efficiently and more quickly being carried out in software and 15is hardware applications. The sharpened image provided by step 66 is also provided to the interpolation step 68, where the original image 60 and the sharpened image output by step 66 are interpolated or combined dependent upon the mapping function provided by step 62 to provide the final output image The third embodiment of the invention shown in Fig. 6 requires the sharpening 20 filter 64, 66 to be applied twice to the original image 60. In a fourth embodiment shown in Fig. 7, being a modification of the third, a more efficient image filtering method can be practiced if the mapping function g of Fig. 6 is allowed to be a continuous or piece-wise continuous function of more than one source, a combination of any of the following: the original image 60, the output of the first sharpening filter 64, and the output of a second sharpening filter 66. Thus, the embodiment of Fig. 6 is shown in Fig. 7.

In Fig. 7, an input image 80 is provided to an interpolation step 86, the input of step 82 for determining the absolute value of a difference between inputs, and to a sharpening filter 84. The sharpening filter 84 provides output to both the absolute 17value differencing step 82 and the interpolation step 86. The absolute value.difference step 82 generates the mapping function provided to the interpolation step 86. In turn, the interpolation step using the input image 80, the output of the absolute value step 82, and the sharpened image produces the final output 88.

s Pseudocode for implementing the third and fourth embodiments is set forth in Table 3.

TABLE 3 Let x image.

Convolve x with blur filter, giving b.

Convolve x with sharpen filter, giving s.

Let g= s x I (calculated pixel by pixel)., Apply color map to g, such that opacity channel is set to average of color channels.

1 5 Output (s in g) plus (b out g).

o..

In Table 3, the mapping function g is determined as a function of the sharpened image and the original image, where the difference between the two is taken 20 on a pixel-by-pixel basis and the absolute value of that difference is computed.

In a variation of the embodiment of Fig. 7, the sharpen filter 84 can be i replaced with a low-pass filtering step. This variation of the embodiment of Fig. 7 provides an image noise reduction technique.

The foregoing only describes a small number of specific embodiments of the invention and modifications, obvious to those skilled in the art, can be made thereto without parting from the scope of the invention. The image filtering processes can be implemented as computer software comprising a set of instructions to be carried out using a processing unit, which may be implemented using a general purpose microprocessor, a customised signal processor, or the like. The set of instructions may -18be stored on a recording medium, a non-volatile memory, or the like, for retrieval into memory internally or externally coupled to the processing unit, where the processing unit executes the set of instructions to carry out the process modules. Further examples of computer readable medium include a floppy disk, a magnetic fixed storage device or s hard disk, an optical disk, a magneto-optical disk, a non-volatile memory card, and networked resources. Networked resources include client/server system that provide information over a network such as the Internet and Ethernet network, for example.

*i -19- The claims defining the invention are as follows: 1. A method of filtering an image signal, the image signal representing an image, said method comprising the steps of: filtering the image signal to provide a filtered image signal representing a Sfiltered image; determining a mapping function from a predetermined arbitrary continuous function of said image signal; and combining the filtered image signal with the image signal in accordance with said mapping function to produce a final image signal representing a final image.

2. The method according to claim 1, wherein said predetermined arbitrary function is piece-wise continuous, being characterised in that the domain of the function has more than two corresponding range values.

3. The method according to claim 1, wherein a smoothened transition is produced in said final image signal between said filtered image signal and said image 15 signal.

4. The method according to claim 1, wherein the filtering step comprises the step of applying a sharpening filter to the image signal, and the filtered image signal 0 is a sharpened image signal.

The method according to claim 4, wherein the sharpening filter is a high-pass filter.

6. The method according to claim 1, wherein the mapping function comprises a spatial gradient map function of the image signal.

ooooo 7. The method according to claim 1, wherein the image signal, the S* filtered image signal and the final image signal are digital signals comprising picture 25 element information of the image, the filtered image and the final image, respectively.

8. The method according to claim 7, wherein said picture element information further includes information on the opacity of the picture elements.

9. The method according to claim 8, wherein the combining step further comprises the step of adjusting an opacity of each picture element in accordance with the mapping function.

The method accordingto claim 1, wherein the step of determining the mapping function is also dependent upon said filtered image signal, and the mapping function is an absolute value of a difference between said image signal and said filtered image signal.

11. The method according to claim 1, wherein said steps are implemented using a computer.

12. A method of filtering an image signal, the image signal representing an image, said method comprising the steps of: applying a first filter to the image signal to provide a first filtered image signal; applying a second filter to the image signal to provide a second filtered image 15 signal; -,generating a mapping function from a predetermined continuous arbitrary a o function of said image signal; and .combining the first and second filtered image signals in accordance with said mapping function to produce a final image signal.

S 20 13. A method according to claim 12, wherein said predetermined arbitrary function is piece-wise continuous, being characterised in that the domain of the function has more than two corresponding range values.

14. A method according to claim 12, wherein a smoothened transition is produced in said final image between said first and second filtered images.

15. The method according to claim 12, wherein the first filter is a sharpening filter, and the first filtered image signal is a sharpened image signal resulting from the sharpening filter applied to the image signal.

16.. The method according to claim 15, wherein the sharpening filter is a high-pass filter.

-21 17. The method according to any one of claims 12, wherein the second filter is a blur filter, and the second filtered image signal is a blurred image signal resulting from the blur filter applied to the image signal.

18. The method according to claim 12, wherein the mapping function comprises a spatial gradient map function of the image signal.

19. The method according to claim 12, wherein the mapping function is an absolute value of a difference between said first filtered signal and said image signal.

The method according to claim 12, wherein said steps are implemented using a computer.

21. An apparatus for filtering an image signal representing an image, said apparatus comprising: input means for receiving said image signal; at least one filtering means, coupled to the input means, for filtering the image signal to provide a corresponding filtered signal; 15 mapping means, coupled to the input means and capable of receiving the image ~signal, for providing a mapping function of the image signal in accordance with a predetermined continuous arbitrary function; compositing means, coupled to the input means, the at least one filtering means and the mapping means, for combining said image signal with said each filtered signal 20 in accordance the mapping function to provide a final image signal.

22. The apparatus according to claim 21, wherein said predetermined arbitrary function is piece-wise continuous, being characterised in that the domain of the function has more than two corresponding range values.

23. The apparatus according to claim 21, wherein a smoothened transition between said image signal and said each filtered signal is produced by said mapping function.

24. The apparatus as recited in claim 21, wherein the at least one filtering means includes sharpen filtering means for sharpening said image signal to provide a sharpened image signal as said corresponding filtered signal.

-22 The apparatus as recited in claim 21, wherein said at least one filtering means comprises means for blurring said image signal to provide a blurred image signal, said blurred image signal provided to said compositing means as said image signal.

26. The apparatus as recited in claim 21, wherein the mapping means further includes differential filtering means for spatially differentiating said image signal.

27. The apparatus as recited in claim 21, wherein the mapping means further comprises means for determining a difference between said image signal and a filtered signal, and means for calculating the absolute value of said difference.

28. The apparatus as recited in claim 21, wherein the at least one filtering means comprises edge detection filtering means for detecting edges of said image in said image signal.

29. The apparatus as recited in claims 27 or 28, wherein said mapping 15 means comprises means for normalising said mapping function.

30. The apparatus as recited in claim 21, wherein said filtering means is implemented using a graphics processor.

S31. The apparatus as recited in claim 21, wherein said combining means comprises a compositing circuit having predefined compositing instructions usable for 20 combining said image signal with said each filtered signal.

32. The apparatus as recited in claim 31, wherein said compositing circuit is implemented using a graphics processor.

33. A computer program product having a computer readable medium having a computer program recorded thereon for filtering an image signal, the image signal representing an image, said computer program product comprising: means for filtering the image signal to provide a filtered image signal representing a filtered image; means for determining a mapping function from a predetermined continuous arbitrary function of said image signal; and -23means for combining the filtered image signal with the inage signal in accordance with said mapping function to produce a final image signal representing a final image.

34. The computer program product according to claim 33, wherein said predetermined arbitrary function is piece-wise continuous, being characterised in that the domain of the function has more than two corresponding range values.

The computer program product according to claim 33, wherein a smoothened transition is produced in said final image signal between said filtered image signal and said image signal.

36. The computer program product according to claim 33, wherein the filtering means comprises a sharpening filter, and the filtered image signal is a sharpened image signal.

37. The computer program product according to claim 33, wherein the mapping-function determining means comprises means for generating a spatial gradient 15 map function of the image signal.

38. The computer program product according to claim 33, wherein the combining means further comprises means for adjusting an opacity of each picture element of said filtered image signal and said image signal in accordance with the mapping function.

S 20 39. The computer program product according to claim 33, wherein the means for determining the mapping function is also dependent upon said filtered image ~signal, and comprises means for determining an absolute value of a difference between said image signal and said filtered image signal.

The computer program product according to claim 33 wherein said computer readable medium is selected from the group consisting of a floppy disk, a hard disk, an optical disk, a magneto-opticaldisk, a memory device, a networked resource, and a non-volatile memory card.

-24- 41. A computer program product having a computer readable medium having a computer program recorded thereon for filtering an image signal, said method comprising the steps of: first filtering means for filtering the image signal to provide a first filtered image signal; second filtering means for filtering the image signal to provide a second filtered image signal; means for generating a mapping function from a predetermined continuous arbitrary function of said image signal; and means for combining the first and second filtered image signals in accordance with said mapping function to produce a final image signal.

42. The computer program product according to claim 41, wherein said predetermined arbitrary function is piece-wise continuous, being characterised in that the domain of the function has more than two corresponding range values.

S: 15 43. The computer program product according to claim 41, wherein a smoothened change is produced in said final image between said first and second filtered image signals.

The computer program product according to claim 41, wherein the first filtering means is a sharpening filter, and the first filtered image signal is a sharpened image signal resulting from thesharpening filter applied to the image signal.

The computer program product according to claim 44, wherein the sharpening filter is a high-pass filter.

46. The computer program product according to any one of claims 41, wherein the second filtering means is a, blur filter, and the second filtered image signal is a blurred image signal resulting from the blur filter applied to the image signal.

47. The computer program product according to claim 41, wherein the generating means comprises means for producing a spatial gradient map function of the image signal.

25 48. The computer program product according to claim 43, wherein the generating means comprises means for determining ani absolute value of a difference between said first filtered signal and said image signal.

49. The computer product according to claim 33, wherein said computer s readable medium is selected from the group consisting of a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a memory device, a networked resource, and a non-volatile memory card.

DATED this Nineteenth Day of June 1997 Canon Kabushiki Kaisha Patent Attorneys for the Applicant

ST

SPRUSON FERGUSON °A 113 C

Claims (4)

1. A method of filtering an input image to produce a filtered output image, said method including, for substantially each candidate pixel in the input image, the step of: if the gradient of the candidate pixel and neighbours of the candidate pixel is greater then a predetermined noise threshold, applying a sharpening function to said candidate pixel to produce a filtered output pixel; and otherwise utilising said candidate pixel as the filtered output image pixel.

2. A method as set out in claim 1, wherein said sharpening function includes the steps of: applying a sharpening filter to said candidate pixel to produce a corresponding sharpened pixel; and compositing said corresponding sharpened pixel with said candidate pixel so as to produce said filtered output pixel. S3. A method as set out in claim 2 wherein said compositing process 20 utilises a factor proportional to the gradient at said candidate pixel.

4. An apparatus for filtering an input image to produce a filtered output image, said apparatus including processing means for processing substantially each candidate pixel in the input image, said processing means comprising: means for applying a sharpening function to said candidate pixel to produce a filtered output pixel if the gradient of the candidate pixel and its

9. neighbours is greater then a predetermined noise threshold; and (CFP0641AU)(OPEN29)436644) [O:\CISRA\OPEN\OPEN29]394754:MXL -8- means for utilising said candidate pixel as the filtered output image pixel if said gradient is less than or equal to said noise threshold. An apparatus as set out in claim 4 wherein said sharpening function includes: a sharpening filter for application to said candidate pixel to produce a corresponding sharpened pixel; and compositing means for compositing said corresponding sharpened pixel with said candidate pixel, thereby to produce said filtered output pixel. 6. An apparatus as set out in claim 5 wherein said compositing means utilises a factor proportional to the gradient at said candidate pixel. 7. A method of filtering an input image to produce a filtered output image, the method being substantially as herein described with reference to any one of the embodiments of the invention shown in the accompanying drawings. ee6 8. An apparatus for filtering an input image to produce a filtered output image, the apparatus being substantially as herein described with reference to any one of the embodiments of the invention shown in the accompanying drawings. S Dated 8 October, 1998 Canon Kabushiki Kaisha Canon Information Systems Research Australia Pty Ltd Patent Attorneys for the Applicant/Nominated Person SPRUSON FERGUSON (CFP0641AU)(OPEN29)436644) [O:\CISRA\OPEN\OPEN29]394754:MXL