JP6831389B2 - Processing of multiple HDR image sources - Google Patents

Description

The present invention relates to methods and devices for accurately processing the composition of multiple images (or video of temporally continuous images) from different sources, at least some of which have different luminance characteristics with high dynamic range.

Until a few years ago, all videos were encoded according to the so-called low dynamic range (LDR) principle, nowadays also called standard dynamic range (SDR). That is, whatever the captured scene, the maximum value of encoding (eg, 8-bit Luma Y'= 255 or 100% voltage driven by an analog display) is the standard 100 knit display peak brightness PB_D (ie, display). It should be standardized and defined for white rendering on LDR displays (the brightest white that can be rendered in). If people actually buy a slightly darker or brighter display, the viewer's visual system (for example, if there is a night view of a horror movie) adapts the image to be appropriate rather than being too bright, for example, annoyingly. Seem. This is a relative rendering paradigm that always maps the maximum brightness color of the encoded image to the peak brightness of the display, whatever it may be. This works if the display actually available on the rendering or video / image consumer side is not really much brighter than 100 nits, but the display is much brighter, say 10 times brighter, ie PB_D = 1000 nits. In some cases it can have undesired consequences.

Of course, even with perfectly uniform lighting, the reflections of different objects already give a contrast ratio of 100: 1, and there are differences in lighting in different areas, so this is for everyone to work together. Is a television system colorimetric analysis that is precisely and strictly defined in, and for actual programming, this typically means maintaining tight control of the scene lighting settings. Typically, the light is radiated and exposed (by selecting the iris setting), and the approximately code white, that is, the white (in the brightly illuminated area) of the scene mapped to the maximum Luma code, has a coded peak brightness of PB_C. Correspondingly, typically, brighter objects or areas in the scene can be clipped to their PB_C brightness values. Typically, cameras, especially the early generations of digital cameras in the early 2000s, had problems capturing both very bright and dark areas at the same time, i.e., for example, room windows or car windows. Some of the scenes visible outside of are typically clipped to white (their red, green, and blue additional color components R corresponding to their square root Lumacode values R'= G'= B'= 255. = G = B = max).

However, recently, both high dynamic range cameras (and, importantly, new HDR image capture or common image creation modes) and HDR displays have begun to emerge, with other video and / or image technologies such as image coding. The conversion, image composition, etc. must be adjusted to meet these new technical level requirements. In this application, if the dynamic range is initially specified only for peak brightness (ie, the brightest rendered brightness), it is assumed that the bottom edge of the range is actually zero (actually, the display front plate or It should be noted that the light reflections of the cinema screen, eg, depending on viewing conditions such as 0.1 knit), these further details are irrelevant to the particular description. High dynamic range displays that initially simplify to high display peak brightness may differ from low dynamic range displays, especially SDR displays, unless technically specific is done for the colors of the darkest images. It can be said that there is no particular reason why the darkest pixel color should be rendered. There are several ways to define dynamic range, the most natural of which is commonly used in the discussion below is the brightness dynamic range rendered on the display, that is, the darkest color to the brightness of the brightest color. It should also be noted that it is brightness (of course, psycho-visual aspects such as a single bright pixel in a dark scene may not be very impressive, but some of the following: In that technical description, only the design of technical elements that abandon this and conform to what humans need is explained). An HDR image can be defined as any image that can include pixel colors, and especially bright pixel colors as described above cannot be encoded by SDR image coding. Since the color measurement of such SDR image or video coding is fixed (in the video Rec.709 OETF corresponding to the sRGB OETF of the still image), the 8-bit Luma coding is only 1000: 1, that is, 0.1 knit. It was mathematically confirmed that it can handle a dynamic range that does not exceed 100 knits. So, for example, if you want to be able to encode the brightness of a scene that is displayed and rendered up to 200 knits, which is twice as bright as the white of a normal SDR scene, a new form of HDR image or video coding is available. Required (the new SMPTE 2084 is an OETF capable of encoding the brightness of brighter scenes, ie up to 10,000 knits as Luma, for example 10 or 10 which are further processed as "normal" Luma for video compression such as HEVC compression. An example of at least how to render on a display under normal 8-bit Luma TV viewing conditions). Because the human visual system can handle much higher dynamic range and is also present in various practical HDR scenes in the world (eg in a cave or in a dark room with small windows in the sun). , There is a need for a high dynamic range video processing chain to enhance the "sharpness" of the image. Since indoor lighting is typically 1 / 100th of outdoor illumination, knowing that a black object reflects about 1% of the incident light, this already requires a dynamic range (DR) of 10,000: 1. Means to do. However, a 1,000,000: 1 DR of encoded image content is normal if humans can perceive 1% of the darkest color in the night view in a dark field of view (some cameras). Can be achieved, for example, can be measured logarithmically). Of course, for example, if a large cave is visible from the inside, in principle in some situations you can artistically choose to actually render the surrounding cave shape as the smallest clipped black, but in other scenarios. So, for example, when something happens in a cave, we really want or need all the different encoded gray values from the brightest to the darkest areas of the pixels in the image (partial of some cave shapes). Some of the appearance looks pretty weird when clipping black as the psychological detection of the foreground and background looks like a torn piece of paper, thus rendering it more dark inside the cave. Can be avoided). So while the display rendering range from 0.01 to 10,000 nits is actually a great HDR range, how do you encode or render two factors that are more than double the 100 nit PB as described above? It is already qualified as an HDR technology with a corresponding need to carefully watch whether it processes differently from the conventional SDR video concept. It should also be noted that the coding system itself does not originally have a dynamic range unless the reference display is associated with it, for example R'= G'= B'= Y'= 255 is 100 knits or Indicates that it should correspond to PB such as 1000 knits. A common mistake from before is to think that the dynamic range of the encoded image is strongly tied to the amount of bits used. This is true for linear coding, such as camera ADCs, which require more bits across a larger range of pixelwell photoelectron conversions, but also at least somewhat higher coding accuracy or higher dynamics. It is good to have bits for the range, but the amount of bits required also depends on the selected shape of the Lumacode allocation function, also known as the analog-to-digital converter (OETF). Therefore, a 10-bit Y'CbCr-encoded image can be both an HDR image with colors that can be encoded up to the maximum PB_C value and a high-precision SDR image. What type of image is, that is, how to render the Luma on a display with PB_D high enough to display all the encoded Luma as the corresponding brightness, is typically received, for example. It is determined by reading metadata such as the co-coded PB_C value in the knit of the image. PB_C can also actually be interpreted as PB_D for an ideal reference display in which the image is color graded, i.e., the display will have an optimal appearance rather than being too dark or too bright.

So, for example, coding an image that can encode an image with rendered brightness up to 1000 knits is a good starting point for getting good HDR, and the reader has mentioned other details. Unless you need to summarize some of the following teachings, you can remember such settings.

Therefore, in reality, there are scenes with a very high dynamic range (for example, a dark object of about 1 knit indoors while simultaneously viewing a sunlight-lit object with a brightness of 10,000 knits or more outside through a window. The display is better (PB twice brighter than 100 nits, now 1000 nits are appearing and thousands of nits are expected), so it's not exactly the same as the original, but at least very Natural or at least satisfying is important, but the goal is to be able to render these images nicely. Therefore, HDR relates not only to the brightest and darkest colors, but also to all the brightness in between, that is, in fact ultimately to the psycho-visual appearance of humans. Of course, technically, for some HDR processing techniques, such as just encoding, it is rendered on the display because it needs to happen by moving the display correctly in order to see the colors that humans appear in exactly. The required appearance can be formulated from the viewpoint of technical brightness. However, if the applicant wants to design a versatile HDR technology system, it is just a direct one-to-one connection of encoding (to help the reader understand the entire story below and what it means). I would like to emphasize that the habitual viewing paradigm needs to be abolished. Rec. At 709, the broadcaster can see what brightness he is shooting by seeing how it is rendered on the monitor using a standard (single) SDR monitor, and the 100 knit PB_D SDR display Since it has only a single taste, it is rendered on all TVs in the same house everywhere. The only thing that still needs to be done is to convert the rendered (camera-measured) brightness to Lumacode by approximately applying the square root function (because such code is available). (Because it makes better use of the technical details of the conversion technology), the inverse (EOTF) function on the receiver side is desired by viewers viewing the decrypted content on substantially the same TV in the same viewing environment as the creator. Guarantee that you will experience the program as. It is still the field of view of HDR10. EOTFs of different shapes are Rec. The colors used in 709, but rendered on a reference display that can be visualized on the created side, are still rendered with the same brightness on the receiving side by simply applying a fixed formula to the received Luma. Will be done. However, the actual HDR world is very variable, such as different viewers having different displays with different PB_Ds and different amounts of lamps in the viewing room. Exactly HDR10-encoded HDR movies are often relatively dark, making it difficult for HDR movies to be seen in bright ambient lighting (although important movements are often performed in the relatively dark parts of the video). It was criticized for containing the pixels of. HDR10 wants display makers to solve it and does not offer a solution to it. However, the applicant has found that this is a complex issue that generally depends on the type of HDR scene image, and the content creator has the final (variable, i.e., certain brightness and desired color of Luma) of the HDR image color. We think that we should have a mechanism that allows us to have a say in display rendering (not an inverse fixed calculation that is directly connected to the representation on a one-to-one basis).

Readers typically see content in different situations (sit in a weakly lit living room, dark house or cinema at night rather than actually standing in a bright African landscape). Therefore, it should also be understood that there is no identity between the brightness in the scene and the brightness finally rendered on the television (or other display). In practice, it is not possible for a TV to accurately render an object such as at least a billion knit sun, and no viewer wants to be blinded by it (when looking at a small screen, it is determined by the angle or it. Glare is not the same as the 4 * pi solid angle ambient environment, so that must be taken into account). Applicants have resolved this conversion of appropriate brightness rendered on the display from relative RGB values taken by the camera by defining a master HDR color classification starting with coding and communication. This HDR graded image set has a reference monitor available to the artist, for example if they can sit in a typical viewing environment and fully grade HDR images against a 5000 knit PB display. It can be created as the artist desires (referring to the artist, but what the technical aspects of the invention should be, allowing the artist to communicate and control the requirements is the reader. Should be clearly clarified). However, that's only half the story, ie (in the new demanding video set) it is possible to consistently determine the artistic and attractive appearance image of the HDR scene (and quite under the same visibility constraints). When viewing the same 5000 knit PB_D display, the image can be faithfully displayed). What happens if the consumer who obtained the 5000 knit HDR image has only a 1000 knit display is described below (display adjustment). This is a completely different issue, quite different from what can be handled by a direct one-to-one communication approach, so start pondering what a single reference system with one reference display PB_D should be. Can be done.

If this application speaks of a "high dynamic range" image, it means that the image has image content suitable for display on at least 1000 knit PB_D displays (of course, the content is 100 knit or 500 knit). It can be shown on a display, but it is not perfect, for example, because the lamp will be darker than ideal). Therefore, this image has visually interesting content when displayed on a higher PB_D display, i.e. the brightest color that normalizes as Y = 1 and has image objects that are much brighter than average. At least some objects that are much darker than typical of SDR images compared to). For example, an indoor scene may have a brightness of 500 knits or more in the original scene, and the outside brightness may be thousands of knits in the real world, depending on whether there is a bright greenhouse or a narrow tunnel area of a few knits or less. In the display, the indoor is rendered to 100 nits or less, and the more available ones of the display are optimally used for the outdoor object, for example, a display capable of rendering up to 1500 nits or more (PB_D> = 1500 nits). For example, a movie or program of "cloudy day" is rendered to a maximum of 500 knits, and a sunny outdoor is rendered to a maximum of 1500 knits. Therefore, even during rendering, there is a ratio between (1500/2) / (100/2) or at least 500/100 bright and dark image areas (which is typically different). It can be defined as the midpoint of an illuminated area, i.e., an area with a maximum contrast of 100: 1, typically 30: 1 due to object reflection). When this is associated with the Luma code in a continuous function, those Luma also spread apart, but are not very useful for specifying content (they depend on the shape of the OETF and, in some cases, technical requirements). When this application refers to an image with a lower or lower dynamic range (than an HDR image), it is at least 1 stop (multiplication factor 2) lower, typically at least 2-3 stops (4x or 8x or more). We talk about images that have peak brightness (or are actually related to the reference display of the PB) (perhaps scene images taken with the exact same original camera, i.e. different graded appearances of HDR images of the same scene). The SDR image complies with the 100 knit standard PB_C. The contrast ratios of all the objects in them are typically 100: 1 or less or up to 1000: 1 or less, i.e. typically see a more uniform brightness or Luma histogram.

The second thing you need to know if you know the image pixel luminance dynamic range where you need to define the Luma code (which should be finally rendered) is how the code is actually distributed, that is, which The corresponding Luma code Y'in the code assignment function (typically which brightness the Luma code 743 should correspond to in addition to 1023 in the 10-bit coding that maps to PB_D of the associated reference display) to the object or pixel brightness Y. Or vice versa.

Over the last 2-5 years, several companies have proposed different aspects of encoding HDR video (ie, moving images or, in other words, sets of temporally continuous images). Due to the rapid move to the first product scheduled for 2016, some corners have been cut despite the complexity of coding and processing HDR images. This gave rise to different coding techniques based on different basic ideas. The fact that it is not ultimately determined is quite complicated (certainly compared to the only colorimetric simplicity of the SDR video era five years ago), as all image processing systems have to take that difference into account. there is a possibility.

Dolby Laboratories Inc. Started a somewhat expensive two-layer system to accommodate cheap chips (ie, a basic image and a corrected image are needed to create a single HDR image in line with the scalability concept in video compression), Single-layer coding was also designed, and SMPTE ST. It provides a basic EOTF, the so-called perceptual quantizer PQ, recently standardized as the 2084. This technique is available at https: // www. smpte. org / systems / default / files / 2014-05-06-EOTF-Miller-1--2-handout. Well explained in pdf.

The idea behind it is that the classical square root (or Rec. 709 to be exact) optoelectronic conversion or conversion function OETF (inverse to square power EOTF), which was conventionally used to convert scene brightness to Lumacode, Practical numbers such as at least 14 bits or less for high dynamic range that can occur in real life, no matter how many bits are used (or for main grading of scenes with 20 stops or more). Not suitable for high contrast ratios between different regions that can occur in HDR images. However, it was a good inspiration. Originally designed for the accidental physical behavior of CRT electron guns, the square root function maximizes the code because it closely follows the way human vision translates scene brightness into visual brightness code. It was a good way to take advantage (the next code corresponds to the impression of the next psycho-visual brightness, so too much code isn't wasted so much that it's just not easily recognizable anyway, and worse, it's smooth. There is no region where there are too few codes to accurately encode the exact luminance gradient). This curve can now be generalized for each successive luminance interval (eg, one stop brighter) to approximately equalize the amount of Lumacode to the maximum codeable brightness, which is usually 10,000 knits in Dolby (for example, one stop brighter). This should actually be enough for practical display rendering, as some people complain that it's too bright, and it's experimental but flashes over 20,000 knits even in large areas. Even regions can produce impressive images, of course all depend on how the brightness of various image objects is allocated along the possible range and always need to meet up to PB for each image type. Those who complain about painfully bright images that may be dangerous, knowing that there isn't, should be aware that the full moon at night is also 5000 knits and no one is blinded by staring at the moon. Must).

So Dolby can standardize the reference EOTF and use its inverse as OETF (or nearly inverse if more behavior is desired) to define Luma for every object / pixel in the image. , After applying conventional HEVC coding techniques, record those codes (classically encoded colors as YCbCr, but RGB encoding can also be used) on, for example, Blu-ray® discs. be able to. Then there is a master HDR grading on the BD, which is rendered by a 5000 knit TV that can decode it, for example, as the artist desires. Therefore, there is a mechanism that provides end-to-end, one-to-one to display an HDR image (only) of a scene on a consumer's premises on a consumer display. Of course, when we say via BD below, it is true that all other image or video communication mechanisms, such as satellite broadcasting, internet video distribution, etc., are also meant, as the concepts presented have great applicability. The vendor will understand. The 2084EOTF has also been selected as the Luma-luminance mapping function for HDR10 coding.

The British Broadcasting Corporation BBC is designing different aspects of HDR video coding. They have standardized OETFs with somewhat different shapes (because they started from the camera field of view rather than the display side which standardizes the EOTF of the reference display). Their views and techniques are http: // downloads. bbc. co. uk / rd / pubs / www / www-pdf-files / WHP283. Well explained in pdf.

They are the classic 100 knit Rec. Classic square root Rec. Up to mid-gray (Y'= 0.5) of 709 OETF. 709 OETF Function Defines an OETF that strictly follows the shape and is logarithmic if the brightness is high so that brighter colors than the white object color scene can be narrowed down to available codes. It was inspired by the fact that cameras have been around for a long time when it was necessary to create good LDR output images for scenes with a little too much HDR. That is, instead of a pixel luminance contrast ratio of about 200: 1, we tried to narrow down to a brightness that is four times brighter than the white of the scene that should be mapped to Y'= 255, for example, because it has some areas that are too bright. Rendered to brightness Y = 100 knits. White objects in the scene are rendered grayish, but are sometimes psychovisually acceptable because the brain sees and interprets the bright pixel areas in the entire image. Such bright scene brightness can always be mapped to the display PB by simply reducing the camera exposure (eg by changing the iris), but the newsreader's face may be too dark. What the camera did was to follow the curve to, for example, Y'= 80% or some other knee point, from which it uses a curve with a small slope, so there are so many highs in the remaining 20% of the code. It can represent the scene brightness. That is, you can use a specific log curve of the camera that slightly reduces the brightness of dark pixels (the viewer is standing in a relatively dark (probably shadowy) area, and on a bright display anyway I understand that I can adjust the brightness properly with, so I'll probably darken my face a bit so that it doesn't necessarily matter in high contrast scenes), but I'm free to do some code about the colors of the actually bright pixels. Then, for example, it is soft clipped and expressed a little coarsely. In this way, there is a balance between the colors that ultimately need to be rendered bright enough and the visual quality that is still reasonable for the bright colors. The problem, however, is that each camera logarithmically mapped the captured relative scene brightness to the Lumacode in a different way (of course, essentially absolute), which is still a very good LDR camera. It was still reasonable if it was only, but it is more problematic for a camera that is a really good HDR camera that is supposed to shoot complex HDR scenes with 20 stops. So the BBC wanted to define a fixed curve for all practical purposes. Nevertheless, by applying the fixed OETF curve as a brightness mapping curve to the graded image of the master HDR as input, a somewhat reasonable SDR version is obtained, and when this image is broadcast, the conventional display will SDR. The image can be rendered directly and the HDR display can apply a fixed inverse curve to reconstruct the master HDR image. The advantage of the BBC method is that there are few technical changes as everything works as before (but this is overly deceptive and pays during production or at least some video compositing). Comes with a price to pay).

They have a variety of different starting points, for example Dolby may be more focused on the professional film market, and the BBC is more focused on practical television requirements such as show field creation, television equipment deployment, etc. There may be. However, an interesting big difference for this patent application was that the BBC thought that pixel brightness (or actual colors) should be represented relatively, while Dolby thought they should be represented absolutely. That is. This is a big difference in thinking, as it determines whether the gaps that need to be crossed for optimal rendering stand on the scene reference side or the display reference side. It is a big difference in which brightness is encoded in the two images that need to be combined, but going further, it is that such alternative philosophies combine in terms of the essential approach and characteristics. Affects how to define what needs to happen.

So the BBC defines a basic OETF that defines HDR brightness and Luma that are relatively master graded (or at least taken with a camera with appearance determinations such as selecting exposure and possibly gamma curves). By approaching the problem, the appropriate brightness rendered on the display should be calculated by applying system gamma, which depends on both the display PB_D and the brightness of the viewing environment. This means that if the display maker gets BBC-encoded content that contains up to 12 times the white level of the scene but does not depend on the actual scene content and the camera's iris settings, then either of the two Means you can do it. He can use the relative philosophy of BBC and can always map the code of PB_C to display PB_D no matter what the display PB_D is (gamma brightness depending on which PB_D actually has). And apply the contrast correction function, but at least white is rendered to white, which means that there are no such overly bright areas but only relatively dark pixel colors such as foggy scenes. , Such darkly encoded colors also have at least one advantage of establishing an easy calibration mode in which they are rendered darker than PB_D, while the brightest part of the scene is with variable PB_D brightness. Rendered. This looks good for compositing image data, but is it really?). That is, he finds that this encoded image brightness data is suitable for, for example, an accurate absolute brightness PB_D value of 1000 knits or its periphery, and the television has only 500 knits PB_D, for example (by blind full area mapping). If so, you can claim to downgrade yourself. However, Dolby should specify the absolute conditions under which the brightness should be rendered, at least if he has a high quality 5000 knit Sim2 or pulsar display, and the required object brightness along the brightness reference range of the master EOTF 0-10000 knit. I thought it was not important how this image was generated artistically as far as I could define. So, for example, if you have a dark face in a horror movie, in absolute terms this image should be rendered, for example in 15 knits, and can be rendered on both low and high dynamic range displays. It is usually said that it can be done. However, in a relative way, even if 15% of 100 knits does not mean 15% of 5000 knits, this face because the display PB, which relies on the gamma function, modifies it somewhat. The brightness of is adjusted, but the brightness at which the face is finally rendered on the display is not determined very accurately. This makes the artist think about what they like when significant changes occur on the rendering side, which is beneficial for more accurate absolute rendering (when using the full dynamic range of the display, of course, viewing). The sensitivity of the person adapts accordingly, but typically uses a subrange of an HDR display if the content requires rendering, for example up to 900 knits, at least up to 99% of the image pixels). If the actual PB_D is lower than the encoded PB_C, i.e. the brightness that needs to be rendered to optimize the image, then some downgrading strategy is still needed, so the absolute brightness code. Rendering also does not have a complete answer and should not be too blind to the emergence of reclassification needs for a particular HDR scene or scene type. The reader can see how these different philosophies not only lead to different codes that determine EOTF, but also different processing of the image, i.e., the brightness of the object pixels in different regions along the curve or the corresponding reference brightness range. It can be thought that the allocation will also lead. And if you have the relatively simple problem of mapping a portion of the input image with a particular PB_C to a lower PB_D display, and if that already leads to a difficult discussion, you can design some at will. (For example, when using what is called an r-curve, the shape has a slightly increased brightness, so the slope becomes smaller and smaller for brighter brightness, that is, the first quarter of a circle or ellipse. Effectively compress them into smaller subranges of the SDR brightness range, such as 1), stretch the enhancements for dark colors in the HDR to SDR brightness mapping curve to further enhance the darkest colors. Because it can increase the brightness of a typical object in the input luminance range or part of the curve, eg someone's coat, the reader can see different images, different dynamic ranges, usually different lighting, and potentially different codes. You can imagine things getting more complicated if you want to reconcile the brightness of the object that has been transformed.

Applicants like Dolby also consider it best to define a master EOTF that we have defined as 0-5000 knits. One reason is that images taken in bright African scenes are basically absolutely different from Norwegian autumn, with all sorts of mathematical (at least affecting brightness) color conversions. You can apply it to finally get the look of the right brightness, but why haven't you already encoded this difference in the Luma value of the reference brightness range and the corresponding absolute brightness? (It is advantageous to design those lumas to correlate with psychovisual brightness, which is appropriate when grading from master-initiated HDR images corresponding to images with different dynamic ranges that have neither HDR10 nor BBC. Artistic demands can be easily encoded in additional curved shapes.) However, most importantly, only the final rendering of the final pixel color (even if it happens on a different PB display) Yes, and at least without the relevant reference display (defined by grayscale behavior as an EOTF as well as peak brightness), the image is just a set of numbers (preferably correctly decoded) and nothing more. .. It's about the last experience of HDR images, not about how they were designed up to that point, but what the grader sees on the reference display. It should be possible for the creator to create an explicitly defined start image. Therefore, we must start by standardizing the reference EOTF that fixes this (otherwise, for example, the digital Luma number in the computer used for color grading is undefined), and on top of that. Further systems can be built (even if one wishes to return some relativity to the system, at least it is then relatively controllable, based on some common preferred reference brightness measurement systems. It is determined).

Now that we have summarized the background of the basic HDR reality that needs something new above, we use a different image that the author has decided for some reason that it must be encoded in one or another way. Move to a more practical situation so that you can.

Differences in behavior between Dolby EOTF vs. BBC EOTF (applicants have designed an EOTF very similar to Dolby PQ or can use Dolby PQ) can be seen in FIG. OETF113 is a PQ EOTF (only the portion up to 1000 knits is shown because the other two curves are obscured when the graph is shown up to 10000 knits). Since the BBC OETF112 is a relative curve, it is possible to discuss the absolute brightness to be encoded, but it is usually up to 500 nits (or perhaps up to 1000 nits, but if it starts to show errors and becomes less appropriate. There is). Rec. It should be remembered that the 709 curve can only encode absolute brightness up to 100 knits and for various reasons there may still be a lot of SDR content in the distant future (eg HDR). Not easily converted to version, etc.). Since we want to avoid the discussion of bit depth, we show a relative luma on the Y axis, which is not very relevant to this application (maximum because all coding uses 10 bits for 3 color channels). Luma 1023 is assumed to be 1023/1023 = 1.0). When obtaining an input Luma of 0.5 (ie 512, or 128 for 8 bits), it is clear that it means something as a corresponding brightness, depending on the coding method used (around 0.7). A simple pixel Luma replacement switch can suddenly produce very dark or very bright replaced pixels, as it can be 10 times different somewhere). FIG. 1B shows how the curves differ when a curve is also defined for the relative 0-1.0 input luminance. If it starts with the Luma of the encoded image, it is clear that the rendered brightness can be calculated from the inverse curve.

Ignoring all practical issues like color errors after various coding qualities, chip redesign costs, etc., three coding mechanisms work well when only a single HDR image set needs to be transmitted. Although it works, it is the part focused on in the first phase of research, development and standardization.

However, a practical HDR processing system needs more, or the user is left with a lot of quirks and questions that he cannot fulfill as a system designer.

Applicants do not have a typical single type display in the field (those who need to have a 5000 knit PB display, or perhaps a 100 knit legacy TV that can only buy a 1000 knit PB TV) For people), we designed an encoding system that could not handle only the communication (encoding) and use (eventually proper rendering) of a single standard HDR video, but the system was per specific display. By being able to calculate the optimal (re) graded image for, it is possible to process various possible display types with different PB_D in the field, the technique referred to by the applicant as synchronism or display tuning. .. The main features, and in particular the decoder 251 are summarized in FIG. 2 (as necessary for this application).

Applicants say that the final appearance optimization, which depends on the viewing environment, is not the final gamma proposed by the BBC, but rather the television (though calculated by the interior of the receiver, eg, television or BD player), television (ie, the BD player). Image content due to the complexity of the image, not determined solely by the designer), but solely on the basis of receiving factors such as actual display peak brightness and viewing ambient brightness. It takes the approach that it can be done on the display side by a mechanism that depends on itself, and so is the optimization for the characteristics on the rendering side. This crosses the old and difficult gap between content creation and content use. The receiver is the only one who actually knows the display capabilities that are actually available, while the creator is definitely the only one who knows about the content, such as artistic demands. If the receiver cannot sell a movie director or DOP instead of a TV to optimize the color appearance, then at least the creator can be at least roughly parameterized with the intent, and the image creator It can be mathematically specified that "if I have this PB_D display, I at least prefer to render various pixel brightness in this way".

The applicant's technician said that the part that depends on the image content is that the artist's color grader corresponds to different dynamic ranges (simply PB_C) at the two ends of the possible range (of the peak brightness of the display). ) Claim that it can be processed by allowing it to be specified according to the optimal color transformation that changes at least the object pixel brightness for the two appearances (although automatic image analysis functions can also be used, i.e. the current The brightness mapping function can also be determined based on the established brightness characteristics of the image). Therefore, the HDR appearance image should correspond to the slightly higher PB_C that the applicant chose 5000 nits (although the applicant can use another PB_C without hesitation), and the low DR image is a legacy SDR display. It may be 100 knit SDR grading as it satisfies, for example, 90% of the original existing market of the user. In certain scenes, the color grader says, "If I only have 100 knit PB, I would put the lamp on the image rendered with 4000 nits to make the lamp object look nice and bright in the rendered image. Now not possible if you first want to have enough brightness space in the display brightness range to render low brightness in the room at 95-100 nits, i.e. objects between 10 and 90 nits, for example. So maybe every lamp pixel is exactly 100 knits. " For example, the same can be done to properly darken the night view and brighten the streetlights sufficiently for both 5000 and 100 knit display renderings. The reader looks good at night in both HDR and SDR (especially with additional constraints such as IC complexity), i.e. the correct brightness of the object (eg, black sitting in a dark shadowed area) at night. Grading should be stopped a bit to ponder that it's not trivial, but at least (unless, of course, the display is mostly spec-compliant and you haven't created a completely different image by yourself). If it is not perfectionism, it can be done reasonably.

One is for situations with a high quality HDR display for rendering images, and the other is for conventional 100 knit SDR displays. These two graded images (M_HDR and M_LDR, their). The appearance of the luminance distribution is typically determined by humans), and the applicant can encode and communicate two full sets of images, i.e. dual-encoded, which requires two channels, etc. I wanted to avoid the need. The applicant also wanted to be able to encode two different DR graded images with existing MPEG type DCT-based encoding, so a set of images that are actually communicated (eg, not necessarily). Although not, it was decided to parameterize one of the two sequences of the video image with the color conversion function (F_L) for use in calculating the image from the SDR sequence of the image). .. Thus, a master HDR image M_HDR, such as, for example, the HEVC-encoded video signal S encoded by the video encoder 203, can be communicated via some video communication systems 250 (the person skilled in the art can use the system). Can be used in many systems, such as aerial broadcasting between professional video delivery services to receivers that are computers or between consumer devices connected in home networks or professional securitized cable-based communication systems. Yes, understand that the receiver is a professional device in a movie theater, or an internet-based system, etc.). The encoder also adds all the parameters that define the color conversion function F_L, for example as a SEI message or similar metadata. Or, conversely, the master LDR grading M_LDR can be transmitted with a function that allows the receiver to recalculate images with high dynamic range, such as M_HDR.

Thus, on the creation (or transcoding) side, for example, a pre-created master HDR image M_HDR, typically 5000 knits of PB_C, which may be done simultaneously on the same color classification device or another. Several colors, including a color conversion unit 202 configured to determine and apply various color conversion functions F_L as desired to obtain the master LDR appearance image M_LDR from (which may be predetermined in aspects). There is a grading device 201. As an example of image processing, the reader may refer to WO2014 / 056679, and the teachings of this application are incorporated herein by reference in their entirety in countries where they can. As an example of a transformation, one can imagine a function that does not affect chromaticity, but only affects pixel brightness (all discussions below focus on luminance conversion for ease of understanding). This bends the high brightness somewhat downwards and boosts the dark colors of the relative 0-1.0 HDR input (x-axis) to the function's LDR output plot, resulting in a relatively dark 100 knit display (r shape curve). ) To make it look bright enough. However, as long as it is possible to create two related optimal appearances of the LDR image and the HDR image, the brightness of some objects can be customized and potentially more complex functions that only locally fine-tune, and It is also possible to convert chromaticity planes such as saturation.

Here, the next thing that can be done with this coding technique is that an M_MDR intermediate dynamic range image with an actually connected peak brightness that has an accurately graded appearance for, for example, a PB_D = 1500 knit display 280 is required. This is a case where it is not necessary to actually recalculate the 5000 knit M_HDR image, and the M_MDR image accurately graded from the received M_HDR (or M_LDR in other embodiments / applications) image using this display tuning algorithm. Can be obtained. The receiving side is a classic, eg, HEVC composite that outputs a normal (eg, linear RGB or YCbCr) image M_HDR and a color conversion function (F_L or its inverse mirror function) required to change M_HDR to M_LDR. Includes vessel 260. Based on all this information, it is assumed that the color processing unit 270 (in this exemplary embodiment, for example, separate from the TV in the Settopbox, or the server of the video-on-demand provider, etc., but inside the TV The color tuning unit 271 in (may be) not only determines the optimal function F * _L required to determine the required M_MDR image from the received image, but also finally those intermediate dynamic range images. Output M_MDR. The display tuning algorithm for obtaining an appropriate F * _L function for the brightness conversion from SDR to MDR based on the brightness conversion function from F_L SDR to HDR is of any embodiment, for example, considering the details of the viewing environment. Although it can be very complex depending on whether the transformation is desirable, all these details are more than necessary to understand the present application, so the reader may read from any input encoded and MDR images. It can be noted that there is a luminance function F * _L applied to obtain, for example a gamma function.

With this mechanism, not only a single conceivable type of HDR display (eg everyone needs to buy a 1000 knit PB_D display, no one can buy a 3000 knit PB_D display, at least a color measurement system Optimal HDR image rendering is now possible for any possible display, but this additional technical expertise is fully flexible HDR. The processing ecosystem is still inadequate. The technologies summarized so far have been to capture a single HDR set of video images from a single shot HDR scene and accurately output them to various consumer sites, and on a single television available. The focus is only on aspects that render properly. This was a difficult task to prepare for the new HDR video technology, so further aspects were delayed and dealt with later.

However, video programming is more complicated today. Video (even consumers) on a simple portable device (even a sportsman who wants to get supply from different sites of all kinds and shoots action with a life news supply from a helicopter or a GoPro tied to a helmet) The content is easy to create and can be easily communicated via a means of communication such as the Internet. In fact, at least in theory, a big consumer deployment is still needed for takeoff, so the TV becomes like a computer with an internet connection and various data sources presented as visual information. It can be used or jointly supplied. Expected to have a variety of uses for video, such as playing a video game in a display-rendered area and watching a news program behind the scenes, for example, whether in the same display area or adjacent displays. There is.

This problem started in the LDR area, but all videos are the same Rec. It wasn't that difficult as it was defined in 709 colorimetry and was more or less similarly illuminated and exposed. And even computer graphics for games, for example, were defined as well. But today, HDR video or images can be anything you want to provide, and in the future if you don't have a good framework strategy for processing and synthesizing multiple image (partial) posts in particular. Can predict confusion. As mentioned above, previously all videos were defined in the same LDR framework, that is, all maximum Luma codes correspond to the same 100 knit rendering brightness, and the brightness corresponding to Luma below the maximum is a single existing Rec. .. It could be decoded by the means of 709 EOTF.

Since the camera may make some "minor" setting tweaks, it required some concerns and careful handling (especially in complex creations or external broadcasts and other field creations). What matters is typically white balance (ie, the neutral white chromatic aspect, that is, whether the white object is too bluish or yellowish, it switches between different camera feeds. It became noticeable) and the exposure setting. If you reduce the exposure of one of the cameras to shoot the same scene, the camera will be detected as soon as it gives an image that is too dark. Especially for professional multi-camera creation of the same show that stands out, moving to a live report in a news program will shoot different shots (especially if it's a video on an amateur cell phone) or at least shoot scenes that look completely different. Expect to be (whether it's timed or picture-in-picture) (for example, a night view of Cairo, not a well-lit US studio). Here, there are two situations to distinguish. Of course, when switching between a night scene and a desert noon scene, the viewer is a reasonable difference, for example, for a desert scene that does not surprise or squint the viewer, but on average. Predict the difference in the brightness of the scene or the brightness of the objects in the scene. However, on the other hand, the combination of camera exposure and the final brightness of the rendered display can be used to create any scene that looks like night or bright. In the SDR era, it could lead to strange images if the photographer had to make inconvenient exposure selections. For example, a person standing far from a window in a deep room will be much darker than one near the window. The photographer exposes the person in the dark area so that the person looks nice and colorful, the left side of the image and all the room objects there are clipped completely white, which is offensive to the viewer. I will show you a "half image". However, at least colorimetry was reasonably defined. We knew that such colors would come out, and that they were the same regardless of whether such images were mixed or rendered alone. In the HDR era, by making all areas codeable rather than clipping white, for example, exposure depends on how bright the bright parts of the room correspond to the brightness of the actor or presenter in a particular scene. I wanted to solve such a problem by lowering. However, it comes back nasty as a problem if you want to harmoniously mix the video content, so the techniques of the following embodiments are needed.

In addition, more complex cameras these days have additional settings, such as black gamma. It's not easy to match two different cameras exactly (for the same brand, you can at least switch off or set all similar "exotic" parameters), but at least it does. It was the job of a camera shader, an engineer (or technical director) who had a special eye for color differences and remotely controlled all of these color measurement settings. This allows the actual camera operator to focus on framing, which is a difficult task enough, especially in some creations such as actors flying around the scene. In any case, in the SDR framework, at least his subject matter is rational, coupled with a limited range of 100: 1 that corresponds roughly to various diffuse reflectances, even if it is a bit wrong. Misplaced compared to the brightness of other objects, with the fact that it was confirmed to be illuminated by (ie, at least 10% of white, or about 1 stop below mid-gray as an impression of luma or visual brightness). The inaccurate combination with the brightness of the object is generally not too inconsistent (at least, critical viewers feel something is wrong, for example black becomes a little milky gray. In some cases, nothing that can actually occur in HDR creation that actually looks strong and bright with a large difference in brightness).

However, in the emerging HDR era, there is no such single 0-100 knit LDR reference range, and cameras (and other systems) do not work relatively simply within that range (ie typically). Perform non-extreme color conversion). As can be seen from FIG. 1, even if one chooses to use only one of the HDR codecs, which is not clear but has potential in the future, there is still content defined in the LDR codec. And the difference between 5000 knits and 100 knits is so extreme that it's not a "minor" look change (which may actually only be noticed by experts), but it's very noticeable and potential. This is a very annoying change (if various devices such as televisions operate according to designer preferences along the vast brightness range of the vast color space of the future, the variability becomes even more extreme and undesirable. Is expected).

This problem is less noticeable when looking at a single program from a single source. For example, if someone is looking at HDR content on a high PB_D display in a dark environment, the colors may be a little too bright, but at least the viewer's eyes are adaptable. For example, if there is a criterion for judging an image by presenting PIP (picture-in-picture) of two videos, the difference is always very clear.

Images come from everywhere today, so not everything is under the control of the director who determines the appearance of the image and is not displayed by them in a controlled framework, but can potentially come together first on the display side (on the display side). There is an urgent need for a good framework for coordinating the composition of various original HDR images (eg a dedicated commercial is inserted).

Thus, as mentioned above, most of the (new) HDR technology focuses on making HDR image information totally communicable by communicating explicitly defined pixel luma that can be converted to rendered brightness. I'm guessing. If you have a display with a PB_D lower than the PB_C of the content, you need to do something because the brightness cannot be rendered to PB_C on that display. Linear compression (relatively white-to-white mapping, i.e. rendering pixels in an image with encoded luminance PB_C to actually render on a display with luminance PB_D) is possible, but the rendered image. May appear to be suboptimal (for example, some dark areas of the image may appear too dark and probably invisible even if action is taking place). Therefore, smarter brightness mapping can be attempted, for example using r-shape curves to keep dark brightness reasonably visible. For example, common gamut mapping techniques are possible inside a TV, but they are not strictly optimal as they are blind, i.e. not compatible with special HDR images.

US201550245004 teaches that two images can be mixed, at least one of which may be an HDR image and the other may be an SDR image. In this application, the width of the RGB triangle is the main essence of the color space mismatch (ie, because the other additive primary colors are used in the DCI P3 color definition rather than the usual video definition of sRGB. The color aspect is irrelevant to the description of the present application, as it can be selected in addition to the brightness of the image pixels and the corresponding dynamic range of the image coding), and the dynamic range mismatch. It states that there are both. Although this application does not elaborate much on the mechanics of video mixing, it states in (0029) that mixing can occur in the color space of the display device, i.e. the dynamic range ending in PB_D in the naming of this application. .. Furthermore, this process appears to be related to conversion to a different conversion function TF or gamma correction (0027). Finally, it corresponds to the fact that the compositing unit can evaluate the dynamic range of the video content and probably does not teach in more detail that statistical analysis of the input image can be done (0028). There is a teaching in (0036). Those skilled in the art will interpret this in the light of existing knowledge. There are actually teachings of SDR video optimization that examine the statistical properties of the input video, i.e. the end points of the luminance histogram, and optimize the histogram for a particularly available display range. For example, for a variety of reasons, it is unusual for black or darkest pixels in a captured image to be brighter than they are (or probably must be), for example, 30 Luma instead of 0. No (again ignore complex details about overflow protection coverage, display light leaks, lighting-dependent visibility issues, etc.). The histogram can then be stretched to minimize the image Luma, which is 0 instead of 30, which improves the contrast of the image structure and gives a better black, resulting in a clean image. Especially suitable for dark surround viewing.

In practice, if it is necessary to design a video mixer without being taught further invented details, one of ordinary skill in the art may have two options to consider as described in FIG. Apart from mixing Lumacode, as illustrated in Figure 1, this is nice and simple for pure SDR image mixing, but not so obvious or smart when the images are defined separately. Absent). If the teaching says that it needs to be transformed for a different transformation function, those skilled in the art will consider a naturally designed mixing system in FIG. 13a. In this mixture, it has a sufficiently large range of natural brightness (1301), eg, up to 50,000 knits, and in fact converts them to brightness whatever the code-defined OETF shape. If both images contain brightness (physically uniquely defined), it is of course possible to mix them in principle, but with practical psychological visual and artistic considerations. And the question is whether it is enough. Second, for example when mapping to a common preset display dynamic range (1302) of 700 knit PB_D [skip OETF steps and pure reasons for corresponding brightness for simplification], then: Another possible natural mapping (such as the relative mapping paradigm) adjusts the image to the capabilities of the display (based on the "statistical properties of the image") (ie, adjusts the dynamic range of the image of the display. Toward the dynamic range). A natural way to do this is, for example, to linearly extend the range (ie, correspond to a white-to-white mapping). It is possible to apply some gamma shape mapping functions, but the question is what the gamma coefficient should be and why. However, this has already produced results that we want to avoid exactly in some cases. For example, if you stretch an SDR image of a snowy landscape to 700 knits, it becomes annoyingly bright.

Moreover, although these natural mapping systems are not perfect only from endpoint considerations, such as where the brightest colors should be mapped, such techniques are probably the most important aspect of HDR, as mentioned above. That is, it completely ignores the brightness distribution of all objects between endpoints with a (precisely controlled) dynamic range (currently this point still seems to be unknown to many and has a low dynamic range. Images that are rendered correctly on the display often look better than poorly controlled images on high DR displays, such as contrast and HDR effects, but HDR is simply white brightness, if not very expert. It seems to claim that any technique can be reached by dividing the value by black). By focusing on the example of HDR master grading snow with PB_C = 5000 knits, it is easy to understand that the dynamic range is not just about technical endpoints. A grader, for example, maps the brightest part of the snow under the sun to 800 knits, but another grader may make only 550 knits, so most of the codec's capabilities are not used for this image. So this looks like a situation with a low dynamic range in nature. SDR grading usually assumes that the grader has 100 knits for the brightest pixels of the snow. With MDR graded images, it is possible to consider what brightness the brightest pixels should be, even when rendering only one image, rather than a possibly very complex mixture of different image content. So how do snow pixels know the final (mixed) brightness to be finally rendered, whether it comes from an SDR image, it is relatively encoded up to 1.0 Even if the OETF shape is ignored, the snow pixels are seemingly 0, for example, as they may not be the most important determinants, whether they are or come from "some" HDR image coding. Appears somewhat arbitrarily with a relative brightness of .126.

FIG. 14 shows a method that simply focuses on the dynamic range of any technical system, eg, how the blind brightness conversion of Gamat mapping works, the coding is primarily defined by the endpoint. It reveals another important point that is overlooked in whether it corresponds to a theoretical reference display or an actual display as a gamut. Especially in HDR, the lighting of objects does not need to be tightly controlled, for example in an SDR news studio, but in principle anything can be used. If you walk past an empty space in the forest, for example in the sun in a grassland area, you can visually recognize that the interior of the forest already looks black. That is, of course, not at all, because it is black and the trees there are brown like anywhere else, and the bushes are like fresh green, but the local lighting is the full light of the sun and the sky. Since it is, for example, 1/20 or less than the local illumination of a free spot without trees, 1/20 looks psychologically black to humans until viewed closer.

Therefore, objects such as T-shirts can have different brightness even in the same movie (in other words, it can fall anywhere in the range of 1000 knit PB_C_HDR, for example, graders are master graded. It can be decided to limit the variation somewhat, but it will be explained using the natural approximately one-to-one relationship with the relative brightness taken by the camera in the original scene). In the sun, the T-shirt is + -5 times brighter than the starting penumbra, which is mostly illuminated by the considerable solid angle of the blue sky. Indeed, 100 knits are artistically expected to render to a good HDR display on a "cloudy day", and 500 knits may appear in the real sunlight. Furthermore, as a person enters the forest, the area of the sky that can shed light on that person is blocked by the leaves, so that the brightness of a particular object, in this case a person's T-shirt, will be, for example, 100 knits. It is therefore 50 times darker than a bright T-shirt and is therefore likely to appear slightly black in the rendering of this particular HDR scene. Perhaps in cinematic HDR grading for home viewing, brightness is conservatively graded, for example, to half a given value, but the various brightness distributions prove the same.

Therefore, the problem now is that using any of the above relatively static mixing aspects will scatter the rendering of these various T-shirts (along with their brightness details) in the running video. This is always "automatically" reasonably good, and if not, what to do.

The above problem, which requires a method of reasonably adjusting the composition of images with (potentially very) different dynamic ranges, is two images or two videos of multiple images (Im_HDR, Im_LDR). It can be solved by having a device (301) for synthesizing two images or two videos, one of which is a high dynamic range image or video. A device (301) for synthesizing two images or two videos of two images or two videos of a plurality of images (Im_HDR, Im_LDR), one of which is a high dynamic range image or video. ) Is
The maximum brightness of two of the images or at least one of the two videos or video, and the peak brightness of the display for rendering the two images or two videos of the multiple images. A dynamic range establishment unit (302) that establishes a combined luminance dynamic range (CombRng) characterized by at least a maximum luminance (LMC) determined based on at least one of the above, wherein the dynamic range establishing unit is a composite. A dynamic range establishment unit (302) further including a luminance anchor determining unit (303) that determines the anchor luminance (anc) in the luminance dynamic range (CombRng).
A color conversion unit (310) that performs at least a luminance conversion on at least one of two images or videos, wherein the color conversion unit (310) converts the first image or video (Im1_LDR) of the two images or videos. The color conversion unit includes a source anchor luminance readout unit (311) that reads at least one source anchor luminance (L_SA1) from the first source (350) to be supplied, and the color conversion unit is a color conversion applied to the first image or video. FF_1) is established and its color conversion depends on the value of the source anchor luminance (L_SA1) by having the property that the source anchor luminance (L_SA1) is mapped to the output luminance in the vicinity of the anchor luminance (anc). , Color conversion unit (310),
It includes an image compositing unit (320) that synthesizes two images or two videos of a plurality of images to form at least one output image (Im_o). First, to avoid doubt, rather than compositing images, it means various possible options for organizing image content either spatially or perhaps temporally in the case of video. Compositing must be understood in the broad Latin original sense of "combining the two", and in this text we are taking to weight add two sources per pixel, for example a limited mix. Don't understand in a sense. To those skilled in the art, various embodiments of the invention are particularly good (harmonious or coordinated matching of brightness) composition of video resulting from the composition of various picture-in-pictures, composition of mixed mixture such as fades, or Understand that it solves the need for temporal clutter, such as regular cuts into commercials in a movie (generally, this is now the first, for example, as if it were a page. The amount of change over time corresponding to flipping the image to display the pixels of the underlying image, including spatially interleaving at least some pixels of the first image into part of the second image. However, the present invention ensures that pixel colors, especially their perceived brightness, correspond exactly). Both when the spatial region of the image is replaced with another (part or all) second image, and when the image of another video sequence is shown temporally between or after the image of the first video. There should be no too much lightness mismatch, especially if there is flashing content etc. (ie, one of the images should not appear too brighter than the other, for example with respect to its main histogram lobe or key object. ). In such a scenario, the composite brightness dynamic range (CombRng), which is the dynamic range of the composite image (in the representation of the sky), can be established to be filled with the composite image data (eg, temporally scattered or continuous). Can be considered as the two maximum ranges to do).

To those of us, what is a high dynamic range image, that is, an image that is not a legacy LDR image defined for a peak brightness of 100 knits, that is, an image that cannot be combined by the classical LDR method? Obviously (there seems to be some discrepancy between image objects or pixel brightness defined by two different image coding frameworks). It will be apparent to those skilled in the art that if the device obtains only two still images (or, eg, 5), the output image (Im_o) will typically be one still image. If two or more videos of successive images are combined, an output video, i.e. some form of sequence of those images, is generated, for example if each video has an image at each time, the resulting video will be the original. The two image sequences may be alpha-blended mixed videos that have exactly the same amount of images at the corresponding times, or the resulting video may have images at different times, eg, the first. It has 100% length of the video and + 50% length of the second video, and its various images are mixed and assigned at several times.

In order to keep the teachings of this patent readable and concise, the teachings give top priority to how to handle differences in brightness dynamic range and the impression of brightness that it gives the viewer, so I dare to simply say An example of performing the conversion is shown (the reader can pretend that the image is a grayscale image). However, of course, those skilled in the art will understand that in practice color conversion is typically performed, such as how luminance corresponds to a linear RGB color definition, and linear RGB, such as WO2014 / 056679. You can understand both how to produce pixels with the desired brightness by converting the color on top. When saying something about video for brevity below, one of ordinary skill in the art understands that it can also be applied to still images. In some cases, from a real-time perspective, video may want a simpler mode of processing, which is why an elegant and quick solution for mixing is offered. Better execution of embodiments may work with the intervention of a human artist, as humans can best determine what the HDR image is exactly, what is needed for various mixing and presentation. However, in other embodiments, it operates with automatic annotation by an image analysis algorithm.

The device needs to be able to establish an optimal composite brightness dynamic range, which is a new dynamic range that is part of the reason why very complex composite problems become more manageable. As explained below, this simply does not have to be one of the dynamic ranges already present, i.e. already invented and defined, i.e. not necessarily the dynamic range of the source image, but the intended display of the content. (Some creator devices can have reasonable estimates of what a good reference display is on a consumer's premises, where the expected display in the field is typical. Nor is it the dynamic range of the final display to be rendered (which may not be apparent in that embodiment) (even if it is known during compositing, this is not true for all embodiments). Its composite brightness dynamic range should ensure that the composite presentation has the best visual quality or impact on the viewer. It, of course, depends on various aspects of the defined content. If the main content (referred to as a second image or video because the claims are easier to read) is, for example, a movie, the filmmaker does not like, for example, disturbing the movie. For example, after a bright commercial, the viewer should not lose track of what is happening in the dark scenes of the movie. In such a scenario, synthetic video, cinema is therefore the most important (or) because typically one of the two videos will have a distorted luminance characteristic and will be more in harmony with the input video of the other. It is the main (major) video and needs to be (if possible) not significantly distorted in brightness. However, commercial producers and sponsors also want to see the commercials come out as best as possible in the final render. The color conversion unit (310) and / or the compositing unit (320) enforce a policy that commercials cannot cross movies, but for example LDR commercials must also not be rendered unnecessarily dark. For example, the face may not look very clean and look almost black compared to other related objects in the movie. As mentioned above, when the human visual system adapts the interpretation of gray values in its bright areas, forests can appear black compared to areas of very bright sunlight. Therefore, in an HDR movie or broadcast (or game, etc.), there is a very bright, eg 2000 knit, important (large) area, especially if it is in close proximity to synthetic SDR pixels, an SDR of only about 20 knits. The object looks black, so it is desirable to make it somewhat brighter before compositing. Also, for example, the visibility of facial features or details of other objects decreases with the average brightness and contrast of the objects. I've experienced some people complain that the brightness distribution of an object (eg, a pixel brightness histogram for HDR rendering) isn't perfectly optimal, even for a single image, in some situations, too. In too bright scenarios, visual adaptation and the human brain can, when a single image is rendered, allow the brain to determine and adjust what all objects in that single image should be. Therefore, it is usually relatively tolerant. However, if the object is spatially composed of different images, the viewer will notice that the difference is significant even if the content is within the frame. This is because the brain determines a part of the content of the image as a reference for the brightness distribution of the scene, and the scene is usually not so separated as PIP, but rather has a smooth illumination change. As shown, it is not easy to tell at a glance which is the better lightness composition. For example, a movie creator can see which local commercial is inserted (eg by a cable operator) or while watching a movie (perhaps streamed by a home computer or a central home appliance that manages content). You can't do that, at least while creating separate videos, because you don't know what second image to choose from several image sources, for example PIP. It should also be noted that, for example, the visual importance of a relatively bright object depends, among other things, on its size. By making the two images available, the device can make smarter compositing decisions. For example, if it is an SDR image, it may be necessary to reduce the maximum renderable brightness (LMC), for example, to make the HDR content more like the first image. This relies on seeing additional properties of pixel or object brightness, where an object is a smartly grouped set of semantically related pixels. For example, if the brightest object is a lamp, the brightness of the lamp within the composite dynamic range (CombRng) can be reduced for some time. This is not the most spectacular HDR, but HDR images can still be used to render lamps clean and bright, especially for connected displays with lower peak brightness (LM_MDR) than composite images (LMC). In any case, a very bright encodeable lamp cannot be rendered accurately with the brightness of the codec. Therefore, there are still movies with a high dynamic range, but there is a dynamic range that is more suitable for the composition situation.

In fact, two typical alternative scenarios can be considered. In general, the composite dynamic range, or at least its top brightness (LMC), is an assumption that, on the one hand, the input needs of at least two images, and on the other hand, the composite should be rendered if something in the rendering has already been identified. It is defined by a device that sees the dynamic range (at least peak brightness) needs of the displayed display (ie, the HDR needs of properly presented image content in a typical HDR display). Therefore, in some embodiments, if nothing has been said or is so wise about how to adjust for their use, the CombRng will be based on only two images. It needs to be determined (however, in some embodiments it may have already been tuned for the expected typical use, for example a master HDR image of PB_C = 5000 knit has a lower dynamic range primarily. If it is shown on the display and is expected to be significantly scattered in the SDR image material, for example with a dynamic range lower than PB_C = 5000 knits for communication video compression, eg the typical consumer side expected. To create an image that can be combined or combined at 1.5 times the peak brightness of, for example, 1000 knits on the display (meaning that some image information still needs to be converted in brightness on the receiving side). Think, therefore the synthetic dynamic range has, for example, 1500 knits of PB_C). The first typical example then has a larger dynamic range of the HDR movie, i.e. the two input videos of the continuous image, and the SDR content reasonably harmonizes with the main content HDR movie ( It is appropriately annotated by being brightness-converted or by placing at least one or more source anchor values in the co-communicated metadata so that it can be harmonized in the final presentation. However, an alternative typical scenario can pre-convert HDR content to a lower dynamic range that is more compliant with either of the typical display rendering and / or SDR content insertion envisioned. For example, if you have a real display for rendering 750 knit PB_D, such a system is, for example, both videos (which can occur at completely different production sites and times, and is a consistent anchor force. ) Can be converted (or annotated with anchor brightness to determine the brightness conversion) and harmonized with a synthetic DR of, for example, 1500 or 2000 knits of PB_C or perhaps 1000 knits of PB_C. This synthesized or compositable content can be easily downgraded to the display dynamic range defined by the required PB_D = 750 knit.

In particular, the applicant is not actually as an actual encoded image (ie, having a final and fixedly defined pixel color in the color components of a typical DCT compressed image), but any primary image. Designed a framework in which (various) final (re-graded) different dynamic range appearance images are communicated as a function for calculating the various possible pixel brightness of such secondary images from. So these functions can be redefined and thus convey more complex definitions (eg, image brightness for at least some special image objects, in different situations, eg different synthetic dynamics). Details of how it should be calculated under the range). Classic image or video coding simply encodes and communicates the "that" image with pixel brightness to be used, ie rendered (!), But the applicant is of a completely different image (ie, of the object). It is possible to send (with the required spatial structure and texture but not the correct color measurement, especially not the exact image object or pixel brightness), and then in this application the correct final image, which is a composite image. The formulation of how the brightness is ultimately produced is defined by all pixel brightness over the synthetic brightness dynamic range CombRng.

For example, by compositing a relatively bright LDR image (for example, if it is not darkened very accurately), the overall (or local) brightness of the composited output image at least in the dark part of the HDR image is the HDR image. Similarly, the minimum value (LmiH) of one of the images, which is a dynamic range image (of dynamic range Im2_Rng), is also compared, since the dark part that comes exclusively from is not so well visible. There may be a reason to increase the minimum value (LmiC) of the synthesis range. That is, they can be properly brightened by color-converting them to CombRng with bright black (this is caused by the choice of compositing range, and the HDR image itself does not need to be processed smartly, ie it is. Even a simple color conversion to CombRng will brighten properly).

Thus, the reader is at various points in the image processing chain, eg, at some content creators whose final content is defined by a device typically under the control of the content supplier, of the image consumption image chain. Before entering some or some image consumers (eg, only at the end user's responsibility, eg, the end user mixes the two types of content using their remote or computer controls). It is possible to understand how the appropriate synthetic dynamic range is potentially semi-automatically or automatically established by humans. In general, such a device is an image content (ie HDR quality, that is, what PB_C the various contents have, various objects on the dynamic range ending in PB_C, or the distribution of pixel brightness, any bright object [ Whether there is an HDR effect, for example, or whether brightness distortion can easily occur without making very unpleasant changes to the appearance), as well as the need for final use of these images, ie typically display rendering (typically display rendering). For example, if the image is fed to the installed base of the SDR display consumer, a low CombRng would be appropriate). When done automatically, the algorithm produces what the human combiner uses as a methodology (ie, a composited image and / or metadata for it, eg, at least two images or videos that are finally composited. Use a heuristic model that resembles one source anchor point (specifying an LMC value and at least one anchor point). On the dark side of CombRng, on sides such as ambient light dependent masking of dark colors due to, for example, glare from the brightest areas of bright image objects, expected display leakage, and typical display surface plate reflections. By modeling such visibility on the basis, it is determined that the darkest colors are still relatively well visible. A new element of this model compared to the typical (eg home cinema with dim lights) viewing environment expected to be the only display is the introduction of secondary image content. However, this estimate can be rough (eg, the brightest, potentially bright, part of the SDR image, as the establishment of CombRng is typically done first (before determining the appropriate anchor). Even does not accurately check where the SDR content is introduced, whether or not it falls geometrically next to the dark part of the movie, and therefore some embodiments have a CombRng upper brightness limit and All of them can be taken into account when establishing the lower limit brightness, but can potentially be quite masked, for example in direct or iterative decisions, the latter is what a good anchor brightness is. And the corresponding appropriate CombRng limit brightness is repeatedly checked).

Coarse compositing may already be suitable for a variety of purposes (for example, if a movie viewer wants to enjoy the movie in the end, it should not be interrupted or composited with additional images. If so, even if it is only psychologically and visually in the evaluation of the overall image impression of the composite image and there is actually no change in the brightness of the movie compared to what it was before the composite. It should be recognized that there is always distortion in the colorimetric measurement of that major video content), which may already be appropriate for quickly calculating a heuristic low brightness LmiC based on typical content. For example, if the histogram shows that most of the SDR content is between 10 knits and 100 knits, and about half of the display area of the composite image is occupied by SDR pixels with a brightness of 50 knits or more. Can establish that it does not require deeper black than 0.1 knit, for example 0.01 knit, for its composition (a typical heuristic algorithm for displays is such that darker image structures are such. To judge that it does not look good under the conditions). What makes this establishment of the compositing range very interesting is (for the original content creator of HDR movies, if they enjoy the movie on an OLED display that can render, for example, very deep black in a dark room. Although you may be making a movie with a defined ultra-black up to 0.0001 knit, HDR before the synthesizer (eg STB or TV itself) places HDR movie pixels in the composite image. Some of the darkest colors of HDR movies by determining the appropriate brightness mapping conditioned between the movie (or generally the main content) DR and the composite dynamic range, especially the lower parts of these dynamic ranges. It is possible to decide to make it brighter. On the bright side of the brightness range, the algorithm usually involves determining how much clipping or contrast compression is still appropriate for the original HDR image (this includes, for example, for sunlit clouds). Unlike lamps that do not need to have a rendered internal structure, [ideally for encoded brightness, so that the determination of the final image is reasonably simple based on the encoded received image. Ideally, the final rendered image should have sufficient contrast and any future image coding so that the final rendered brightness of all pixels is determined. Therefore, less than the color of the brightest image, whether it is already present in the composite image or a pre-synthesized HDR image before it is actually converted to composite image pixels. Compress to define a bright area with a larger subset of Lumacode). If necessary, inserted, for example, should not look annoyingly dark (for the expected filmmaker, for example, knowing the film on a particular channel, therefore this potentially regrade of the film (Separated definitions are provided with the commercial) In commercials, at least the SDR image is converted for compositing, so the cloudiness is reduced in a slightly smaller area of bright brightness that is not too far from the brightest brightness of the SDR image. It is necessary. This image need remains most appropriate (generally, that is, at least part of the SDR image, and both some important parts or objects of the HDR image) and, for example, for various introduced SDR commercials. Ideally, the HDR image would specify a number of important brightness, eg, three for those clouds, among which the synthesizer would have different cloud regions to be able to tune. Can typically be changed to darker (which also reduces the contrast of the dark gray areas of the thundercloud, for example). However, this is where the brightness of the anchors appears as taught here, and the current part of the teaching only explains that the synthetic range naturally does not end with such a low LMC value, which is the main (For example, if the movie is about a thundercloud, if part of the cloud is strongly illuminated, for example, the edge of the sunlit cloud and the bulge of the cloud will be illuminated by the sun. Illuminated only by ambient light, they typically have a nice different gray, so they are much darker and CombRng so that these clouds must be compressed with only 10 Luma, for example, in the upper subrange of CombRng. If this is the case, the core aspect of this movie is lost, as using strong functional brightness extension as a display correction attempt will never give good HDR renderings worth these clouds. Will be). However, if not, the synthesizer decides to reduce the quality of the brightness and contrast of the HDR image somewhat (especially, for example, some content is in the sun and some content is in the shadows). (From a less demanding HDR scene, such as a soccer match) HDR images can be harmonized with typical or actual lower dynamic range images to be combined.

Of course, the dynamic range (at least the maximum, and in some cases the precisely specified nonzero minimum) determines the (launch) characteristics that are very important to the final appearance of the image. It does not completely complete how the lightness of an object or pixel is placed within its range. Simple image processing methods actually claim that they need upper and lower dynamic ranges and do the mapping (which is often less wise than simple linear compression, i.e. white input content white. Map to the final range of black and map the black input content to the final range of black). It can vary considerably, especially when you start using potential with a large dynamic range in a very critical scene, seeing outside through many deep blacks at the same time, and small cracks, for example in a cave. Both of the many sunshine colors that are used, and even objects that require significant control of contrast within an object, such as a person passing through fog, need expressive brightness. However, as mentioned above, the inventor has found that good control of the appearance of HDR images, especially control of the appearance of composite images, is good for all or at least more of the brightness during the range of CombRng, or at least the most important one. It is importantly related to control and is therefore a good HDR image processing system, especially a device that can produce a final composite image of the proper appearance without further difficult human intervention for any situation that may actually occur. I think that something else is needed to have (because it is not possible to sell the same Hollywood director on each STB or TV to determine the final composite brightness that is appropriate for the viewer). In order to obtain reasonably high quality results, at least in the most practical cases of image content composition, there is a need for a quick mechanism that can be successfully tailored by the content creator, with or without critical or complex needs. is there.

Therefore, a second component for adequately controlling the composition is that the device determines the anchor brightness (anc). This can be a variety of things, but is typically the semantically relevant brightness of a semantically important object. For example, it is a particularly typically important object that is illuminated in a particular mode, giving brightness to a subrange of the entire HDR image range. It is the brightness that can be expected to be good in some synthesis range to determine the other brightness around it (ie, is this brightness the sum of the brightness is adequate brightness or is it too dark? , Or is it too bright). In other words, whatever is correct for a particular compositing range, if the anchor's brightness is rendered correctly, the other brightness will not be bad (in Figure 16, how more technical configurations are made according to the wishes of the creator Explain how you can control them with elements and get better).

FIG. 15 shows a typical HDR scene with two regions of significantly different lighting: a building (Ins) with dim lighting (see the geometry of the scene in FIG. 15a) and a sunny outer region (Outs) with fairly bright lighting. Is shown. In order for the content grader to create an HDR image that looks good for typical television viewing (dim environment) from physical scene brightness, it is necessary to grade the indoor brightness like an SDR image, ie indoors. It is wise to grade up to 100 knits for the brightest objects in. The outdoor brightness in the real world is about 100 times brighter, up to 10,000 nits, but it's for TV rendering (even if the viewer typically has such a high-end 10,000 nit PB_D display). Has the experience of being too bright. Thus, the grader will perform this scene of HDR, for example, in a master HDR range (DRH) of up to 1200 knits for the brightest object (excluding small specular reflections on metal) and 250 knits for typical average outside brightness It can be selected for bright outer brightness (in the outer pixel brightness subrange HDROU).

The power with anchor brightness is shown in FIG. 15c. This example simply assumes that the darker brightness of the inner pixel brightness subrange SDRIN can be rendered with equal brightness under all circumstances (forming a stable, unchanging set of brightness in the composite range). But, of course, it doesn't have to be always the case, for example, if the SDR commercial contains many bright colors, the device is actually the darkest, using the lower anchor point AncS_ins, which characterizes the internal pixels of the HDR master source image material. The pixels can be raised somewhat, but at least the pixel brightness of this first master HDR image (ie, along with the second anchor point AncS_outs of a set of bright outdoors) is accurately placed in the composite range 1 The core principles of the embodiment will be described with a scenario in which only one anchor point is used). First, when any brightness of an interesting object with a typical light reflection spectrum leading to the appearance of a typical gray value is used, but one wishes to determine the brightness of the pixels around such anc value. In addition, it is usually not bad to use average brightness anchor positions in the sublobes of all image brightness histograms that correspond to reasonably similar areas of illumination. The reader, referring to FIG. 16, if for some reason the device wishes to "relight" such an area with, for example, more light, the actual reflective object will have the illumination illuminating it. It can be imagined that the brightness around it becomes continuously brighter at the anchor point, so that it shines more as it increases. In the case of automatic determination when calculating a typical anc value, care should be taken not to have too much deviation from the reasonable average, for example the brightness of the cloud edge in very bright sunlight, and therefore. It should be noted that the best results can be obtained, typically if humans can determine and convey the anchor brightness along with the encoded image. Thus, the reader may read that there are two types of anc values, i.e. the brightness of the image corresponding to the same semantic meaning of the anc definition is close to such anc brightness value (ie, illuminated differently). A position in the composite range that will eventually fall or be written to the composite image when rendered in a differently encoded source dynamic range), and a second (correspondence) for all or at least most of the input image. AncS value (if some images lack an annotated anc, the device must estimate a value, eg, an SDR value that should not be too unreasonable for the most typical SDR content). It should be understood that there is. If such an anc value is defined (even if it is semantically different from one of the composites, it can be associated with one of the composites, for example by multiplying it by a factor of 4), the input content. Can be harmonized with the compositing framework and, therefore, with all other image content.

The reader has determined the composition of the anc value (ie, the anc value of the result of the image composition, and the range prepared prior to the actual mixing of the two images), for example for a synthetic PB_CS = 2000 knit with excellent HDR quality. You can see the brightness dynamic range, or typically at least its peak brightness and how to adjust it. When an excellent quality composite range (CmbRngSup) is selected by the dynamic range establishment unit (302), a high quality dynamic range close to the dynamic range of the master HDR content becomes available. Therefore, it also makes sense to place the resulting ank value (of the semantic type "average brightness, second sublobe, position where the outdoor color should fall in the composite image") at 250 knits. (The device followed typical rules for bright area rendering of television movies, which was also the basis that led to the (approximately) 250 knit selection of the master grader for master HDR images). In this compositing situation, most of the associated HDR brightness of the external pixels is rendered well as the content creator intended, i.e. with the same brightness as encoded in the master HDR input image, and only specular reflection, for example. The result is that it needs to be somewhat dimmed to less than 2000 knits. This is an example of the first type of scenario described above, as the compositing range follows the original master HDR video content that is guided and effectively rendered without luminance distortion, so it is only necessary to harmonize the SDR content ( This is done by examining the corresponding one or more anc values in the SDR image).

For inferior quality composite ranges (ComRngInf), the device can already take into account the required quality degradation for HDR master images as well. Here, the device needs to lower the position of the Anc_Outs value, which is typically done, for example, with the need for heuristic estimates of contrast between and within regions (without looking at specific images and scenes). Several global values have been determined for good anchor brightness positions, which should work consistently for various HDR shots of the movie in the future). The second result, the anc position Anc_Outs2, should depend on which range amount is above 100 knits, or in other words above the low brightness of the SDRIN placed at the brightness position in CombRng. (For example, at stops, 10x is 3 stops to locate all HDR effects, whatever they are, not only reflective objects in bright sunny lighting, but also lamps or specular reflections, typically It also takes into account the values that the customer wants to see for a high quality HDR image, i.e. large object areas that are neither too bright nor too dark), so the bright areas or areas are well contrasted (in fact) better than 100 knits. For example, a coefficient of 5-10 is a good value to have a tanned appearance, which can be formulated as the division of two anc values), small specular regions such as objects, flames, explosions, or laser beams, and movies. There is still ample room for bright HDR effects, such as those actually included in and rendered near the upper limit of 1000 knits of PB_C_Inf. Some embodiments of the automated method work by calculating error estimates. For example, textures within a region can be analyzed with complexity indicating the need for higher or lower intra-regional contrast.

So, for example, a movie produced by a movie maker (annotated with at least one anc value and probably already brightness-mapped to a typical suitable CombRng) and a typical average variant of expected SDR content. If only two images are known, the apparatus shall at least calculate the error for luminance distortion compression of the HDR image and distortion or inconsistency of the SDR image in the pre-composite rebrightening, as shown in FIG. Can be done. Such distortion errors are PB_Co1 when applying the actual (optimal) brightness mapping to map all object or region brightness from the original image representation and its dynamic range to the recolored image in the composite range. When downgrading to CombRng with PB_Co2 etc. is needed and / or taking into account the actual image ie the remaining struggle, there is a probability of pixel brightness for, for example, one or more typical HDR scenes. You can then calculate for a typical average scenario that evaluates how such a scene needs to be distorted (mainly in the brightest areas of the image). FIG. 17 schematically shows how error E_cmp can be associated with compression of at least one portion or object of, for example, an image of a cloud structure. Various practical embodiments that mimic human grader behavior, for example, to smaller subranges by extracting textures and calculating texture measurements and other spatial and / or statistical measurements of local areas. Estimate how downgrading is a problem. In the case of clouds, for example, the texture analyzer finds that there are no sharp boundaries, and cloud appreciation is mainly caused by the (smooth) distribution of (many) gray values rather than the sides of a typical geometry. (For example, a linear pattern of Scottish quilt textures allows for more compression or posterization with less Lumacode). That is, when many finely graded and smoothly distributed gray values are mapped to a more limited set and inaccurate brightness, some errors quickly prevent the device from doing this too extensively. Expecting to occur, especially if the main image has important colorimetric measurement importance, for example, because consumers must be able to enjoy the finely graded artistic color quality of the movie, for example. Some commercials should not be reduced to the often extreme and flashy coarse grading (in addition to the main concept of providing anchor brightness, in some embodiments the content creator is between or around the anchors. It allows further metadata to indicate how much distortion is tolerated, for example the range between AncS_outs / 10 and AncS_outs * 10 should preferably not be compressed or decompressed by, for example, a factor 2 or 3).

FIG. 17 shows an example for more specifically explaining an aspect of formulating an error in order to reach luminance harmony in one embodiment based on such an error equation. The HDR building image and the SDR commercial of the soft drink are mixed, and in addition to the range, a histogram of the brightness distribution of the scene image is shown (hist_HDR is a histogram of the low dynamic range commercial image hist_SDR, respectively, and the horizontal direction is a certain value Li. Count N (Li) of the number of pixels in the total number of image pixels with equal brightness). For example, compression of HDR images in the upper subrange with a small CombRng up to 1000 knits of PB_Comb leads to error E_cmp. This error can be balanced with, for example, an error for brightening SDR content or an error corresponding to a CombRng incompatibility in a typical range of expected displays. For example, assuming that everyone in the world has a 1000 knit PB_D display, this error E-cmp can be normalized to zero because changes in the master content need to occur anyway. However, nonetheless, for example, 1500 knit CombRng has a negative error indicating that it is better, i.e., higher quality than 1000 knit synthesis when the visual quality of the movie content is very high and important. Aside from errors that can be calculated outside the brightness of the SDR content, this is less important (because in the SDR paradigm on the one hand the viewer should be able to quickly adapt to different brightness, but on the other hand. Potentially "wrongly" rendered SDR image brightness-for example for this special image many professional or non-professional viewing when rendered on an HDR monitor with the brightest pixels of 1000 knits, for example (Because it is determined by the person to be too bright-because it is determined in relation to the master HDR image rather than itself), the following error can be evaluated. An SDR brightness mismatch error (E_MM) can be determined, which is where the Anc_HDR value of CombRng, ideally the source image anc value (AncS_outs) of the HDR image, is mapped and the brightness of the SDR source anchor brightness AncB_SDR. It is done as simply as the difference between the location actually mapped to (D_AncSDR). That is, in this example, if D_AncSDR is equal to Anc_HDR, the best harmonization occurs according to heuristics. This works because this type of HDR image contains a sufficiently wide range of bright outer pixels (because the content creator defined the upper anchor brightness AncS_outs). Since viewers are already accustomed to bright brightness when watching this HDR movie scene, introducing nearly equally bright pixels for SDR commercials (eg PIP) is less uncomfortable, for example in a cave or basement. 5 minutes When the user is only looking at dark brightness (in this case, the introduction of such bright commercials will probably surprise the viewer, or at least chromatically undermine the rest of the enjoyment of the movie scene, at least considerably. ). This system is particularly suitable when the creator of SDR content uses a particular type of anchor, i.e., when it is as simple as using only two types, above average and below average brightness. Regardless of how the brightness is actually distributed in the 0.1-100 knit range of SDR brightness, the author decides what SDR image it is and therefore how to render it. In particular, according to the present invention, it is possible to show how to harmonize with various possible subranges of HDR images. For example, the SDR image may be the normal content of a cloudy or dark scene (“subaverage brightness”), or a luma that appears to represent a sunny scene. Assuming that the commercial makers want the colors to appear bright and vibrant as "sunny" scenes, it depends on what the colors of the other images are, as well as the bright and vibrant ones in the composition. ..

The reader can already understand the power of anchor brightness in adjusting the image to be composited. Even with such a simple system, it is assumed that the creator of the commercial content is interested in rendering his image "bright enough" (however, he probably is the whole image). You wouldn't be interested in annoying the viewer with some of his overly bright renderings of the content, of course, not so much in HDR filmmakers (many, equivalent, or in some scenarios). ) There is also an important image quality requirement, and compositing can work immediately and rationally if the HDR movie has only "dark" content and the HDR content creator contains only the lower AncS_ins. The device at least understands that this scene in the movie does not have (too many important) bright areas, so the viewer, for example, only the SDR subrange SDRIN in this non-limiting brief description example. Adapts to darker content. Thus, the device can make a harmonious composition if it maps AncB_SDR to low Anc_ins or its vicinity (even if it is "bright SDR content") in that situation. A neighborhood is a general customizable general multiplication of anc values above or below anc values that are fixed within the device or, for example, between 1/3 or 1/2 and 2 or 3 times anc, respectively. Means a ratio. The neighborhood range is the permissible brightness error of the object, the difference in anchor type, for example when mapped (using, for example, an explicitly communicated neighborhood brightness boundary) during execution by the synthesizer or execution by the creator. Alternatively, it can be determined based on various embodiments of aspects such as the expected variability of the anchor. Obviously, however, if AncB_SDR is, for example, 20% of 100 knits and is mapped to 3x25 knits (ie, the upper limit position near Anc_ins), the composition is that the SDR content is blatantly white to white or 100 knits to 1000. It maps to knits, renders the average AncB_SDR to 200 knits (!), And is much more harmonious than in the case of linear stretch, where all SDR image objects can appear too bright to like (perfectness). Rather, a system that works reasonably in a realistic manner balances the complexity and visual accuracy of realizing a real system, as it usually does not need to be achieved) (in principle, the actual histogram brightness, eg, It is possible to specify anc under the mode, i.e. the most occurring brightness value of the histogram, but keep in mind that the actual brightness is overly bright and renders as well).

Returning to FIG. 15b, it is interesting to show how their placement of anchors and any CombRng at different relative positions is related to (affects) various luminance mappings. The function TL_CS is a luminance mapping for mapping the original master HDR image luminance to the luminance of CombRng (ie, ready to perform image compositing such as pixel replacement, alpha blending, etc.). Applicants have found it very useful to perform a relative axis system, i.e. a brightness conversion where the input and output brightness or Luma ends in 1.0 (Luma is used as vertical coordinates and the mapping is OETF, Defined with some fixed definition OETFs in mind, such as SMPTE 2084 or Rec. 709 for backwards compatible image communication). This is very useful for Luma as it bypasses the discussion of the number of bits to be quantified on the vertical axis (1.0 is just the highest luminance code value, i.e. the brightest codeable color). With respect to the (eg input) luminance axis, it can still be considered whether this is the remnant of the relative processing of luminance, but in each case the required curve is (this version always corresponds to the absolute luminance). It can be arbitrarily defined for any dynamic range required for image conversion (because it can be mathematically the same as the framework). If you want to keep the absolute brightness of dark pixels the same, in a relative axis system with low peak brightness, we understand that this corresponds to raising that part of the brightness mapping function TL_CS diagonally, in CombRng. It is possible to clearly understand how the remaining upper subrange is compressed to the extent that it is not too excessive in a good quality synthetic range. However, in the case of poor quality compositing, the compression of bright HDR objects must be more stringent, as evidenced by the shape of the corresponding luminance mapping function to its inferior compositing range, TL_CI. In any case, this situation of peak brightness in the composite range and good anchor points generally requires compression of brightness to typically have this shape, no matter what the HDR shot is. Clarify that (the exact feasibility of this embodiment is detailed below). Also, if it is necessary to relatively convert a good CombRng to a poor CombRng by synthesizing the two functions, the dark pixels need to be relatively bright, which is a visually bright HDR. We also understand that it corresponds best to area compression (relative percentage subranges are a good measure of the coding or rendering quality of some of the images, in addition to the overall available range). In any case, how the applicant defines the luminance conversion in the most typical practical embodiment, particularly very useful properties, will be described with reference to FIG.

HDR video processing (actually just encoding) methods such as HDR10 that simply consider a fixed representation of the HDR image (they can put all HDR object brightness distributions into the image brightness and the OETF is fixed). It is not even necessary to transmit one fixed brightness conversion function (because OETF communication is not required when the Luma is transmitted) if it is selected. Techniques that have a simple view of the brightness distribution of HDR objects, i.e. the possibility of regrading to other dynamic ranges such as the BBC's Hybrid Log Gamma, can communicate using fixed functions, for example as LUTs. it can.

The applicant hopes that both the content creator and the user (TV maker and / or end viewer) can (hierarchically) demonstrate the full quality of HDR, that is, the content creator wishes for HDR image content. Depending on how critical you are and how much time and budget you spend, you can gradually show (from coarse to fine), and the accuracy or end that video consumers also want to follow these specifications. We want to design a system that can determine if the system wants to add a subtle flavor of color processing on the image, for example by slightly shifting it from the contrast of the object specified by the fine grading function.

FIG. 16 uses one of the applicant's priority systems, for example, where a human grader can typically specify the need for brightness reclassification hierarchically using a series of consecutive functions. An example of the power of the anchor when synthesizing is shown.

Consider an input and output range that is perceptually uniformly (almost logarithmically) quantified. This produces a set of visually evenly spaced lightness levels, from, for example, overly extremely dark, extremely dark, very dark, dark, etc. to very bright. It can be imagined that the grader can define the appearance of an image by making the brightness of the object approximately equal to one of these levels, for example a good quality HDR image is 90% to 100% of PB_C, for example. Should have the brightness of the lamp within a very bright brightness subrange between. Of course, the low output dynamic range, here the combined brightness range (L_Cmb_out), does not extend widely to the level that appears as very bright brightness, but graders bring their lamp pixel brightness to, for example, the very bright highest available levels. Rely on mapping. This forms part of the brightness mapping function shape, allowing the reader to understand how the grader (or an automated system using image analysis heuristics) leads to a complete brightness mapping function shape.

Simply defining these two axes maps the brightness defined in one dynamic range to the output brightness, eg, the smaller output dynamic range (which is equivalent to the diagonal function in this graph). This is a typical (rather dull) obvious aspect for doing so. This strategy maps the white color of any source image to the brightest (encodeable or renderable) color possible in the output dynamic range, and the logarithmic definition also inputs a gray value between black and white. Since black is mapped to black to reasonably spread from to the output image, it actually forms a "half-reasonable" output brightness.

However, it goes without saying that this gives a rather unattractive image with the wrong brightness and contrast, which allows the artist to successfully adapt his artistic needs to the composition of any particular HDR scene (buildings). The brightest and darkest pixels of the scene, for example PB_C = 5000 knit encoding, have the same values as, for example, the image of a desk lamp in a dark room with few objects behind, and of course the images of these two images. The composition and meaning are very different, resulting in very different needs for grading and re-grading different object or pixel brightness).

Therefore, Applicants can use a single image, i.e. reconstruct to an approximation of the original 5000 knit master HDR image, or an optimal display to obtain an optimal MDR image, eg, to drive a 750 knit PB_D display. For adjustment, we have invented a brightness mapping definition system that can specify at least two consecutive functions. First, the coarse regrading function F_CRS_MDR totally reassigns the lightness of all perceptual subranges to have a better initial appearance for the current HDR image and its details. The brightest half of the pixel brightness of the HDR scene is the output brightness, for example the brightness of the SDR image to drive the SDR display, or 1000 knits that better matches the synthetic teaching when the master HDR image has 5000 knits of PB_C. Understand the nearly r-shaped curves that are compressed into a slightly smaller subrange of the brightness of the MDR image. Depending on the needs of the scene, the grader can adapt its shape. For example, if there are only light bulbs in some specular spots or the upper half of the brightness of the two pixels, the grader in principle does not significantly reduce the visual quality of that particular scene, so the input brightness. You can even define a function that clips at 1.0 for the upper half. However, if the upper half contains sunlit clouds, even a slight reduction in the local slope of the curve can result in serious quality errors. It should be noted that the reader is talking about the relative brightness axis, and therefore, when the output is in the SDR range, for example, it is used to faithfully render very dark, medium and very bright brightness at the same time. There are not as many Lumacodes and corresponding brightness as possible (eg, a dark kitchen in an area in 3D close to the viewer where blacks are hiding, as outlined in Figure 16b, an intermediate room illuminated as usual, And considering three different illuminated area images with re-sunlit objects visible through the window, it has a different meaning than the bright shop window in the night street, i.e. a building, but can be treated similarly. An example of two other typical regions that have subranges or do not if the content creator so decides is shown in Figure 16c).

FIG. 16c is sufficiently “simple” to allow sufficient grading by adjusting the re-grading of the luminance mapping function using only the specified coarse grading function F_CRS_MDR (HDR is complex). Affects).

However, a scene with three more complex regions (R1 represents a house-like object visible through a window in a bright daytime outdoor street, and intermediate region R2 is normally bright or dim, i.e. with typical SDR brightness. Yes, for R3 is off, i.e. dark brightness), it is reasonable to map all brightness subranges and corresponding object brightness to smaller subranges with particularly low dynamic range (eg SDR range). It becomes more difficult. In such cases, the grader should use an additional curve, the fine graded curve F_FINGR, which applies to the relative brightness resulting from the coarse grade, i.e. deviates substantially equally from the coarse curve shape as shown. Finds the best. However, such a strategy can also be applied to "simple" HDR scenes, such as the nighttime internally lit shop window of FIG. 16c. To brighten the shop window and its objects with a sufficiently striking contrast to make them look really brilliant (cooperatively, but when rendering this single image on both a 5000 knit HDR display and a 1000 knit HDR display, for example) Still good enough to reasonably approximate as the limited capabilities of the display allow on a 100 knit PB_D SDR display), and relatively dark night street object pixels (at least in master HDR grading) To make it look bright enough, the grader can generate an F_CRS_MDR function with a relatively flat top. This provides a reasonable overall look for both shop windows and dark streets, i.e. a reasonable overall image look. However, certain items that do not appear well on the shop window with respect to any one property of brightness, contrast, or color, such as the flame in the shop window, or the white neck of a mannequin or a strongly illuminated part of a shirt. Contains objects. Thus, the grader can define a correction curve (ie, a finer classification curve F_FINGR), which can, for example, reduce the brightness in some part of the brightness range corresponding to the brightness of the shirt, eg, more. Many internal objects get the contrast or colorfulness or what its function achieves as an effect (spatially localized variety to avoid modification of other objects with similar brightness in other areas of the scene). Brightness mapping may even be included, but explaining the further complexity of the system goes beyond the need to understand this application).

Interestingly, the coarse function quickly and nearly accurately arranges the brightness of all objects with a simple gradient action (for example, in one of the embodiments, at the top of the brightness starting at 1.0 and at 0.0. It is only necessary to turn the two dials to determine the gradient at the bottom of the starting function, and the coder can be used by the various HDR image synthesizers and applications of the present invention and the present application with a smooth coarse grading curve. Is automatically determined). However, "actual grading" allows the grading to determine any finely graded curve shape desired by the grader, thereby allowing the brightness of the various pixels of the image object and thus the contrast within and between the corresponding objects. Can be started by fine-tuning.

A normal decoder or display tuner simply uses these two curves as is. Going in the direction of encoding (when using a backwards compatible system that actually communicates HDR image information as a renderable SDR image on a legacy display), first apply a coarse and then fine grading function. , Generates an optimal appearance SDR image corresponding to the HDR master graded image of this particular HDR scene that was the input of the encoder. On the decoding side, the master HDR image approximation is re-applied from the received SDR image, as it is actually the link between these two optimal functions specified to be optimal under a given typical rendering situation. To construct it, the inverse of this very fine and coarse grading function is used. That is, if you have an HDR display, you render an HDR image, and if you have an SDR display that is viewed under typical viewing environments, you render that pair of SDR images (these two graded images). If you have a display with PB_D that deviates significantly from PB_C, this display tuning is used to generate an intermediate dynamic range appearance image that corresponds appropriately to the master HDR image, i.e., at least the appearance of that object brightness to the extent that low display capabilities allow. Sufficiently approximate).

However, there is now a situation where content needs to be composited, i.e. typically at least one brightness and perhaps all images need to be re-changed to harmonize them. The division of the need for grading on these two curves varies within the available output dynamic range (even if optimized for compositing) with object-related needs such as sufficient local contrast. It is very powerful because it can be separated from the general compression of various subranges.

Therefore, the coarse function defined by the anchor brightness is the original for rendering an HDR movie, for example, into a dynamic range with substantially the same peak brightness as the selected composite range, even if it consists of only linear segments. It can be used as an alternative coarse distribution of subranges, as it is clearly better graded for mixing than the coarse function F_CRS_MDR. Various embodiments can now correlate the shape of the various subsegments of the F_CRS_MDR curve, but are usually less necessary. Importantly, the synthesizer can apply fine-grading details to the various newly assigned optimum levels while still essentially maintaining the fine-curved shape (ie, for master HDR images such as inside clouds or caves. Bringing various semantic sub-regions to various sub-regions of CombRng). Therefore, the deviation curve should be shifted to a new position (along the optimal coarse curve for mapping the input HDR brightness to the composite range of F_Cmb) to achieve the finest graded curve F_FINCMB optimal for the new composite. Can be done. This can be done, for example, by simply equalizing the master luminance, i.e. shifting the multiplicative deviation on the vertical line.

The reader finds that such an anchor-based system is very powerful for harmonizing various contents in a relatively simple and therefore feasible manner, even for complex brightness regrading requirements. You should start to understand that there is.

Here, a simpler embodiment in which the device autonomously determines such anchor brightness in HDR images and related aspects within the composite range will be described first. Ultimately it is important to have anchor brightness in CombRng, but in some embodiments this can be easily equalized as it is in the HDR image (not so easily in other embodiments). In that case, the content creator of the HDR image can explicitly co-encode it as L_S2A1). Explain the principle with an important example of having a face, for example (of course, because people are important, many images and videos have at least one actor or moderator, who are typically content creators. It is reasonably well illuminated by, but in the LDR area it is by no means exactly what is reasonably good and whether it is really needed). However, those skilled in the art will understand that since teachings about anchor brightness are common, they can have other anchor brightness for other images even if they do not have a face such as intermediate gray brightness. Should. Readers have found that in the LDR era, intermediate grays were unique and very accurate (rendered as typically average-looking grays in the middle of encoding, with an object reflectivity of about 18% of white or It should be understood that HDR is not quite unique (especially very accurate), as it is a rendering of white relative to white, which corresponds to the original scene with incident illumination corresponding to about 18% of PB_D). Relatively dark if you don't want to force the HDR content to be used again with the LDR straight jacket in a straightforward lighting and capture manner, but want to freely illuminate the scene and freely grade the content artistically. There may be an intermediate gray reflective object in the illuminated image or dark area of the image, and the same gray object in the relatively brightly illuminated image or area).

It would be advantageous if some of these anchor brightnesses were standardized (so that each device could quickly understand what each particular brightness situation of any image input was), but the first image If specified by a different type of anchor than the second image, or if the image anchor is different from the anchor (in the case of only one) that the device determines to be optimal for identifying the CombRng. Those various specifications may still be roughly related. For example, if black is not specified correctly (of course there are different whites and shines and different blacks as in HDR), the device is in a situation (PB_D or other capabilities of the display, and in some cases the viewing environment). Depending on (whether it is the first anchor brightness anc or the auxiliary second anchor brightness), for example 2.5 stops or more than what the device determined as the main middle gray. Suppose it is rendered 4 stops lower. Black can be described (ie, video-encoded) and processed, eg, rendered with lower or higher accuracy. There is a serious act of a criminal hiding in a bush, for example, where the creator of the content cares about what is happening in the black, which is "half-perceived" by the viewer or overlooked without being too noticeable. If not, the image or video should be annotated with additional descriptive metadata for its subrange or color processing regime. In particular, the author should define at least one anchor brightness for black. In today's HDR scenes, there are only photographic background textures in the dark underground, and the creators don't care too much because they are rendered with more or less the same semantic quality, whether they are common or not. If the author can rely on a typical receiver or intermediate optimization of black, the renderer can rely on the renderer as long as it still looks reasonably black (ie deep black, good black, or opalescent black and white). ), They can be rendered at different brightness levels.

Therefore, to give a simple example so that the user can easily grasp the main points of the present invention, the anchor brightness (ank) in the composite brightness dynamic range (CombRng) is the face in the composite (output) image. Specifies what the brightness of the face should be so that it looks good. This, of course, depends on whether the face is well illuminated, strongly illuminated, or in the shadows. This avoids that the actor or person in, for example, a commercial, BD commentary, or videophone conversation is not much brighter than in the movie. Of course, the device has a means that does not necessarily have to align the two face brightness exactly, but renders with offset brightness (eg, when the major HDR movie actors are intentionally very dark, but the HDR movie If the rest of the movie is (very) bright, the commercial does not necessarily have to be darkened by the brightness characteristics of the movie actor, and some embodiments of the synthesizer, for example, how long a dark scene in a movie has already been shot. It should be noted that by using a heuristic as such, it can be further determined, for example by maintaining a significantly different anc brightness earlier or, for example, the elapsed time of the daylight scene type. It is even possible to annotate the anc brightness at the beginning anc of a movie scene with additional metadata, eg, a second data element indicating how much dark scene remains, but content creation. Explain the essence of the embodiment, assuming that some people do not want to get into trouble with many annotations, and use pointers to somewhere else in the representation, such as an image or a histogram. A simple click action is taken to define the current anc value, which can of course be done in the user interface and is in or near the range near the red pseudo-colored anc value as a validation of human anc value selection. It can be indicated by showing all the brightness in the image).

Therefore, the device knows where good facial rendering brightness (anc in the example of this description) should really be, depending on the details of the situation (for the time being, the spectral reflectance averaged over a wavelength of about 36%). Assuming a white face with, of course, other skin types with a reflectance of 5%, for example, fall to the brightness position of the relevant CombRng under its local illumination). Without being bothered by further details, the reader can more easily understand the assumptions of various important aspects that CombRng is exactly the dynamic range of the HDR image (Im2_Rng in FIG. 4) in the current description, and the device. Knows that the most important facial brightness is in that range. As mentioned above, in various embodiments of the apparatus, various alternative determinations of optimal CombRng are possible, on the one hand, the brightness characteristics of at least some input images that are already known or can be estimated. Depends on the encodeable brightness range, the content that is actually within the range, etc., and on the other hand, the actual established or expected use of the image content, such as the viewing environment (ie, the image that is actually common). It depends on the brightness capability of the display on which the content is rendered (typically the average or test image, and the typical appearance of a particular current image). This is already complicated, as the face tends not to have just one color or brightness (not to mention that in some movies the face is painted blue or the like). When studying various faces, even in LDR images, many faces usually have less contrast between the illuminated and shaded areas, i.e. normal, but in principle contain pixelulma to the limit of the LDR range. Understand that (ie, in very contrasting captures or grading, at least some pixels in the dark areas of the face are nearly zero, for example in backlit shots, and highlights are clipped to 255 and consumed. In his content, half of his face may even appear eroded as clipped white). Here, for example, in real-life on-site creation, content can be divided into brightly illuminated images and specially illuminated images (in modern times, even incorporating images created by non-experts in news programs, etc. If it's a night view, the face can actually be badly illuminated). In such complex situations, it may be safe to give up accurate rendering or compositing, but on the other hand, some image content is shot more abnormally (out of good capture habits) in the first place. It can be said that many benefits can be gained by using the appropriate anc value and trimming it later (for example, if the on-site photographer does not stand in the dark part of the scene, for example for some reason). If you encounter an inconvenient situation, such as when you cannot illuminate a stylus, and you have only a camera with low DR capability, you need to optimize the captured image in a non-perfect manner, but at least create it using the anc mechanism, for example. You can quickly annotate the (exotic) details of this image simply as using a stylus on a camera display or on a portable computing device before transmitting the content to a house or the like). With the advent of good cameras and other image processors, such as cameras that can capture dark night scenes as if they were daytime scenes, news content in the next few years, for example still old bad quality mobile phones. It is expected that the complexity of (HDR) image processing will only increase due to the existence of various production criteria and levels. So the best solution is to create a technology that can handle all such situations well. For example, a gangster movie may have intense lighting, such as a cinenoir with intense highlights on a contrasting face and hair or a dull face. Talk shows are sometimes illuminated in low-contrast fashion with shadows that are barely noticeable to the general public, sometimes illuminating women with brighter, more uniform lighting. It is, for example, the original illumination ratio of 4: 1 on the face of the scene when what is important but rather contained in the encoded image (how the contrast is master graded) was taken (most). It should also be noted that it is not half the bright face and half the darkest face). This contrast typically undergoes Luma mapping from RAW brightness in the camera, and if the image is a scan from a celluloid image or movie, the final contrast depends on aspects such as development, push, etc. obtain.

This aspect is specified as having a range of complexion, for example, in the explanatory example of FIG. 4, the complexion of the LDR image is softly illuminated (thus, the brightness R_f is small range), and the face in the HDR image is a horror movie. Severely illuminated, leading to a larger range R_f2, for example, one can walk in a corridor rarely illuminated by spotlights, and the face becomes darker and brighter over time before and after the average value. It also has to do with how the face is geometrically illuminated. For example, if you have a small white spot on your bald head and this image is too bright and improperly mixed with other HDR movies in CombRng, you may hear a statement that the lamp appears to stick out on your head. There is (the viewer can make such a confusing perception when such a person moves quickly in the image and potentially loses part of the story or at least distracts from the wrong part of the image. There is sex). However, a small spot of 2 pixels is not necessarily a big problem. This can usually be ignored to determine a typical face color. When rendered on an HDR display, the person appears to be illuminated by a very bright lamp, but at least the spots are small (eg 2 pixels). If the same thing happens to the entire nose, eyes, or half of the face, it may be more confusing.

Thus, by being able to indicate the brightness of the face in the content (eg, the first L_SA1, eg the LDR image), the creator can not only see where the brightness of the face is, but also potentially which brightness distribution type. Can be shown easily and quickly. For example, if the hair has only small highlights, the average of the dark pixels of the actual face is used as L_SA1 and some brightly rendered pixels look good. On the other hand, if half (or more than half) of the face is heavily illuminated, its mean or other characteristic brightness value is used as L_SA1 and the rest is rendered dark, which is typically this face. This is a better way to merge all the content to be merged. In this case, the creator can show with additional metadata that the face is actually an anomalous brightly lit face, for example an X-stop on normal lighting. Such information can be advantageously used by the synthesizer to determine (approximately) to what brightness this face should be rendered (above or below the anc position in CombRng). Normal illumination of a white (white) face can be obtained from a reflection spectrum that gives an average reflectance of 36% across wavelengths. This is one stop above 18% of the mid-gray (creating these two related values). As mentioned above, the classic LDR framework renders this intermediate gray to about 18% of the display peak brightness, for example 12%. Of course, other races need to be rendered a bit darker depending on the skin type, otherwise it's difficult for anyone who wants to actually detect the skin type of a movie actor, which can be confusing. Because there is. For example, black skin has a low reflectance of about 5%, that is, 3 stops darker than a typical white person. In terms of rendering, the face is rendered somewhat brighter in the LDR range than 36% of the peak brightness of 100 knits for a nice "bright" face. For example, the image creator can indicate in the first field of the metadata the Luma code of the "average" or reference brightness of this face object, which is, for example, 85% or 80% in the normalized encoding of Luma (EOTF). In the second scene, this means that the rest of the place in the scene and its image etc. were illuminated by 2 stops brighter than "2SB". It can be shown that the face brightness is of the type. The reader understands that the same can be achieved by different coding mechanisms, such as the color of the face (eg the brightest, or 95%, or the brightest in the face and hair highlights. A second anchor point L_S2A1pkS, which is a k-stop bright brightness that still contains (but not), and a third anchor point L_S2A1mksS that specifies some dark edges of this face in this HDR or video image shot. It can be encoded. Thus, different types of anchor points can be presented in a standardly understandable manner (as agreed by the various standards that implement the system), eg, "face 36", "face 18", "face". An enumerated set such as "72", "Face 5", "Dark Intermediate Gray", "Intermediate Black", "Deep Intermediate Black", or hierarchically such as "Face" + "1 stop or more" + "Dark" You can use any of the continuous annotations of the data, the face will reflect 36% of the average amount of light present in that part of the image with a particular type of illumination, or more importantly the correctly rendered scene. Means a specific subrange of the dynamic range of the display used to render those pixels to give the appearance of. "Dark" is an intermediate light color that is typically rendered on most or all displays in the 10-100 knit range, for example, under which there is at least one black area, and preferably slightly less illuminated ". Dark lighting values divided by up to 10 used in areas of "black", i.e. typically used for deep shadowed landscapes seen when the protagonist is simply walking a candle in a castle, for example in HDR art scene rendering ( Or typically there should be 1-10 knit brightness), and a deep black area up to 1/100 of the normal, dark illumination (ie 0.x to 1 knit brightness). Adding a "bright" area on top of dark lighting should give you great results for well-controlled, high-quality HDR scenes (rendering about 10 times brighter outdoors for TV viewing). In an actual scene, the brightness is 100 times or more, that is, the brightness is 100-1000 knits.) For "extreme brightness", for example, a lamp, an explosion, etc., that is, an upper limit subrange of 1000-10,000 knits, for example. It is possible to render effects that still make it interesting to have a high-end HDR display with. Display tuning (finally bridging the gap between scene-referenced image coding and display-referenced image coding by reintroducing the correct level of display relevance after absolute coding of the image structure) Controls for correct image composition, as in this application, or for correct image composition, must allow and be sufficient for accurate differentiated control in the five scenes. In practice, when investigating the types of HDR scenes that actually exist or can be created by the artist, one often sees two area images, such as the night shop in Figure 16c (one area). Is a uniformly illuminated image and of course typically SDR-encoded), the chances of encountering a three-region image as shown in Figure 16b are somewhat low, and the full power of five different illumination schemes. If you want, you're actually creating a complex HDR scene (perhaps some movies show the ultimate potential of HDR appearing only at the beginning as a commercial). However, the reader will now understand that this principle is not limited to smaller or more complex embodiments. If the content grader specifies only one anc value, it is already possible to harmonize various image contents with that anc value. This typically occurs when there is only one critical object in the image, for example a single actor or moderator (however, as mentioned above, this actor is even very contrasting. It has only one typical illumination and brightness when illuminated, but can theoretically be anything between 0 knit and PB_C). If nothing is specifically required for the anchor type, the synthesizer is expected to operate with normal anchors associated with intermediate gray values, and typically "dark" normal rendering brightness. Therefore, content creators who annotate only one anc value should typically use the representative value for the mid-gray position (in the local illumination of the scene and its area of the image) of the actor, eg. The two-mode histogram is present in the sun, then annotating that type, for example, "bright", which allows the synthesizer to make a good decision on what to do with this simple small amount of information. (For example, if the movie shouldn't be too disturbed, it can be decided to render the "bright" actor commercially as a dark PIP in the movie anyway, or to allow the impression of lightness, for example (bright). + Dark / 2), for example (30 + 300) / 2 knit, can render a bright actor slightly above the dark level, and the standard allows further metadata communication, if this metadata is filled in. Authors of content, such as commercials, can show that they don't want to deviate too much from their favorite "bright" rendering, but for example, end viewers want to enjoy a movie using the UI. The device can ignore this for a variety of reasons, i.e. it is preferable to reduce jamming errors in brightly inserted content in cinematic histograms).

Of course, if the grader further annotates the anc value, the grader specifies, for example, the "deep black" of the grader, and is the synthesizer prepared for bad black that can ultimately be rendered in a bright observation environment? Regardless, this can be taken into account in its final compositing (ie, compositing takes place in the theoretically complete framework of an ideal rendering system prior to display tuning to the actual system). ..

Therefore, we obtain the complexion from the HDR image because we assume that the HDR complexion is already at that luminance level for the sake of brevity of this embodiment. Is a trivial matter that does not require the consideration of complex luminance mappings), and by having at least an aspect of identifying where the principal complexion is in the LDR image (source anchor luminance L_SA1). As an output image, all complexions of both images that are accurately composited in the composite image can be arranged relatively simply. And even in the simplest embodiment, the ambient color (starting with the face) and all other image colors are automatically dropped relatively well around their anchor brightness (anc). Philosophy for other colors is also relatively simple, as it does not need to work in more detail for these technical rendering scenarios. A good image is obtained when all other colors fall within the range of about 2-2.5 stops around the mid-gray (eg, white is typically 90% or 5 times brighter in linear brightness). Printed matter often fails to obtain very deep black, and practical LDR televisions typically have a 32: 1 (peak white vs. still reasonably distinct) considering the reflection of light on the windshield of the television. It was limited to the deep black color that can be produced. Having 5 stops of image object content makes sense even on the dark side of a well-lit LDR scene, given that dark colors, especially important blacks, are within 2-2.5 stops from mid-gray. It means that there is no problem. In principle all of this means nothing in HDR image processing, especially in compositing and rendering, so in terms of improved mechanics, and especially in terms of complexity (images can contain anything), what is semantically We need human guidance on what is involved, and thus what is a good composition. A more complex embodiment can make a smartly tuned selection of aspects in which the other object brightness is smartly placed around the main object, which is the face in this easy-to-understand example, positioned by the anchor brightness anc. .. You can annotate, for example, that the anchor point is the limit of the range. To avoid doubt, when we mean video, we are not limited to simply referring to temporally continuous images, and that a particular strategy should be applied to the whole movie as well, but rather that. It can only be applied to parts of a movie, eg shots of images of the same scene (ie, for example, when new commercials with different object brightness characteristics are later combined, certainly at different time moments CombRng Can be different). Since most of the features of the embodiments of the present invention are similar, at least with respect to spatial composition, whether steel or video is used, two embodiments are to avoid boring and confusing formulation. It should be understood that video or images can be used in the description where it is not necessary to distinguish one of the forms, and the reader can read the other as well. The determination of the function that maps the first (source anchor) brightness to the second brightness (composite range anchor anc) should be clear enough to the reader for various embodiments. As described above, for example, the color conversion unit of the synthesizer increases the brightness of a specific input (source image) from the diagonal line to the required position, and sets the line as the maximum and minimum values (0 and) of the input range and output range. A specific shape that can be easily configured to create a bipartial linear function that connects to [PB_C_image_1; LMC]), but reprocesses the brightness of the image to make it more suitable for different dynamic ranges or brightness etc. If the function is provided, for example, the function can be linearly scaled so that the point of ancS becomes the anc output value of the Y-axis. It is not necessary to use accurate anchor collocations, but it must also be clear how similar systems can be built that operate in the vicinity (not overly large) around their anc value. The available neighborhoods are those shown by the content creator in the source metadata (for example, this anchor is rendered with a deviation of 20% of its exact value) and what the synthesizer determines in context. It can be established as an arbitrary composition between and (as mentioned above, for example a face in a commercial comes out of a simple calculation, for example a calculation rendered in 150 knits, but the composition device is a variety of programs. You can decide to deviate from it in any way according to the rules, for example some PIPs that are desired to be placed in a very important center position of the image, for example all information of all parts of the image to be composited. The only device that has the, and even the rendering system and environment in which the composite image should be displayed, should be bright enough to determine that it is optimal or reasonable for the current total display of the composite image content. Can be determined). If nothing is specified, coefficients less than or equal to twice or more than twice the ideal anchor position can be reasonably close in a realistic manner.

Finally, in various embodiments, in an actual device (eg, a synthesizer) that actually creates the composite image, since the applicant's embodiment can operate with a function that specifies the final appearance of the image. I want to mention that it doesn't have to be. For example, if the compositing device is an STB, for example, if the standard is sufficient for the image, then send the already fully optimally composited image to the HDR10 TV encoded over HDMI®. The TV renders directly as a silent display. However, the STB can also determine and prepare the situation and send all the necessary data, at least some image data and some transformations (it is the optimal transformation FF for calculating the composite image). It should be sufficient if this happens, but in other embodiments it is possible to transmit an anchor value, perhaps even an additional anchor brightness value determined by the STB to the television). In that case, the TV has all the information to uniquely perform the compositing, i.e., in PIP compositing, not only does the pixel selection occur, but the corresponding SDR image received before being stored in the compositing image buffer. FF_1 is applied to the pixels and FF_1 is applied to the brightness of the HDR image pixels to obtain the exact brightness of those pixels in the composite image. The simplest embodiment of the synthesizer determines only one good anchor position. For example, depending on what the TV is displaying, how it is set (eg by controlling the brightness of the user), etc., in the current situation, for example, anc_1D = 30 knit, a nice looking image. It only determines the average gray anchor value of the dark light that should be rendered. Of course, this doesn't give the best control over image composition, but the first content shows an anchor where the action is taking place on a dark night (eg the average brightness value of black) and the second content is bright and flashy. When presenting a commercial, this simple synthesizer embodiment should already be able to fairly reasonably harmonize both images around its anc_1D value. Of course, a more complex embodiment of the compositing device determines if there is a good compositing situation for some types of content and shifts to the above five anc value brightness types (deep black to extreme brightness). On the other hand, specify the optimum anc value. The device then examines the source anc value (and potential additional metadata, such as a regrading function that indicates how black can be grayed out if necessary) and the specified HDR. By looking at the situation or type of effect of, the input content can be strongly harmonized to the different lighting subranges of CombRng in any situation (low brightness, eg not enough display peaks). If the composition is done or prepared for the expected further processing such that the brightness rendering system renders the ultra-brightness much brighter and more spectacular than the brightness, the composition is some of the posterization for the brightest image pixels. It can fluctuate towards the format and is not the best theoretical composition of the image pixel colors themselves, but it is a good composition for such poor quality systems). Therefore, the anchor brightness within the composite range is typically determined as at least one good brightness for determining the other brightness of the surroundings (general needs for good quality image rendering, and / Determined by the synthesizer (or based on the details of the actual or envisioned content and / or display capability), the source anchor does the same in the source image dynamic range, typically the creator of his image. What we consider important to us, at least any particular important object and the pixel area around it, is rendered and well controlled (at least for brightness) under various possible usage scenarios, especially in image composition. Determined according to whether it should be used in the state. It is understood how to define anchor brightness suitable for a particular object, the spatial and temporal subpart of the image, for example with the first scene that occurs in the living room of a house under daylight on a rainy day. It may be necessary to redefine the anchor as the movie changes from a different scene to a new image (shot), such as the next scene in the basement illuminated by one practical light bulb in the center of the basement. Obviously (both scenes have similar objects that can be represented by anchors, such as actors' shirts, but of course the master image has differently graded brightness and of a variety of displays. The final MDR image for this requires different rendering).

The color conversion unit (310) is an output determined as a result of applying color conversion (FF_1) to the input brightness of the pixel color of the first image or video (Im1_LDR) equal to at least one source anchor brightness (L_SA1). It is advantageous to determine the color conversion (FF_1) so that the luminance (LF1_o) is equal to the anchor luminance (anc). Therefore, one aspect of adjusting the brightness of at least two images (or videos) to be combined is that the selected anchor brightness is the same for all images and the final image composition (eg, PIP or alpha). The composition strategy is designed so that it is set to an appropriate anchor brightness (anc) value in the composition dynamic range of the output image (Im_o) at which blending is performed. Those skilled in the art will understand how the device can calculate a function that has FF_1 (L_i = L_SA1) = anc as a characteristic and then maps all the brightness around the anc to L_SA1 respectively. To do. In particular, because you work in a linear luminance color space (content creators create differently graded images for image dynamic range, and the corresponding regrading luminance conversion function details when communicated, so luminance. Smart choices regarding object brightness placement along the axis are mainly made by the decoder 251 that processes at least one of the images, often simply using linear mapping (or sometimes non-uniformly spreading like a gamma function). A simple non-linearity with respect to contrast may be used). Understand how the brightness determined by the coding of the input image by some coefficients (using the linear dimming coefficient) must be darkened in the case of a dark image, as illustrated in FIG. 18a. A slightly different and similar mapping is made to input a brighter image brightness than the original anchor L_SA1.

In the case of a linear function, it is easy to understand that the function can be determined (for example, in an axis system normalized to 1.0, x-axis = linear input luminance of the LDR image, and y-axis is the luminance at CombRng). (Normalized to), that is, L_out = FF_1 (L_in) = C * (L-L_SA1) + anc. C is a contrast constant and the device can be further optimized to better match the two images. This can be determined by looking at the semantics of the two images, such as the histogram and spatial brightness distribution (eg, a small shape like a lamp at the top of the image), the average brightness relationship of the images, and so on. For example, if an HDR movie consists primarily of a dark basement where a person is walking in the shadows and the high brightness area is only a single lamp, the relevant contrast (most of the HDR images) is not very high. Therefore, if it must be combined with a very high contrast LDR commercial, it is desirable to reduce the contrast of that commercial somewhat to match the low contrast perceived and expected by the visual system in a dark environment (eg, Figure). The brightest of LDR commercials, such as 18c, by applying a gently sloping portion of FF_1 to the brightest relative color, whatever is semantically contained in the pixels of a sunlit outdoor object or light bulb. Part can be constrained). On the other hand, raising C slightly above 1 can increase the vividness of the commercial (ie, all possible SDR images, even if it is not shown at the maximum or HDR average brightness of a very high contrast HDR movie. The contrast in the local subrange of HDR CommRng to which brightness is assigned can be controlled). Of course, the reader can read that the device has a relatively high contrast, eg, just below the point (L_SA1, anc), but begins to decrease its slope and becomes horizontal at some threshold black L_out = L_b, etc. Understand that other FF luminance (color) conversions such as can also be performed. In practice, in principle, any function can be determined by the synthesizer as long as the source anchor brightness is roughly mapped to the anchor brightness, but typically for regrading the various brightness subranges. By communicating the specifications that the synthesizer follows, such as partial brightness mapping function shapes, or by identifying and optimizing (contrast between different pixels or regions, quality of texture rendering, etc.) Unless the synthesizer embodiment has an important internal image analysis unit that allows it to propose a more beautiful looking composition, the resulting function is relatively simple (eg, multilinear segment functions are usually at least important). Good results for non-HDR scene images).

The color conversion unit (310) has a brightness ratio obtained by dividing the second output brightness (LT2_o) determined as a result of applying the color conversion (FF_1) to the second input brightness (LT1_i) by the output brightness (LF1_o). It is advantageous to determine the color conversion (FF_1) so that the ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1) is multiplied by the constant (C). As described above, in a simpler embodiment, a fixed value of C with respect to the overall (partial) luminance range (input luminance of the processed image) can be determined, where C is the input luminance L_i (C = CF (C = CF). It can also be a variable function of L_i)). The partial luminance range can be defined in various ways by the synthesizer (whether guided by the additional metadata formulation received or itself), eg, determining the range of the main histogram lobe. The range can be determined for the brightness of the LDR image between ancS / k and k * ancS, where k is, for example, 4 and the rest of the LDR brightness is the two dashed lines in FIG. 18c. It is mapped to a partial function. As an example of more variable function FF, if there is important information (eg, a strongly illuminated face), the contrast of the brightest part of the LDR image (eg, on a constant L_it, such as a percentage of the histogram). ) Can be increased, but if there is only irrelevant information in the main action, eg the device increases the average brightness of the first mixed eg LDR image somewhat (eg by mapping to anc + d_ank). However, the contrast in the bright areas can be reduced instead so that the highlights rendered overly bright in the brightness-mapped LDR image in CombRng do not exceed the HDR movie. This makes it possible, among other things, to respond more to the expected lightness distribution calculated in the human brain. Thus, for example, LT2_o can be selected as a certain percentage above LF1_o (equivalent to, for example, anc), such as 150% LF1_o, or k-stop above LF1_o, and C1 for its luminance interval, etc. specify. A smart fit of one such (at least one or more) subregion of a typically non-essential image (eg, an SDR commercial) can greatly improve the harmonious appearance of the overall composition. ..

The color conversion unit (310) includes a deviation determination unit (312) that determines the luminance offset (d_anc) based on at least one source anchor luminance (L_SA1), and the color conversion unit includes at least one source anchor luminance (L_SA1). The output luminance (LF1_o) determined as a result of applying the color conversion (FF_1) to the input luminance of the pixel color of the first image or video (Im1_LDR) equal to) is the anchor luminance (anc) + luminance offset (d_anc). ), It is advantageous to determine the color conversion (FF_1). As mentioned above, there are significant differences between such object-related brightness aspects, such as facial lighting (eg, very high contrast), and the relationship between facial brightness and the rest of the image. There is. Therefore, the device does not render the LDR face (or the first image anchor object of any dynamic range, or the area and the corresponding brightness subrange) exactly at the output brightness anc, but rather, for example, twice the brightness. It can be judged that it is more appropriate to render with. This is determined, among other things, based on the type of facial illumination (eg, typically indicated by the anchor type encoding), eg, the source anchor brightness (L_SA1) is half that of a very brightly illuminated face. If determined as average brightness and the face is rendered in a large size in the final image, the device can determine, for example, to determine a negative d_anc of 50%. The heuristic considerations behind these programming rules and calculations are, for example, bright areas radiating excessively radiating surrounding dark areas, or excessively noticeable attention, or viewed by the viewer, such as in a movie. Consider simply stimulating brightening compared to the rest of the main image. This consideration can also evaluate the brightness composition of HDR movie images. For example, the image may consist of a dark indoor area and a bright outdoor area visible through an open garage door. If a relatively much brighter LDR commercial PIP is to be placed in the dark, it is the contrast between the two images and the local contrast seen from it by the brain in adjacent objects of the two images. It should be darkened so as not to be excessive (ie, the average brightness of the face in the LDR, or the additional anchor brightness that characterizes the LDR image, with a single representative brightness of the LDR image, anc and the darkness of the second HDR movie image. Relatively large d_ank) to weight in the middle between the region's local average brightness. However, if the bright PIP is displayed near the sunny outdoor part of the HDR movie, it needs to be brightened with a positive d_anc. Thus, in general, these deviations d_anc are ideal composites (eg, of two faces) in an ideal situation (eg, when the face is surrounded by an intermediate gray background in both images). And the actual situation of at least the main image (ie, in this example, the HDR movie pasting the LDR commercial) around where the compositing occurs (eg, PIPs of small sizes X, Y) is overall (ie, average brightness, It is calculated based on the size (what kind of object it contains, such as potentially more semantic features such as internal complexity by texture measurement) and what is local. Therefore, the offset is generally determined based on what harmonizes the two images, and in various embodiments the characteristics of the image itself (contrast, source anchor position, the type of object content in the vicinity, and the content around it). , Geometric size, etc.) and composition (whether the PIP is placed in a small, less important place in the lower right corner of the image, or how the mix shines with partial information, such as in a blend mix). Depends on. A simpler embodiment simply uses a relatively simple harmonized structure that is reasonably good in itself (especially to reduce the risk of excessive contrast if the anc + d_ank value is determined to be commercial, for example). (Can be optimized), more advanced compositing allows more accurate viewing of where and how the second image information is placed, or even additional properties of compositing.

The color conversion unit (310) provides at least one second source anchor luminance (L_S2A1) obtained from a second source (351) that supplies a second image or video (Im1_HDR) of the two images or videos. It is advantageous for the readout and deviation determination unit (312) to determine the luminance offset (d_anc) based on at least one second source anchor luminance (L_S2A1). We have described a simple embodiment in which the luminance structure of the main image (referred to as image 2 in the claims) has a known luminance structure and can therefore be easily transformed to form a basic reference in CombRng. .. This is a useful aspect of compositing when the HDR image is the primary image and distortion should be minimized and therefore it is primarily the second content that is transformed in that direction. HDR images are in the 0-400 knit range of Im_HDR dynamic range Im2_Rng (which is, for example, the 0-300 knit range of CombRng, or the same 0-400 range of CombRng) for all Lambertian reflective objects (eg studio talk shows). This can happen if you know that it is the type generated to ensure that you are always given the exact specified brightness to be within (corresponding), all of the above are mostly Highlights, such as small specular reflections on metal. In that scenario, the bottom is produced in a manner very similar to LDR television production, the HDR effect is mostly light, and has certain types of HDR scenes that are not overly important for their exact brightness (eg clearly visible). Not only that, its external objects are not objects that pass through windows that should have a certain brightness to convey some artistic mood). In this case, the problem is mainly adjusting the LDR commercial to the lighting conditions of the HDR talk show mainly up to 400 knits, and to some extent what exactly is in the bright area (eg LDR commercial). The above contrast mechanism, which has a multiple constant C, brightens small highlight areas (identified or assumed) within and further corresponds to the brightness of these highlights that are actually present in the HDR talk show. Alternatively, any general determination and application of the brightness mapping function can be used).

However, in general, the HDR content can be anything. For example, its face may not be exactly illuminated by about 25% of 400 knits, but if the actor is running in a dark mine with sparsely placed lamps, it falls across the brightness axis. Thus, for anyone who annotates the content in any content, the HDR image also has at least one second source anchor luminance (L_S2A1) to indicate where one or more of its faces are on the luminance axis. Then it is advantageous. By reading L_S2A1, the device can verify where the brightness of the face is within the second image of HDR, so it is more certain that both images will be harmonized in the final compositing ( See an example of matching in FIG. 4). HDR images can be optimally converted, taking into account that whatever the best final compositing situation (especially CombRng) is, the movie is most likely to be displayed outdoors instead of, for example, dark surroundings. it can. For example, the FF function can modulate different facial lighting to a lesser extent, especially if the viewing conditions are such that the rendering of the texture of dark objects deteriorates compared to the optimal cinematic viewing of the movie. ..

The color conversion unit maps the first image or video from the dynamic range associated with the encoding of the first image or video to a dynamic range having a peak brightness that differs by at least a multiplication factor of 2. Alternatively, establish a color transformation (FF_1) applied to the first image or video depending on the first color remapping function (F1_L) that specifies the change in the brightness distribution of the object in the video (Im1_LDR), or The color conversion unit is used to map the second image or video from the dynamic range associated with the coding of the second image or video to a dynamic range having a peak brightness that differs by at least a multiplication factor of 2. It is advantageous to establish a color transformation (FF_2) applied to the second image or video depending on the second color remapping function (F2_L) that specifies the change in the brightness distribution of the object in the image or video (Im2_LDR). Is. As explained above, in a basic HDR video or image coding framework, how the brightness distribution of an object should change as it moves from high dynamic range to low dynamic range (or vice versa). Is shown by the communicated color conversion function (at least the brightness conversion is defined, but the saturation conversion is also needed to increase the saturation of the object that needs to be darkened in the appearance of the LDR, for example, so that it looks sufficiently active. The function typically translates, for example, one extreme or intermediate grade from, for example, 5000 knit PB_C to another, for example, 100 knit PB_C, at least a coefficient of PB_C of 2. (Specify a different dynamic range appearance) (this is generally a complex optimization for the lower capabilities of inferior PB displays or codecs, taking into account the semantic characteristics of the HDR scene). That is, when converting from a master 5000 knit HDR graded image to, for example, a 100 knit image or a medium dynamic range image optimal for driving a 1200 knit display. Typically, the creator can decide how he wants to change the appearance, for example by gradually darkening the highlights and dark areas.

The power of the compositing framework is not the different (single) optimal rendered image dynamic range, but how to formulate a new kind of function that maps to the compositing dynamic range, especially depending on how accurately you want to control the compositing colorimetrics. It is possible to decide whether to do it (this is especially with parameters that quantify the demands of the content creator, owner, or distributor, viewer, eg, the viewer who is willing to watch the movie, or the display maker. It may itself depend on various external parameter settings). Of course, the overall appearance of the composition depends on the particular composition and what is contained therein, but (in the brightest subrange, eg the upper 20% of Luma, clouds that desire careful re-grading or less important. It does not mean that the regrading function itself (with or without objects) does not contain any interesting regrading needs or requirements that the synthesizer wants to follow to some extent.

Here, if the input brightness of the image around the anchor brightness anc is mapped using the simple linear function FF described above, the details of F_L become unnecessary. However, more sophisticated mappings can vary with linear strategies, specifically as the content creator implicitly needs in the function F_L of a particular image of that particular HDR scene in that brightness subrange. You should do it according to what you are doing. For example, if the author prefers to quickly eliminate darker brightness to black, the color conversion unit (310) can take this into account when determining the optimal function FF, eg LDR rather than linear function. By quickly clipping a large proportion of the brightness to black, it is possible to extend part of the contrast in a brighter area (eg, around the complexion anc). Therefore, the device can evaluate at least the functional shape of the luminance conversion function of the set of F_L functions (for ease of understanding, the reader can normalize to 1.0 input luminance to 1.0 output luminance. It can be assumed that it is a function mapping of one normalized arbitrary shape), and a part of that shape is used to design the shape of the final function to be mapped to CombRng. It should be noted that such a function gives various requirements for regrading parts of an image by itself or in relation to other possible brightness in other parts of the image. is there. For example, for a geometric area of an image, specify that the brightness present there should not be over-boosted, for example, even if the rest of the image is pixels with the same brightness that can be greatly boosted. can do. Any such partial functional shape can be shown in principle (eg, the previous embodiment developed by the applicant satisfies the property of falling within a particular geometric area, such as a rectangle. Allows enumeration of a subset of pixels in an image having a brightness between, for example, L_min and L_max, for example, a constant of k and l for the brightness of that region of the HDR image between L_min + k and L_max-l. Can be used to define a partial function of the desired mapping behavior). For example, a single or parameterized boosting function can be given to these brightness L_out = B * L_in + O, where B and O are functions of various properties such as LMC, or data from HDR images. It may be the average brightness of the composite output image surrounding the rectangle in which is placed. Any subdivided reclassification behavior may be indicated in the functional specification prior to the final compositing, or decomposed (for at least part of the composited content) after the compositing image has been created. You can even make it possible.

It is advantageous for the dynamic range establishment unit (302) to establish a composite luminance dynamic range (CombRng) depending on the luminance in at least one of the two images or videos. Not all embodiments require or can take into account the details of the final viewing environment. For example, the output images are mixed in several master control rooms for transmission to various users. The final situation is unknown and can vary considerably (when one viewer is watching, for example, on a train, and another is watching in a movie room created in the attic). In that case, it is advantageous that the composition is optimized based on the characteristics of both images, since the characteristics of both images (or all cases where more than two are mixed) are always related. Of course, the reader understands that the synthetic paradigm is even more extensible when applied successfully. For example, the first content creator mixes two image sources, but this does not mean that more content cannot be mixed completely. It can occur both after the fact or with the expectations of the first author, and at least some synthesizer embodiments need to meet all of these situations. For example, the first content creator makes the first statement about what should happen, so you can determine the best CommRng (and / or anc) for your content, but perhaps some entities will be in the future. Some typical additional characteristics (eg, ticker tape with news information at the bottom, or in the context of a movie theater room or store or museum, other or part of it, such as the first rendered image content. It is a second projector that projects a second image information across a target, and this second content has dynamics such as dynamic range, average brightness, and temporal evolution of average brightness). Is expected to be added. The second content mixer follows the instructions of the first content creator, for example, given that there is additional information about the final composite image or its presentation, it is better to maintain CombRng, or perhaps fine-tune it. You can still decide if it is better. Two classes of embodiments are possible. In a simpler class, the two images are already inseparably mixed at this stage (eg final PIP compositing) and the final receiver is the rendering details (display PB of the rendered display, average ambient brightness, etc.) ) To optimize the whole image. That is, for ease of understanding, this composited image is taken and applied as a square root function to that brightness to obtain the final output brightness to be rendered. This is sufficient for many scenarios, as the mixed image already has harmonious brightness and the viewing environment is not significantly different from what was intended.

More advanced embodiments allow the receiver to redetermine the object / pixel luminance distribution of the two images, and some embodiments include, at least to some extent, the redetermination of the original image (eg, their basics). Coding, or at least the degradability of the original brightness as described above). For example, in some embodiments, PIP synthesis can be specified in certain luminance distribution modes, but the CombRng specification and the function FF are co-coded so that at least one of the two images is properly harmonized within the CombRng. Make it brighter. For example, if you simply put the LDR PIP in an HDR surround image with brightness linearly scaled to 5000 knit PB, you can of course imagine that if you don't do the right thing on the receiving side, you'll get a terrible look composition. However, the receiver gets all the information needed to make the combined rendering accurate on that side, i.e. in a harmonious PIP image. The receiver knows everything (ie, for example, a partial luminance mapping function received for that area, or a special additional anc value for that rectangular area, or, for example, pixels around that area. It can be done by appropriately changing the pixel brightness in the rectangle, taking into account that it looks much brighter than the above, etc.). Those skilled in the art will understand that these functions FF are not the functions F_L that regrade for the different dynamic ranges themselves, eg, a single LDR image reasonably to the dynamic range of an MDR display, eg 1500 PB. Rather than a simple function that maps properly, it is installed on the creator side (or anywhere) to give a good mix of the characteristics of the two images together for the LDR image harmonized with the HDR image. Understand that it is a function specified by the synthesizer).

The dynamic range establishment unit (302) depends on the peak brightness of the display on which at least one output image (Im_o) is rendered, and preferably also on the brightness characteristics of the viewing environment, the combined brightness dynamic range (CombRng). ) Is advantageous. When embodiments of the device of the invention are incorporated into an end-rendering site device or system, such as a television or computer, it may be appropriate to further specify the composition depending on the viewing details. If the LDR image looks excessively bright when viewed as is in a dark environment, it is dark when combined with the movie, especially because the movie has already been optimized for that particular viewing situation. It may be too bright for the viewing surroundings (as shown in FIG. 3, the automatically regrade appearance image Im2_MDR of the appropriate MDR has the same content reservation or the same broadcast content. Calculated from, for example, 5000 knit Im_HDR received by all viewers to view, for example 1200 knit PB displays viewed in dark surroundings). Partial images can be mixed in the appropriate MDR dynamic range (Im1_LM and Im2_HM). This is distracted by bright content (perhaps spatially adjacent), especially from other composited images, especially when it is difficult to see the darkest pixels of one image in a bright viewing environment. If so, the additional anc value of darker pixels in at least one image (eg, the main image) can significantly improve the optimality of the final composition. Those skilled in the art, of course, need to optimize the optimization to combine and present the two images, typically different from the optimization of a single image, depending on the overall situation as the observer perceives. Understand that (in practice, of course, constructing a device that optimizes more or less accurately based on the complexity it can afford, taking into account the positioned market price of the device, and in a simpler embodiment. Some already do pretty reasonable work). Looking at a single LDR commercial that is too bright, the viewer adapts greatly to what is visible on the screen, even in a dark viewing environment. However, when two images are combined, what human vision adapts to is not a trivial a priori, and therefore the brightness and brightness that the brain feels about the composition of the rendered pixel brightness. A system is desired that is relevant (he is unlikely to start watching a major movie badly) and therefore allows content creators to provide at least the minimum required information and guidance (practical). The technical system handles complexity, such as the effort spent by the content creator and the cost of the IC that needs to be calculated, and at least some minimal required color conversions to make the system behave at least not badly. Always balancing with the need to be half-reasonable, typically standardized various embodiments further work in what particular control function and to what extent. decide). Therefore, one of ordinary skill in the art will determine the CombRng (and typical anchor brightness, and the specific shape of the FF function if not linear, or the multiple constant C if linear), at least depending on the capabilities of the display. Understand that is advantageous (its PB is sufficient in most situations, but further characteristics are incorporated if it is not fully used in terms of power saving etc.). And if the device is connected to evaluate the brightness of the viewer's surroundings, they want to do more. For example, an illuminometer can be used, but somewhere in the receiving room, eg, various areas of the imaged viewing environment, are calibrated to the luminance meter and connected to a display already on some TVs. It is better to use a calibrated camera. This makes it possible to estimate how the viewer will experience the image. A simpler version can be used, or an advanced version, such as a camera that checks the various brightness of the (look around) area behind the TV up to 1TV width on each side.

The brightness anchor determination unit (303) is a composite brightness dynamic range (CombRng), at least one brightness of two images or videos, the peak brightness of the display on which at least one output image (Im_o) is rendered, and the brightness of the viewing environment. It is advantageous to determine the anchor brightness (anc) depending on at least one of the characteristics. Also, at least one anchor brightness anc should be established from which the brightness distributions of both images can be distributed in a harmonious manner. The reader should understand that various embodiments can make CombRng and anc decisions in any order. For example, in some embodiments, depending on the capabilities of the display (actual or expected) and the brightness characteristics of the viewing environment, for example, typically for the current movie or image shot-by-shot situation, etc. First establish what a good CommRng is.

For example, some embodiments analyze a first image of a movie, or a characteristic set of images sampled throughout the movie, before starting the movie. Alternatively, the cinema has co-encoded metadata from which the device can establish a reasonable at least early CombRng before decoding the first image. For example, the metadata can specify that the average brightness of the movie is 200 nits in the lower range up to 400 nits, and in the most extreme sequences, for example shots taken in the desert, objects like the sky or sand. Diffuses the brightness to 900 knits (and an average of 600 knits) (preferably should be rendered on any MDR display that has sufficient capacity to do so). Next, an embodiment of the image compositing apparatus captures a fixed, over-bright region in CombRng (especially when servicing displays of various PBs), eg (small specular spots, lamps, or very scenes). For that movie so that the top 100% of the brightness can be used for all highlights (whether in the bright areas above or not, including image details such as brightly illuminated areas). It can be determined that up to 200% of the 900 knits, which is the most excessive requirement, should be reserved. Of course, the more advanced specifications help in accurately determining the bright brightness region of CombRng by encoding what kind of HDR effect is typical in a program or movie, or part thereof. .. Working with a pair of such subranges, especially when complemented by a pair of good anchor brightness for these ranges (they primarily indicate the need for content or foreseeable rendering). Powerful and versatile (whether or not it shows the ability of

After determining the optimal CombRng, the device can determine where, for example, the anchor brightness of the face color should be based on it. For example, in the case of a Lambert object whose half of the range is fully illuminated, it can be determined to use 40% of that 50% of the peak brightness as the anc point of the face brightness. However, some embodiments can work in reverse. Taking into account all rendering characteristics, the device can determine the anc brightness suitable for the mid-gray of the Lambert object in the face or main area so that it looks appropriately bright to the viewer. The device can then build a suitable CombRng around it. For example, the device examines the most demanding HDR images and sees bright objects (see lamps) up to 20 times the brightness of the face. The device then sets a maximum range at that brightness value or determines that, for example, 80% of that value must be sufficient for composite rendering of the two sets of images (both of which are composites of a particular embodiment). The high value of each Im2_MDR of Im_HDR (used in) is usually clipped to the maximum LMC of the composite dynamic range. This is the case if you want to harmonize the content with, for example, low dynamic range capabilities, especially if the HDR movie is not the main content, but most of it is a PIP on a computer display that shows other image areas, for example, with information. Can be very useful. Thus, in a simple embodiment, when an object that is twice as bright as the complexion is generated and is ready to be added to the composite image or image (Im_o), it is generally one of the images (ie, the first and second). The anchor brightness of either the HDR image or the LDR image when the LDR image is used in compositing is twice as bright in each image pixel. Alternatively, some brightness can be made darker or brighter by more advanced color conversion (or some of the compositing may weight or change the pixel luminance in the compositing calculation). Similar considerations may be made by the device to result in a lower limit brightness LmiC of CombRng, eg, the appearance of a composite image (now having a much brighter object than at least one of the originals) being overly dark or overly contrasting. It will be slightly brighter than at least one of the source images so that it does not become.

Further embodiments may be implemented as corresponding devices or methods (whether small as part of an IC or large as a consumer or professional system as a whole).

A method for synthesizing two images or two videos of two images or two videos of multiple images (Im_HDR, Im_LDR), one of which is a high dynamic range image or video.
An establishment step for establishing a composite luminance dynamic range (CombRng) characterized by at least maximum brightness (LMC), the establishment step further comprising a determination step for determining anchor luminance (ank) in the composite luminance dynamic range (CombRng). Including, establishment steps and
An application step of applying a color conversion (FF_1) consisting of at least a luminance conversion to at least one of two images or videos, wherein the color conversion is the first image or video (Im1_LDR) of the two images or videos. ), And an application step established based on the value of the source anchor luminance (L_SA1) read from the first source (350) to supply.
It includes a forming step of synthesizing the images using their pixel colors in the composite luminance dynamic range (CombRng) to form at least one output image (Im_o).

In the method of claim 10, the color conversion (FF_1) performs color conversion (FF_1) to the input brightness of the pixel color of the first image or video (Im1_LDR) equal to at least one source anchor brightness (L_SA1). The output brightness (LF1_o) determined as a result of the application is determined to be equal to the anchor brightness (anc) or to be equal to the brightness offset by the difference (d_anc) determined from the anchor brightness (anc). To.

In the method of synthesizing the two images, the color conversion (FF_1) outputs the second output brightness (LT2_o) determined as a result of applying the color conversion (FF_1) to the second input brightness (LT1_i). The luminance ratio divided by LF1_o) is determined to be a constant (C) times the ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1). As mentioned above, other brightness that occurs in the image around the anchor brightness may simply be mapped around the anchor brightness, or may be mapped in a more accurately determined manner.

In the method of synthesizing two images, the color conversion (FF_2) of the second image or video of the plurality of images supplies the second image or video (Im1_HDR) of the two images or video. Determined based on at least one second source anchor brightness (L_S2A1) obtained from the source (351). A second image as well as a first image so that different brightness ranges (eg bright light, bright shadows, deep shadows, lamps, etc.) can be optimally adjusted and mapped to each other in the output range of the output image CombRng. Various anchor brightness of the image can be determined.

In the method of synthesizing two images, at least one of the color transformations (FF_1, FF_2) applied to each of the two images or videos is from the corresponding dynamic range in which each image is encoded. It is determined at least in part based on at least one receiving function (F1_L, F2_L) indicating how it is converted to a dynamic range of at least twice or more. As mentioned above, these functions have at least twice the brightness of all objects in the scene (that is, the PB of the reference display corresponding to the EOTF that defines the Lumacode Y'of the received input image. Determines how accurate it looks in any dynamic range (different). Therefore, that information can also be used, for example, when it is necessary to regrade to a slightly lower dynamic range (eg, CombRng is slightly smaller than the MDR range of Im_MDR). This does most of the smartness to get the adjusted brightness of at least one image or video used by itself, but the first pixel of the image mapped to CombRng for the presentation of composite image content. It's a good starting point for roughly determining the set of brightness. Since the shape characteristics of the functions F1_L and F2_L, which are inspected and used in both embodiments, are specified by the creator of the content (eg, are smartly determined according to the object partitioning algorithm and the corresponding luminance range) or For example, by dividing the CommRng into 5 equal parts or unequal subranges and roughly determined by doing the same for the two input image luminance ranges and their corresponding subranges), eg, by a synthesizer or method. Somewhat simplified further tweaks, such as determined heuristic linear stretches, or advanced further tweaks that take into account the specific required brightness adjustments in those objects or images within the brightness subrange may occur.

In the method of synthesizing two images according to one of the above methods, at least one of the combined luminance dynamic range (CombRng) and anchor luminance (ank) is an object in at least one of the two images. Luminance distribution-dependent property of, information summarizing at least one luminance characteristic of two images, the dynamic range of the display on which at least one composited output image (Im_o) is rendered, and the viewing environment in which the output image is viewed. It is determined based on at least one factor from the set of brightness measurements of.

All of the above may be incorporated into various systems and forms, whether they are consumer devices or professional systems whose components may be present on servers in other countries connected via the Internet or the like.

Specifies how to map the CombRng specifications (eg, its upper brightness LMC and its lower limit brightness LmiC) and / or, for example, the typical anchor brightness of a face, and in some cases the image pixel brightness to CommRng as well. It is advantageous if the image is supplemented with metadata such as at least one color conversion function.

At least one semantically relevant (preferably a pre-agreed type) by the content creator so that the lighting choices of the image when received can be understood by the device wishing to combine the various images. It is also advantageous to annotate the image with anchor brightness.

Each of these methods is advantageous to implement as a computer program, which allows the processor to perform all steps of any embodiment of any of the above methods when the code is executed. It can be communicated via a physical carrier, such as a computer program, that contains software code that allows the code to be executed.

These and other aspects of the methods and devices according to the invention will be apparent from the embodiments and embodiments described below and the accompanying drawings, which will be described with reference to them. These drawings merely provide a specific, non-limiting diagram that exemplifies a more general concept, the dashed lines are used to indicate that the components are optional, and configurations other than the dashed lines. The elements are not always required. Dashed lines can also be used to indicate that an element described as required is hidden inside an object, or for intangible objects such as object / area selection (and how it appears on the display). ..

Schematic representation of how (in 2015) various different aspects of encoding video that result in images with very different luminance characteristics that are not easily related in the (recently initiated) HDR era were proposed. It is a figure. Outlines the basic components of how the applicant's HDR coding technology can encode an HDR image, along with different appearances (or color grading) of the low brightness of the LDR of object brightness in the same image HDR scene. Illustrates how this information is sufficient to calculate the appropriate appearance with the correct image object brightness on any available HDR display with any peak brightness (PB_MDR). It is a figure. FIG. 5 is a diagram schematically showing a basic embodiment showing a main unit of an image or video synthesizer capable of HDR of the present invention. FIG. 5 schematically illustrates how proper color conversion for optimally selected composite dynamic range occurs prior to video or image compositing. Occurs when one of the possible compositing of picture-in-picture configurations is to compose the pixel brightness (or common colors) of two images in a simple manner rather than carefully preparing them for colorimetry. It is a figure which shows roughly the problem to be obtained. A diagram schematically showing how the device can determine an exemplary suitable color or at least brightness, conversion function (FF) for at least one of at least two images to be combined. is there. Automatically artistically optimizes an image from a first dynamic range to a different second image where the dynamic range (at least its peak brightness) may vary due to each of the two factors, as the content creator desires. It is a diagram schematically showing which algorithm can be used as developed by the inventor of the applicant to regrade to. How many how the device adjusts the (FF) function to color convert the image to be composited to the composite range (CombRng) based on the semantic nature of the object luminance distribution in the imaged HDR scene. It is a figure which shows the more advanced example schematicly. FIG. 5 is a diagram schematically showing an example of an application of embodiments of the devices and methods currently described, ie, multi-video original creation such as a television talk show. The embodiment of the device and method is a diagram schematically showing a second example used in a system in an intermediate node of a video communication network, for example, a transmission control room for servicing a local area (. The reader should understand that other embodiments of the compositing device may exist elsewhere in the video or image processing chain, eg, indoors of the end user of the video). FIG. 5 is a diagram schematically showing how an embodiment of an apparatus can determine CombRng and allocate source image luminance from a good position of the luminance value of the anchor luminance to it. It is a figure schematically showing an example which further explains how the embodiment of the apparatus can determine CombRng and allocate source image luminance from a good position of the luminance value of anchor luminance to it. It is a diagram schematically showing how pixels can be simply combined from two images without using the technical aspects of the present invention, which can lead to poor quality composite images. There is sex. How very much to the same object, especially if you want to take full advantage of the possibilities of HDR imaging, even with one image capture method encoded by one technical HDR video coding method. It is a figure which shows roughly whether it can have a variable brightness. Typical for how different compositing ranges can be determined, and especially for establishing relevant perceptible anchor brightness (anc), and from which partial content from at least two images or videos is harmonized. It is a figure which outlines what it can actually mean about a typical luminance mapping function. In particular, how very strong harmonization can be achieved when identifying the early parts of the video processing chain, the video coding itself, in certain aspects of various coarse and fine regrading of image brightness. It is a figure which shows roughly. To various algorithms that can determine the cause of errors such as both image content distortion and / or content inconsistency that can be determined by the human content creator in the case of human grading and image annotation. Based on this, it is a diagram schematically showing how the automatic version can formulate the determination of various variables. FIG. 5 schematically illustrates some further examples of embodiments of a useful luminance conversion function for determining harmonious luminance for image composition.

1 and 2 have already been described above, where FIG. 2 preferably encodes an HDR video of the same scene, more precisely an LDR grade image and an HDR (typically 5000 knit PB) grade image. Indicates whether to encode (by applying a brightness conversion function to the received LDR image) and anywhere within or outside the dynamic range that spans the LDR and HDR grading as the edges. Can produce optimally regrade images for display peak brightness.

FIG. 3 illustrates at a higher level some of the underlying components of most embodiments of HDR images, or video, harmonic synthesizers or methods. The situation is illustrated below using the example of PIP, but the same technique can also be used to adjust the brightness of the grading, for example in the temporal continuity of videos interleaved from different sources. For example, a good CommRng level for LDR commercials can be established and the brightness of the HDR image can be gradually adapted towards it (various additional metadata can be sent with respect to the temporal evolution of different videos). ). Those skilled in the art will be more complex when two images in very different states and with differently defined intensities (not to mention Lumacode) are converted into those with similar intensities that correspond and are related (to be combined). It is understood that a mixture of contents can be initiated in various aspects, such as alpha blending, or that a portion of the face can be replaced with some computer-generated facial structure having properly adjusted brightness, etc. Let's go. For example, the face brightness mapped to CombRng in the first image (eg, both sides of a differently illuminated face) is x and y, that in the second image is x + e1 and y + e2, and e1 and e2 are. If the deviation is small enough, a mixed formula such as L_out_comb = alpha * L_im1_comb + (1-alpha) * L_Im2_comb can be applied, and L_im1_comb and L_Im2_comb are each colored to CombRng (using the FF function). The pixel brightness of the first and second images after conversion, alpha is a real number between 0.0 and 1.0, and L_out_comb is the output image Im_o, that is, in a composite image having CombRng as the dynamic range. The final brightness of the mixed pixel.

The reader is optional if the creator has as much control over the original image as possible (especially if he needs to know the details of the intended rendering environment), of course until he is completely satisfied on request. It can be understood that the composite image can be fine-tuned by using the above means. However, this embodiment is not so easy or possible, for example, because there is no time to manually tweak everything, or because the inputs and variables that define the optimal appearance are uncontrollable or unknown. Provide the situation. Something can always change depending on the image processing situation, so this is what the latest HDR processing technology should provide. That is, at least the present embodiment enables a more versatile application in which various instantaneous colorimetric fine adjustments can be performed, for example, in response to new demands, and retains a part of the appearance information of the original image. Can be done. The important thing is to retain at least some of the most important aspects of the image at each stage. Of course, the end user can decide to completely recolor the video, but the efforts of all content creators are ignored and lost, which is actually useless to anyone (for example, TV makers, You want to add some of your taste to image rendering, but you don't necessarily have to do so by ignoring everything about the image, especially what the content creator has specified for the image, but on the other hand, receiving The side should not feel that there is nothing to say about image coloration, even if the situation seems apparently inappropriate).

The decoder 251 is similar to that described in FIG. 2, i.e., the HDR coding method described (extracting the luminance mapping function according to various possible image global or local embodiments, from which the new MDR reclassification is performed. (HDR, LDR) image grading can be processed according to (determining the image). Of course, the reader understands that this is merely an explanation of one of the possible realizations, and the device usually processes (ie, composites and composites) any kind of HDR or LDR video or image. ), That is, Im_HDR can also be defined by the PQ EOTF or BBC-HLG method. However, in this example, two images to be mixed from some (second) image source 351 (eg, a satellite channel, or a connection to video storage over the Internet, or a connected camera). It is assumed that a second image of is obtained (eg, the main and most important (main) program is an HDR movie that is accurately and artistically color graded). For example, it is a 5000 knit PB HDR grading in which the image is converted with PQ EOTF, then DCT encoded and encoded according to some HEVC profiles and levels. Metadata including the function F2_L that downgrades this HDR image to a 100 knit LDR image is also communicated. The decoder can produce the optimum image for the connected display, eg 2500 knit MDR image, Im2_MDR. This is the correct grading in this film, and it is monitored by itself whether it is scattered spatially or temporally without being combined with other (first) image data.

FIG. 7 summarizes how such regrading from the appearance of the first dynamic range to the appearance of the second dynamic range can be performed according to one of the present embodiments. It is assumed that L_in is normalized to the [0.0-1.0] HDR input brightness corresponding to the actual 0-5000 knit brightness (rendered on the display when rendering the HDR image). L_out is the LDR brightness that is normalized but corresponds to PB = 100 knits (ie, SDR brightness). The function 702 that maps this HDR to this LDR image is an example of the F2_L color in a particular luminance transformation (for simple understanding it can be assumed that it simply has a grayscale image, but is required in this embodiment. It also shows that color conversion can be applied to RGB colors). For example, suppose you want to derive the optimal regrading function 703 for the PB_MDR of a 3400 knit MDR display. The diagonal line 701 indicates that if you want to map a 5000 knit input image to the best image for a PB = 5000 knit display, the HDR image is already optimally graded for that display (ie, by identity conversion). (Because it is converted symbolically), it becomes the required brightness mapping. For example, a vertical induction direction DIR is established. The scaled function for the 3400 knit display PB is for any input luminance Ls between F (Ls), i.e. the optimally determined downgrade function F (F2_L) is present at its luminance point and Ls. Should also be given the value F * (Ls). This point can be determined as Ls + FI (PB_MDR, Ls) * (F (Ls) -Ls), where FI is a function that gives a value between 0 and 1, and in simpler scenarios k (PB_MDR) *. It is only a function of PB_MDR that can be multiplicatively written as Ls. Which function is used correctly depends on the reclassification principle of the embodiment used, and the reader depends, for example, on the details of the viewing environment (or the tastes of the viewer are also taken into account), but more often. It can be imagined that various embodiments, which are more or less complex, can exist. However, the details go beyond what needs to be explained in this application and are already sufficiently complex (readers have established some reclassification functions, applied to the input image, and of the MDR display. It should be understood that it is possible to obtain an exactly regraded image with the correct image object brightness for).

The first image above may be anything, but the principle will be described using LDR commercials, and the principle of this embodiment is similar to other contents. Due to the large difference in dynamic range of at least two images, as well as the extreme distribution of object brightness in LDR commercials (ie 2500 knit is not the best LMC value for CombRng, but another value can be chosen). This 2500 MDR is not properly graded when mixed with 100 knit LDR commercials (ie 2500 is not a suitable PB). High dynamic range is understood not only in terms of "high" or bright, but also in terms of "dynamic", that is, where on the luminance axis the author has assigned various semantic objects and / or areas in the image. Should be. In contrast to common misconceptions, simply displaying an image in high brightness can result in a rather flat LDR-ish appearance and annoying brightness, so all or more between gray values. It is desirable to control the. For example, the creator of a commercial makes the LDR commercial brighter and more flashy in order to give a high visual impact to the 100 knit LDR screen. However, this impact is too high even with a dynamic range of 2500 nits (not to mention the 5000 nit PB_D display for viewers who purchased such a display), so there is a need to display a clean bright lamp in a movie, for example. CombRng, which takes into account both the extremes of LDR grading and LDR grading, is required in such cases, for example with a 1500 knit LMC. The device can use methods such as measuring the overall contrast of the LDR image, for example, how large the amount of pixels above k% = eg 80% brightness in the LDR image, and how often it averages. Counting how bright or extreme, or how the amount of pixels above k% changes, etc., and determining the extreme criteria from which, the brightest part of this LDR commercial was mapped to CombRng. Sometimes it determines how bright it will eventually be, for example what the LMC should be (eg, the brightest of the LDR pixels is which non-linearity is used. Depending on whether it depends, or how much brightness is needed to place the brightness of the face, it can be 300 knits, the brightest Lambersian pixels in an HDR movie (ie the lower of that movie). In the range, for example identified as 5 times the encoded mid-gray of its main area for HDR movies) is, for example, approximately k times or 200 or 450, and bright lamps are 5 times 300 knits. And give LMC = 1500 knits.)

Those skilled in the art understand that there are various aspects leading up to CombRng (or anc), such as which is mainly determined by the HDR movie and then the LDR image is mapped to that CombRng. The brightest HDR object in the LDR image is largely or largely determined by the LDR image characteristics as illustrated here (as in the first described example above). Sufficient quality of HDR images is protected when determining how much to extend above the brightest or average color, or when iteratively determining until two reasonable optimal values are found. It all depends on the needs of the particular situation and the behavior of the embodiments of the particular synthesizer. Which of the two images has the highest importance can be preset in various ways, for example, the viewer, in some embodiments, using a remote control, what kind of image. Can indicate that they want to minimize the obstruction of the movie even if they are synthesized, or the content creator can give instructions on the content that the synthesizer ultimately determines.

Returning to FIG. 3, the first image source 350 is, for example, Rec. A first image is provided, which is assumed to be an LDR image Im1_LDR encoded in a standard manner according to 709 (of course there may be a decoder unit not shown). Recent videos are annotated with their own function F1_L for converting to HDR, and older SDR videos are not annotated. That is, it has a function F1_L related to its metadata, enabling artistic optimal conversion to a dynamic range other than the original 100 knit DR. If the function is missing, the synthesizer can always apply a function and estimation that works reasonably well on average. Again, the image source 350 is a variety of sources, for example, a security camera at the front door of a viewer's house PIPs an image when someone rings a doorbell or the like.

As described, according to one of a variety of possible algorithms, the dynamic range establishment unit (302) analyzes situations such as the luminance characteristics of the first and second images, LmiC in the above example. An appropriate combined luminance dynamic range (CombRng) is established with = 0 knit and LMC = 1500 knit. The brightness anchor determination unit (303) determines the anchor brightness (anc). For example, the luminance anchor determination unit (303) reads the source anchor luminance (L_SA1) of type "face luminance" from the first image source in the metadata associated with the first image, and this value is, for example, 45 knits. Is. If at least one face in the image is known to be important, then it is necessary to establish which is the best complexion anc in the CommbRng (of course, in other embodiments, the source anchor at the specified location). The anc can be determined first without examining the brightness, or if the anchor brightness is seen to be average gray in, for example, a bright outdoor subregion, for CombRng, within the overall average gray and CommRng. Determines the d_anc position that positions the bright external brightness compared to the overall neutral rendering of the color). Depending on the complexity of the embodiment, a typical embodiment confirms which value of the second source anchor brightness L_S2A1 of the type complexion indicates how the complexion is currently in this shot of the movie. Want to. This is of course a low value on the normalized luminance axis, since the upper range is used for HDR effects such as brightly lit objects or areas, or lamps. Still, if a reasonable value of, for example, up to 400 knits in the range of diffuse objects is 180 knits and only a value of 30 knits is found, then we know that the face is essentially dark, dim, or both. .. In any case, even if the face looks darker than the commercial, it will be darker if you decide to set the anc value to, for example, 180 * 1500/2500. Therefore, an anc value somewhat close to 30 knits is calculated and the face brightness of the LDR is placed at or near, for example, 0.8 * 180 * 1500/2500 knit values, for example an anc value of 0.3 * 180 * 1500/2500. I would like to map the HDR complexion in the vicinity. This ensures that the face is properly dim and the face is properly dim for the LDR image, as both should be adjusted according to the optimal anc value for the HDR movie according to the story. Of course, the same can be done by considering other or additional anchor brightness, such as intermediate gray, especially in the HDR lighting area.

The only thing to do after establishing a CombRng with an anchor brightness anc is generally to have the dynamic range of the two images according to one particular brightness value determined to be directed to or around the anc value. Is to map optimally to. One way to do this is to make a linear assignment of the luminance using the optimally determined contrast value C, which is the slope of the line of the luminance mapping function, and then calculate the luminance by calculating the linear equation until clipping occurs. Is to allocate (note that the amount of clipping can be controlled with the C value and can be clipped to a value within CommRng, i.e. a value below the maximum LMC). The down-mapping to CombRng in this example takes into account the same calculations used by the decoder 251, i.e. because of the brighter and darker parts of the image, typically as determined by the human grader. It can also be done by using at least a luminance mapping (F2_L) that has the characteristic of encoding the behavior required for. For example, knowing CombRng, the decoder 251 can have pixels corresponding to the second image mapped directly from the 5000 knit HDR image Im2_HDR. The compositing unit (320) can apply a simple algorithm (all colorimetric optimization) because the color conversion unit (310) performs all the necessary pixel color conversions, especially at least the appropriate brightness adjustment. For example, simple addition weighting or pixel replacement or switching unit) because the conversion has already been performed. The source anchor brightness readout unit (311) is one source anchor brightness associated with one of the two images, for example the face brightness of the LDR image, or a good mean of black in the imaged scene. At least read. As mentioned above, in some embodiments, from both or generally all images, some source anchor brightness (eg, a special gray value, or average) that is characteristic of a particular brightness region within the image. Read the gray values associated with common semantically important objects such as the brightness of the image or the brightness of the dark sky. In non-simple scenarios, rather than mapping the brightness of two images based on equalizing only one similar anchor brightness (fixed to anc), the output brightness is smarter than anc. Since we want to position it, the deviation determination unit (312) determines an appropriate luminance offset (d_anc). As illustrated, this is because, for example, commercials contain very bright and high contrast faces, and horror movies contain muddy dark faces, so for example anc is close to the brightness of a dark movie face. If determined, a sufficiently large positive offset d_anc is required for where to map the complexion of the LDR commercial face. The images Im2_HM and Im1_LM with intermediate and accurate brightness mapping then enter the compositing unit (320) and then the HDR colors mapped whether they are linear RGB or recalculated to Y'CbCr. (That is, it is as simple as a pixel switcher that depends on the (x, y) position of the pixel that puts the Im2_HM or LDR Im1_LM color into the output image Im_o.

FIG. 4 further clarifies all of the selected examples by showing mappings over the luminance dynamic range. Im1_Rng is the LDR 0-100 knit dynamic range of the first image of the LDR, i.e. has a coded brightness that must be between the minimum brightness Lmi_L = 0 knit and the maximum brightness LM_L = 100 knit. Im2_Rng is, for example, the dynamic range of an HDR image with LMH = 5000 nits (or in the above example, a display with an LMH of 2500 nits could be an optimized MDR image). From this it is already known that the CombRng need not have to be the same as or close to the range of Im2_Rng or Display Di_Rng. Although this display range is shown for comparison, it is also not used in compositing in some embodiments, rather Im_o goes to the display adjustment color conversion unit and the defined composite output image Im_o is eg 1400 knit or Any value is mapped to, for example, 3000 knits on the display dynamic range (for those skilled in the art, for example, the synthesis is done on another device, such as a cable headend, and a particular user is limited to, for example, 950 knits There can be scenarios where the LMC is higher than the LM_MDR if you have an HDR display of PB, or if mixing occurs at all viewer locations and the viewer has a high PB display, eg, 7000 nits. Understand that, currently, when receiving HDR program and LDR image content such as news programs with PB = 1200 nits or less, LMC is determined to be much smaller than LM_MDR = 7000 nits). The LDR commercial is already in good harmony with it, so the adjustment function can be, for example, for HDR movie content (ie, F2_L). Of course, a more complex embodiment optimizes both images directly for display dynamic range. The reader will understand that there are situations where the CommRng is not, for example, in the display dynamic range, for example in a production or video transmission site where the characteristics of the viewer's television are not yet known. It's one of the complexity created by the HDR market, which needs to be met. Video processing was simple because all TVs were about the same (PB is about 100 knits) in the LDR era, but now some viewers have 1000 knit HDR TVs and others watch. One has a 7000 knit HDR TV, the other viewer has a legacy SDR 100 knit TV, and other viewers want to see the image on, for example, a 500 knit PB Ipad or mobile phone (still various images) Not beginning to reveal all the details of the content). It should be understood that due to their great variety, not all of these display dynamic ranges are the most suitable CombRng to combine two images and optimally represent both. However, especially if you have to render to a 1000 nit TV or monitor at the end, you can upgrade the LDR up to 5000 nit image 2 HDR dynamic range and synthesize it, then significantly downgrade again, especially the F2_L function. Is it optimal to risk losing a significant portion of the brighter LDR color when used?

This point will be explained a little more with reference to FIG. 5 using the example of PIP501 in the main region 500 of the HDR movie (although similar problems occur with, for example, temporary masking and adaptation).

Classic LDR combiners can synthesize in Y'CbCr (ie Luma + 2 chrominance coordinates), non-linear R'G'B', linear RGB, or in principle any color space (working at video speed). This is less common for systems that need to be, but want to save computational resources). Let's assume you have an image with a gray value.

If you simply switch the pixel's Lumacode Y'per pixel, they will be determined to be very different for HDR and LDR images, resulting in very bad results (basically a major misconception of the processor). It becomes a problem). For example, everything visible in the window 510 of the LDR PIP (assumed to be the director's commentary) is too bright for the LDR and the video alone is not interesting anyway, so the maximum Lumacode Y'= 255 or normalized. It is clipped at 1.0. When rendering a composite image Im_o in a Luma encoding where some of the pixel-normalized pixel Luma of the original HDR movie (eg dark night view) is replaced by the normalized Luma of the LDR image, for example in PQ HDR EOTF. By encoding, the PIP looks overly bright (the window color is rendered, for example PB_D = 4000 knit, and instead, in principle, the color of the outside object is well coordinated, or at least less irritating. Has a bright clipped color). This leads to excess radiation 502, also known as breathing or light spread. If this is not due to the physical properties of the display (eg, low amount of backlit LEDs, mutual reflection on the display faceplate), then due to the glare of the human eye, without using hands to block bright areas. It leads to a situation where it is difficult to see a very bright area on the HDR display. In short, the viewer does not want such a thing, and hopes that it will not happen if technically possible. At the very least, it is clear that the brightness of the LDR PIP and the dark areas of the HDR movie night view are not in good harmony. However, the reverse is also true. If the PIP is near the sun or in other very bright areas in an HDR movie, it may be too dark. For example, a color that should be clearly white can be unsightly if it looks too gray. When the PIP face brightness is reduced by approaching the dark HDR face brightness as shown on the right, they too are due to the fact that the brightness of other objects in the LDR image (in contrast ratio) is related to complexion. It gets darker more properly. Therefore, all colors should appear to be relatively well adjusted.

Assigning the appropriate LDR subrange to a PIP image (ie, a subrange of 0-100 knit brightness that will be HDR CombRng whatever it is), i.e. to a 100 knit LDR PB instead of a 3000 knit (relative) display PB. Maximized Rec. Corresponding to decoding it correctly at 709, then by allowing these LDR pixels (assuming CombRng equals the available display DR) to have a brightness of up to 100 knits in the display dynamic range. You might simply think that you can solve all the problems. However, as mentioned above, this may be a good choice in some situations, but it depends on what the dynamic range of the display really is (relatively, it's a random relationship). In very bright sunshine scenes, a 3000 knit display may have adjacent objects with 1500 knit brightness adjacent to a maximum of 100 knit LDR PIP. This makes PIP rather unfortunately dark and its color looks muddy rather than bright, which is not what the person who paid for the commercial wants. The same thing can happen when compositing in the HDR image dynamic range (Im2_Rng). Therefore, it will be understood that proper mixing is not trivial, i.e. a good framework like the present invention is needed. Also, if you have two HDR images, how the content creator used the available range, even if the EOTFs are not different, that is, the entire system (content creation rules, etc.) in terms of colorimetry. I don't know how I designed it. Even if the PBs of the two encodings are not significantly different, for example a 5000 vs. 4000 knit, the author can make very different decisions, for example with respect to a low range with Lambersian reflective objects in an indoor scene. For some reasons (for example, to make a particular story look attractive in a well-designed HDR scene, other authors have other dynamic ranges, such as viewing on a portable display where the image still has HDR punch. Create easily convertible news programs). The first creator defines up to 300 knits (on a 4000 knit system), while the creator of the second image defines up to 900 (on a 5000 knit system), for example, for an indoor scene on a space station. Decide on a knit. It's sometimes as simple as whether you like a bright or eerie dark space station. Even if very strict rules regarding face brightness lighting or at least rendering are followed, for example 50% of diffuse whites (ie, 300 or 900 knits) that do not have a properly harmonious face in composition are considerable. Looks strange, different and vivid. For example, a swimming pool may appear to shine slightly brightly from the inside, or it may appear dull and sober water, how dark the night scene may be, how hazy it may be during the day, especially when it relates to other objects in the scene. And so does many other colorimetrically important objects in HDR images, such as someone worried about the brightness of a bright poster in front of a beverage vending machine when the story and mood are conveyed. Can happen.

FIG. 6 shows two possible examples of color conversion to CombRng. As the input image, there is one of the images to be combined having the relative input luminance L_in. As described, the apparatus needs to determine a color conversion function (FF_1) that assigns the output luminance L_out_comb of CombRng to all possible input luminances. For example, having this function loaded in the LUT can start processing the input pixel colors of successive images. The first simple linear determination produces a linear function 601 determined by two parameters anc_FC and a multiplicative contrast constant C for complexion luminance L_SA1Fc in the input image. This function can be clipped in black or white, which may or may not be desirable (eventually depending on how simply the content creator or device manufacturer decider wants it). .. In more complex embodiments, it can be improved. The brightness mapping function 602 illustrates an aspect of successfully constructing a desired mapping based on only two important anchor brightnesses. Its use is generally not very difficult, but it produces very good results. Perhaps this commercial LDR image has pixels that are too bright in the background. When applying certain photographic compositing rules, choose to shoot the main person against a dark or light background, such as an average gray background, or a very light or white background, or a potentially dark background. be able to. It is common in videos to depend on what the actual background is (for example, if the video is a report on a fashion photographer shooting a model against a white background, the video will be shot in front of the white wall or background. Show a speaking fashion photographer). In the studio, the background may be designed. In the weather forecast, the weather forecaster may be standing in front of the rendering of the photo submitted by the viewer, non-expert (virtually typically by the green screen). Or, for some reason, someone in the movie may be sitting in front of a bright advertising screen, such as an LED screen. This is because in the past LDR weather forecasts displayed nicely tuned LDR photos, but viewers will soon still send HDR photos generated by the LDR method for display, or a few years later. This is another example of the need to adjust somewhere, as someone may send very poor quality LDR photos that are mixed and displayed in the HDR program. In any case, on curve 602, it is considered that there are too many too bright pixels in the PIP-ized LDR image. Not only does the output value of function 602 need to be lowered for some white, but soft clipping needs to be initiated for, for example, the brightest input luminance. Therefore, the curve 601 gives a result that is too bright. The apparatus (301) did this by reading out three types of source anchor brightness in the input image and assigning them to each anchor brightness in the CombRng output image. That is, the content creator corresponds to a good reference level for diffuse white, that is, which brightness in the image is white (ie, for example, having a reflectance of 90%), and lights up on average. Also specify. The device assigns to this LDR image the diffuse white anc_DW brightness of the LDR image at CombRng, which is clearly lower than that provided by the linear brightness conversion curve 601. Therefore, the synthesizer is now smartly deciding how to transform the brightness of such bright image pixels, given the specific needs for the bright pixels in the composite image. Can be done. For example, in this case, in order to avoid clipping, the black anchor brightness L_SA1RBk is a priori specified in the content, and the appropriate black anchor brightness anc_RBk is determined at the time of composition, so that the same appropriate behavior is obtained on the dark side. Can be specified. The device can decide, for example, to continue the function linearly from this anc_RBk point to absolute black (ie 0.0) and raise it above anc_DW with a very small slope. Alternatively, the device can use its knowledge of L_SA1RBk source black anchor brightness to optimize black for a particular rendering situation.

A more complex example may be illustrated with reference to FIG. 8, i.e., not only can some of the embodiments of the devices and methods of the invention design multi-segment functions based on various anchor brightness. The shape of the part of the luminance conversion curve (FF) based on the reclassification behavior specified by the color conversion function (F2_L, and F1_L if available) in joint communication with the image from the metadata creator. Explain if you can determine.

The function (FF) that determines between the three anchor brightness reoccurs as before (by linear or non-linear interpolation), but the device is obtained from the received metadata (F2_L) with respect to the classification by the content creator. Based on the knowledge obtained, it can be assumed that the luminance conversion behavior of the advanced luminance conversion function 810 of FIG. 8B is to be determined. F2_L is the luminance conversion of the second image from its original dynamic range (DR1) to another dynamic range (DR2) specified in advance (the luminance conversion for the second image, but in the composite output image). The same is true for all other images to be composited), that is, the input luminance L_DR1_im2 is related to the output luminance L_DR2_im2, which is the appearance of another dynamic range for the same imaged HDR scene. From this graph, we can see that there is a major part (between anc_DW and anc_RBk), which is well represented (rendered) with good enough brightness (for compositing situations) and good contrast brightness. , Black (L_SA1RBk or less) can be easily discarded as the function rapidly drops to zero and clips, at least in some appearances corresponding to some dynamic range. That is, in this example there are no very important objects in the darkest brightness subrange. The device can use this to determine the appropriate function 810, for example by quickly dropping the function for these dark blacks, how dark or contrasting the composite hope is, or at least in the composite image. This can be determined based on the second image portion (eg, based on the amount of pixels having a brightness of L_SA1RBk or less after proper compositing, such as scaling in the second image or in the output image Im_o). That is, the synthesizer reclassifies the two received graded image codings (ie, SDR100 knits from the master of the received luminance mapping function, eg 5000 knits PB_C graded image). Look at how the darkest pixel brightness is treated when regrading to grading) to determine some similar partial shapes.

It also shows the possibility of the brightest pixel. The discontinuity of the curve F2_L indicates that there is a clear presence of bright colors representing the two regions, eg, the Lambersian objects in the indoor scene, and the ramp (or similar situation). This means that the final brightness can be more freely assigned, especially in the presence of lamps, especially depending on the dynamic range available on the MDR display and / or CombRng. Therefore, the device can determine the appropriate average level L_RefB for these pixel luminances and determine such an 810 curve shape. FIG. 8A shows how the received luminance mapping function F2_L for mapping between the received master HDR and SDR classifications looks along with the functional shapes and behaviors of the corresponding bright and dark areas (802) and dark areas (801). Is shown.

Those skilled in the art will appreciate that different embodiments of the invention can be applied to many systems, many combination scenarios, with different components in different combinations, but with some two figures. A possible example will be further described.

FIG. 9 shows an example of a television production system. This is a multi-camera studio creation (first camera 901 and second camera 902), with some live content coming from outdoor shoot locations and the like. Further, this may be created outdoors by an OB van or the like. Technology directors need to mix camera feeds with other videos, such as HDR and LDR, as opposed to HDR video as well as single type (universal single Rec.709 standard exists) LDR video. There are various types and characteristics. The technical director wants to determine the optimum composition with the composition determination device 903 in the production control room. The technical director determines the synthesis, but this example shows that this synthesis does not necessarily have to be a fixedly defined Im_o (rather, the technical director only determines the appropriate synthesis). , The details of the synthesis can be communicated from the video communication pipeline to various receiving sites further down in various ways, and the receiver wants to redetermine the synthesis, for example, slightly different from what the technical director wants. It may be desired, i.e. the composition may be communicated as coarse images and functions and other metadata prior to the composition). In this case, the image signal 910 having metadata containing two images (eg, having a scaling function to determine the determined PIP synthesis) is the determined CombRng, anc, and optionally FF_1 and Communicated with CombRng to map the image with FF_2 color conversion, any receiver (eg, transmission receiver 904 in some intermediate nodes such as the transmission control room) actually applies the final compositing. be able to. To allow for potential re-determination (at least one of CombRng, anc, FF_1, or FF_2), the creator, for example, whether it contains a lot of bright scenes, 3 than the brightness of the average movie. Double brightness is required (ie what brightness mapping is desired for this content in the final CombRng), details regarding the brightness of the object or area, such as brightness range and / or empty space. Further convey metadata such as semantic information to the video.

FIG. 10 shows another possible embodiment, namely the intermediate coupling device (1001). It is installed, for example, in a transmission control room such as a cable provider. The intermediate coupling device is the main creation (created in FIG. 9 to properly couple them according to the director and communicated as the final image or as an initial image with sufficient metadata), namely Im_Prod and eg local. Images can be obtained from the commercial feed IMLocCom. The intermediate synthesizer (1001) can also potentially perform dynamic range conversions, for example a PB of 5000 knits, which is more suitable for purposes such as further distribution in some pre-established video communication systems. It can be converted into an output image encoded with = 3000 knits. The latter is of course interesting if the device 1001 outputs two separate image streams that are later mixed, but in this example a suitable image stream is finally produced (only display adjustments by the final receiver). It typically already has an LMC that is different from the 5000 knit.

Readers can also synthesize HDR images or videos in the same way by consumer systems, eg, movies are combined with computer-received images in ICs such as set-top boxes, personal computers, etc. You can also understand if it can be done or how it can be used in a video conference room or the like. Im_o of FIG. 3 can be output to a television, and one of the images can be received on a Blu-ray disc via a BD player or the like.

11 and 12 show some aspects of determining CombRng. The brightness axis is given in Log units with a base of 2 to give a better correlation between human visual brightness and the impression of brightness when the image is rendered. Algorithms for determining various pixel brightness can also be formulated advantageously in such logarithmic representations, and some of the practically preferred video coding embodiments are such perceptually homogenized. It also works with brightness expression. In these embodiments, the device has already determined a good value for anc_FC. For example, the device assumes that the LDR value of about 30 knits is also good for HDR rendering. The device synthesizes the details of the composition and the brightness distribution of the two images to reach a good choice of LMC and LmiC. The device has rules selected and modified in this regard by the viewer's control settings. In the first two examples, the high dynamic range image (Im2 shown by the thick line) is the main image and the low dynamic range image (not necessarily SDR) Im1 is the PIP, for example 1/4 of the area (Im2). It occupies one of the control parameters that can show the visual control of an Im1 object or area as compared to the objects in the scene. In these examples, the device wants the appearance of HDR Im2 to be dominant, so it first determines the CombRng and placement of Im2 and then places Im1 harmoniously as a guide. In these embodiments, the complexion has already been assigned to the optimal final complexion anc_FC, so the remaining problem is to allocate the remaining brightness. This is influenced by both the color conversion functions that adjust and fit the endpoints of CombRng and the other brightness resulting from them, and the variability allowed to change the shape of these functions as a brightness impact requirement. Receive. In FIG. 11A, HDR Im2 has a histogram that already fits well into a typical scene with a face when rendered around anc_FC, i.e. the face of Im2 is normally illuminated. The long tail to black reveals that there are dark areas in the scene, for example, this is an unlit room somewhere behind the actor. In some renders some of the pixel colors are indistinguishably dark and invisible, but at least they are encoded and calculated by the device so that the final render device (eg, the TV is in its MDR range). Display adjustments and vendor-specific brightness processing in the darkest areas) can use this data appropriately. It is further found that there is a bright brightness mode HiLaIm2, which corresponds to a lamp that ideally wants to see, for example, some structure (eg, how the lamp reflects in a metal luminaire). The image Im1 to be PIP-ized has a characteristic of having a high pixel count peak HiModIm1 near white (the brightest brightness in the code). This is because a person (with a reasonably large area where a typical complexion should typically be a bright image, as can be seen from the mode near anc_FC) is clipped to, for example, near white or potentially maximum white. Because he stands in front of the big bright screen of the light box. This peak is too bright and is likely to interfere with the composition. However, in this first example, the complexion is in the normal position in both images and there are already reasonably many bright colors in the HDR image (eg under the main set of lamps and 100 than the face). Some highlights of HiLaIm2 (both modes of diffusely illuminated object), normal brightness that can be seen to have occurred in Im2, i.e. not too dissonant, so put HiModIm1 in the normal position. Is possible (ie, the same number of stops on the CombRng anchors encoded in Im1 when decoded into a linear luminance representation, in this example the face luminance is at its normal LDR position, so about It is a normal decoded value of 100 knits). This also means that Im2 highlights can be kept in their normal position (because the current HDR image brightness distribution has highlights where all matching colors happened to this image). (Because no rematching or readjustment is required to fit the profile), that is, the device does not have the upper limit of the HiLaIm2 brightness mode, or the brightness up to PB, the peak brightness of the HDR Im2 range. You can choose any more. This is not always the case, but it can be a reasonable choice unless further details are known, for example, for typical use further below the lines of the HDR image processing chain. In FIG. 11B, there is a similar situation (the same Im1 is PIPed), but this time Im2 has a different histogram (eg, a post-movie shot). Here, the faces are relatively dark (ie, when they are rendered alone without compositing with other image content), eg, rendered in 4 knits. For example, a person may be in the shadow. In this example, the device does not want to darken the face in the PIP, but also wants a more tuned darkness for the Im1 face. The device takes the face brightness of the HDR image, i.e. anc_FC, which is a one-stop setting on 8 knits, and under anc_FC (because it should retain a dark appearance to maintain the artistic intent of this movie scene). Renders a one-stop Im2 face on and (especially depending on the size of PIP) a one-stop LDR commercial Im1 face on anc_FC to CombRng (although it is still a clearly bright image portion). , Not bright enough to be too discordant). If the ratio of all object brightness of Im1 is maintained (linear mapping by anchor brightness anc_FC), the brightness of a large area of HiModIm1 will eventually drop to less than 60 knits instead of about 100. Nevertheless, if the device determines the impact of brightness (eg by calculating the contrast CHL between some reference colors of important parts of Im2 or objects such as complexion anc_FC and this HiModIm1 brightness, or space. By analysis, for example, the large bright area of Im1 can be judged purely by brightness by weighting the large and close highlight area of Im1 so as to contrast more seriously with the important central area of Im2). You might think it's too bright compared to most of the slightly darker appearance. Therefore, the device decides to reduce the brightness of HiModIm1 in this mode by one stop to 30 knits, which is not too distracting for the small PIP in the upper right corner. As shown above, the device reduces, for example, the contrast factor C for brightness conversion to CombRng above anc_FC, or based on some desired maximum position LAMX of some of the brightest possible pixels of Im1. It can be calculated and done in various ways, such as some non-linear functions that are potentially further determined by placing some diffuse white anchor brightness. It should be noted that in this embodiment, the apparatus also performs some harmonization processing in the black luminance sub-region. Since the black of the HDR image will be very deep (at least if the algorithm is applied prior to display optimization and is run with the display connected or inside the display, some displays will do this. It can be rendered, and of course the device assumes that the display actually takes into account what can be rendered with sufficient visual quality or visibility), the device also deepens the Im1 black and sets both blacks. Can be more harmonious. It can be done, for example, by widening all brightness lower than that encoded as La_Bk (some black delimiters) for Im1, eg, by doubling the local contrast. As can be seen from FIG. 11B, in this example, maintaining the brightest region of the HDR image is still useful, i.e., because this problem was mostly to adjust Im1 in the deeper falling luminance subregion, LMC. Is again determined to the maximum value needed to render Im2 faithfully to the display. It should be noted that the device was chosen to limit LmiC somewhat above the darkest color of Im2 in this example. For example, X stops under anc_FC because it determines that a color that is too dark is ignored by the composite viewer anyway. Color conversions that map to CombRng can take this into account, brightening the darkest colors and bringing them into CombRng.

FIG. 12A shows an example where Im1 is the dominant image (ie, in a brief description of the commercial). Perhaps viewers don't want the movie to get smaller when the commercial starts, but PIP will watch it, for example, when something important starts on the second channel that the viewer wants to see. Channel 2 or the first Im2 movie switch you saw can be included in a small PIP (for example, this Im1 image is the start of news that viewers don't want to miss because important news items are expected. Corresponds to). Let's assume that Im1 is an SDR image, but someone specifies that it renders very brightly with a higher dynamic range because there is room for a bright display (eg, the creator of the commercial content has the metadata. In some cases, the brightest brightness of 500 knits is rendered on a display of at least 1000 knits of PB, or how the viewer determines the brightness of the content, even if it is SDR. You may also choose a user setting that indicates whether you want to look dynamic or unobtrusive). Therefore, Rec. When decoded normally according to 709, the histogram Im1Nat can be obtained, but in reality, CombRng is acquired and the histogram Im1adj is rendered with the value of Im1 by compositing up to 500 knits (at least by the compositing device of the SDR content creator). If you fully follow the request, you will be paid by the commercial owner, for example because you are set to free content viewing mode). FIG. 12A shows a calculation embodiment, for example, a young viewer sets his system in dynamic mode. In order to be able to sharply contrast with the large bright areas coming from Im1, the device needs to set the bright lamps to, for example, 8000 nits (at least with this intermediate designation CombRng, display adjustments may occur). That is, the LMC needs to be set to 8000 knits (even if the original Im2 decoding was, for example, 4000 knits). FIG. 12B is an embodiment that contrasts with the first possibility because the viewer has set his system conservatively (because he does not want to see an overly bright image). Since this is another harmonization study that is bright enough, led by the brightness of the SDR content, the reader finds that this is a very different result from the derivation of CombRng's LMC. Since there are already so many very bright pixels from Im1, it contributes to determining or experiencing the overall high brightness of the composite image Im_o, and the device reduces the brightness of the bright HDR mode HiLaIm2. Hopefully (even if the movie is master graded, for example on a 10,000 PB_D grading display, and communicated as a 5000 knit PB_C HDR image, for example with the SMPTE2084 EOTF Luma assignment), the result is an LMC of only 2000. The contrast between the brightest lamp in the HDR image and the bright wide area of Im1 (2 stops instead of 5 or more) is reduced, i.e. the HDR movie loses some of the punch when the PIP pops up, At least the whole image is not overly bright. Viewers may desire such a compositing aspect.

Also, if the range CombRng is specified before anc_FC, the device sees various aspects of the brightness of the pixels and objects in both images, eg one or more corresponding typical brightness. Can be done. For example, the device looks at the percentage of pixels in both images that are at least k times the intermediate gray or at least 1 times the peak brightness of the decoded image, and from these values in both images, some typical. A good value of brightness, at least what is the maximum brightness possible to start with a synthetic LMC. If embodiments of the device use one typical value of the brightest object brightness for each image, how they relate when mapped to CombRng, ie what brightness contrast is there, for example. It is possible to determine whether or not, and propose a more harmonious contrast. Of course, the mid-gray representatives of most colors in the scene (mid-gray can actually have a reflectance of 10% for some typical lighting in the scene, but for example, the pixel mode, multi-value, or average of the image. Further values such as (which can also be used to indicate) determine what the contrast between various regions in the composite image will be, for example the brightest part of Im2 and the intermediate gray part of Im1. Helps to choose a harmonious relationship between them. For example, some device embodiments may always select some typical contrast values along with brightness, darkness, and average areas assigned to some typical brightness subregions of CombRng. Consists (of course depends on whether the LMC is about 1000 knits, about 2000 knits, about 5000 knits, or about 10000 knits), these values are, for example, night view, bright news program, night nature, etc. Deviations can occur when some characteristic image types are detected (eg, mapping k stops higher at CombRng). When determining a good anc value for a good face or gray brightness (or bright light region), the device can measure what viewing environment the viewer is sitting in. For example, a forward-looking camera in a television can summarize the brightness characteristics of the environment with one or more characteristic values, such as the anchor_FC value, and the contrast required for other brightness compared to the anchor brightness, etc. It can be used in equations to scale (eg, move one or more stops up and down). For example, a camera can see an observer (by face detection) sitting on a coach under some local lighting, typically a less illuminated wall or other object. A second region can be determined. If the embodiment has to derive a single characteristic environmental brightness value, local lighting typically comes from above and has little effect on the experience of the environment with the viewer (eg brain). Is established as deep black in the environment and the display forms just part of it), and the brightness of the surrounding walls is calculated to be high (eg a * ambient average_L + b * local average_L). From these various descriptions, in various embodiments, the CombRng, one or more anc brightness, the brightness mapping function shape, and, in fact, the entire harmonious output image tailored to each situation. It should be clear to the reader that there are various ways to reach.

FIG. 18 is a general typical possibility of a determined color conversion function FF applied to at least one luminance of an image or image portion to be mixed (or some or all of them having the respective function FF_x). Only explaining sex, from these examples, the reader can imagine further possibilities according to the general concept of the present invention and its various embodiments. For example, FIG. 18a shows two simple aspects of determining an FF_1 function consisting of two parts (in this example it ends at the exact end of the range but may deviate in other scenarios). If the output result luminance (LF1_o), which applies the function to the luminance of the input image equal to the source anchor (L_SA1), must be exactly equal to anc, the function is extracted from the internal program of this device. If the output luminance LT2_v can be in the neighborhood, we can determine two similar and somewhat different segment functions FF passing through a point with a Y value of anc / k, where k is 1.5, 2, 3 or It is 4.

FIG. 18b shows that complex shaped functions can be designed around intermediate gray anchors for normal / dim lighting, for example with an S-curve for black. The synthesizer typically has a re-grading luminance function (eg, a fine-grading function) communicated according to the coding principles of the present invention to obtain one of the dynamic range images from other actually received ones. ), It is possible to propose such a brightness mapping function unit. If the synthesizer anticipates such S-type behavior against black, the device appears to have an important object area in the center of the black that needs to have good strong contrast, most Understand that deep black seems to be almost negligible as the functions indicate that most of them are clipped to the smallest black (0, or whatever the smallest black occurs). be able to. Thus, some smart compositing program embodiments follow this behavior with their determined FF shape, but the mapping is at 500 knit CombRng rather than between the 5000 knit range and 100 knit SDR range of master HDR brightness. So the function is reformed (thus we need to transform some black according to a similar need to roughly ensure that the middle black is rendered best, but this is a bit different). Similarly, for the bright function, the synthesizer can determine to follow the shape details corresponding to the brightness subrange of the critical image object more accurately or to a lesser extent. FIG. 18c shows an example in which a synthesizer controls, for example, some "intermediate range" contrast C of an SDR image to be composited, in which the device leads to SDR pixel rendering that is too bright in this embodiment. Therefore, it is decided not to bother to extend the upper part of the FF function to the maximum value of CombRng (of the output image Im_Cmb). (For the reader, the HDR image brightness histogram typically has a far-spread lobe with the brightest lobe containing only 20 x 5 pixels of, for example, 20 small mirror patches, and the SDR image is clipped, for example. Even potentially large image areas of white outdoor landscapes contain tightly bundled histograms with many intensities up to PB_C = 100 knits, and certainly those of composite pixels in relatively dim or dark HDR movies. It should be understood that this area should not be rendered as bright as PB_Comb = 2000 knit so as not to destroy the joy of watching the movie). Early synthesizer embodiments have only two such perceptible pre-programmed rules in software or hardware, but further embodiments in the future will eventually have a balanced composite image. It should be understood that complex analyzes can be performed to do this. Therefore, various different system embodiments are in an elegant and less cumbersome manner, by choosing a good value for the anchor brightness of the content long before the (unknown) mixing actually occurs. Can be designed around a core device that correctly handles the mixing of images or videos so that humans can identify what they need. Various embodiments can be implemented in the mixing device, for example the device uses the same type of anchor (but some devices have decided to use different anchor types and coordinated mapping of the two images. Can also be determined to convert from those anchor types to the available brightness of the mixed dynamic range). For those of us, in practice the former situation can be carried out, for example, in a mixer operating in a kind of slave mode, at least for example the main video defining what the critical facial anchor brightness is. Check to establish what is good face brightness in the mixed dynamic range (eg considering the final rendering characteristics) and important faces in the first image (even if they do not contain faces). It can be understood that the brightness of the second image is transformed to match the brightness of (ie, how it was represented by the composite dynamic range). An example of a second strategy in which a mixer is guided is representative of the overall typical brightness (where the rendering of a photo must be (almost) compliant, for example, when the user is watching the photo on a television. By defining the anchor brightness as a single value, and typically, for example, a composite dynamic range that is not too bright (for a soft display) before other image data comes in and is harmoniously mixed. Due to the dynamic range of the presentation already defined by the photo brightness mapped to, the photo is currently using almost half of that range, ie the composite dynamic range for bright pixels if the second content is desired. Despite some margin, we set some criteria for it (by specifying that almost all of those pixels have a brightness lower than PB_Comb_PhotoView / 2). Mixing of the second image content (the first content is a photo that has already been mapped according to a determined luminance mapping) is generated by the device by looking at the anchor value of the content (this is the photo viewing Different from the anchor type, but related to it by the calculation of the mixing device, eg, anc_face_inmix = 1.3 * anc_photoviewaplic), eg, establishes a photographic viewing composite dynamic range with brightness between 0 or k knit and PB_Comb_PhotoView. It is very convenient to have a general single (all situations are possible) HDR image content mixing system that can handle different situations, and the decisions in different types are best suited for the situation and when But it depends on what is already known to the device.

For example, if the final rendering system is not yet known, the best thing a mixer can do when it needs to reconcile two image contents is to at least see how they are best mixed. There is (eg, whether one of the images has an excessive (possibly large) high brightness area and needs to be darkened to enjoy the other image content). In that case, the final display optimization stage (eg in consumer television) is even more complex, taking into account the images already shown how they should (generally) be optimally mixed. It is necessary to perform appropriate optimization. However, in embodiments where the rendering environment is already known, the mixer performs the final appearance (ie, the visual quality needs of CombRng determined on a large factor display, and eg different mappings, where there is a particularly strong demand. By doing so, it balances against some sacrifice by completely sole rendering one of the images). When rendering capability has a major importance in determining CombRng, typically at least some typical considerations are used, for example with respect to the expected properties of HDR images (eg software rules for synthesizers). Most of the "general" typical HDR images are probably present, for example below 500 knits (+-), and bright objects can be rendered, for example, when assigned to display brightness up to 1500 knits. Decide that it must be). If the display has the ability of only PB_D = 1000 nits, for example, it corresponds to some false non-optimal rendering of good HDR images (ie the increase in brightness of 1400 nits is a little too dark). In such cases, the actual PB_D of the available display has a significant effect on the determination of CombRng's PB_C for mixing the image content prior to rendering, but the compositing device has a CombRng PB of less than 1000 knits. It can be determined to be somewhat larger, eg 15% higher or 1150 knit (and apply the final display adjustment of the mixed image to map it to the renderable range on the display). Of course, in an ideal better embodiment, all known factors (ie, the need for ideal content rendering, the current need for content rendering, are also taken into account, taking into account the optimization of CombRng and the appropriate anc points within the synthetic dynamic range. Balance all practical requirements and limitations of the HDR image mixing phase). The optimal anc point determination is typically to look at what is available (but possible) after previously defining the optimal CombRng (but as taught, eg images are semantically). What kind of content is included, and in some embodiments, various other factors, such as what the end viewer would expect from such image content). Thus, different embodiments have different anc type options and are determined in at least one of the source images to be synthesized, typically the main image (ie, such as good complexion). Anc brightness in CombRng of the same type as shown in at least one of the images to be combined, but it should be understood that in CombRng optimal in the compositing situation, each image is different from rendering to itself. The closest thing to (that is) is selected for device on-the-fly determination of what the perceptible anc value is given all the physical characteristics of the situation (eg pixel brightness distribution and display capability of the content). Through the system, anc can be established in various ways across fixed settings in the device that always relate everything to proper facial brightness (if specified as the source image anchor type). In the latter type of embodiment, for example, the complexion must be optimally determined with a given specific rendering limit, for example, how dark the display will display dark objects well, and the brightness of the face on it. It is good when dropping is especially important for the face in the dark areas of the source image. Therefore, for clarity, the two images currently input are because some embodiments of the device generally only need to be associated by the device in order to operate the device. Although it can be defined by the complexion type anc brightness, it can be operated by determining some anc value (eg good average brightness) (ie, the device was established independently, especially first). If the face is, for example, 50% brighter than a sufficiently usable ank value, it is judged that a sufficiently good image can be obtained, so that the typical face brightness of the input image is the "average type" anc of CombRng. Can be mapped near the value). Therefore, the anc type and CombRng of the source image should be related or relevant. Anc can summarize the complexity of all colorimetric analysis of an image in an optimal and elegant manner, and in many situations only one adjustment point for compositing is needed anyway. For completeness, the dynamic range anc value means that it is not the brightness of the endpoint, but rather the information added to it. With respect to establishing a brightness dynamic range, one of ordinary skill in the art will determine the amount of relevant image content in the dark areas of the image, dark, in embodiments where the high brightness is determined and the minimum brightness is not systematically zero knit. Understand how low brightness is established depending on the balanced needs of known factors such as the ability of the rendering system to display colors.

The components of the algorithm disclosed herein are, in practice (all or part) software that runs on hardware (eg, application-specific IC components) or special digital signal processors or general processors. It may be realized as.

Those skilled in the art will appreciate that the steps (as an optional option) of the components and methods that can be improved as optional and feasible in combination with other components correspond to each means of the device and vice versa. Should be understandable. In this application, the word "apparatus" is used in its broadest sense, that is, a group of means that enable the realization of a specific purpose, and therefore, for example, an IC (a minute circuit part), a dedicated electric appliance ( (Electrical appliances equipped with a display, etc.), or a part of a networked system, etc. are possible. In addition, since "arrangement" is intended to be used in the broadest sense, it may include a single device, a part of a device, a group of (a part of) a collaborative device, and the like. Good.

The meaning of a computer program is any physical realization of a set of commands that enable a generic or dedicated processor after a series of load steps (which may include intermediate conversion steps such as translation into an intermediate language and the final processor language). By explicitly including, it is understood that a command may be entered into the processor to perform any of the characteristic functions of the invention. In particular, a computer program may be realized as data on a carrier such as a disk or tape, data existing in memory, data propagated via a network connection (wired or wireless), or program code on paper. Good. In addition to the program code, characteristic data required for the program may be embodied as a computer program.

Some of the steps required for the operation of this method may already exist as a function of the processor, such as data input and output steps, instead of the description in the computer program.

It should be noted that the above embodiments are not limited to the present invention but are described. Those skilled in the art can easily realize the mapping of the examples presented to other areas of the claims, but for the sake of simplicity, all of these options are not described in detail. In addition to combinations of elements of the invention, such as combinations within the claims, other combinations of elements are possible. Any combination of elements can be realized with a single dedicated element.

No reference symbol in parentheses in the claims is intended to limit the scope of the claims. The word "comprising" does not preclude the existence of elements or embodiments not listed in the claims. The word "a" or "an" before an element does not preclude the existence of a plurality of such elements.

Claims (15)

A device for synthesizing two images or two videos of a plurality of images, wherein one of the two images or two videos is an image or video with a high dynamic range.
The maximum brightness of the two images of the plurality of images or at least one of the two videos or the video, and a display for rendering the two images or the two videos of the plurality of images. A dynamic range establishment unit that establishes a combined luminance dynamic range characterized by at least a maximum brightness determined based on at least one of the peak brightness of the above, wherein the dynamic range establishment unit is the composite luminance dynamic range. A dynamic range establishment unit, further including a brightness anchor determination unit that determines the anchor brightness in
A color conversion unit that performs at least a luminance conversion on at least one of the two images or videos, wherein the color conversion unit is a first source that supplies the first image or video of the two images or videos. The color conversion unit includes a source anchor brightness reading unit that reads at least one source anchor brightness from the metadata from the first image or video, the color conversion establishing the color conversion applied to the first image or video. A color conversion unit that depends on the value of the source anchor brightness by having the property that the source anchor brightness is mapped to the output brightness in the vicinity of the anchor brightness.
An apparatus including an image compositing unit that synthesizes the two images or two videos of a plurality of images to form at least one output image. The color conversion unit has an output brightness determined as a result of applying the color conversion to the input brightness of the color of the pixel of the first image or video equal to at least one source anchor brightness. The apparatus according to claim 1, wherein the color conversion is determined so as to be. In the color conversion unit, the brightness ratio obtained by dividing the second output brightness determined as a result of applying the color conversion to the second input brightness by the output brightness determines the second input brightness by the source anchor brightness. The apparatus according to claim 2, wherein the color conversion is determined so as to be a constant multiple of the ratio divided by. The color conversion unit includes a deviation determining unit that determines a luminance offset based on at least one of the source anchor luminances, and the color transforming unit is of the first image or video equal to at least one of the source anchor luminances. Claims 1 to 3 determine the color conversion so that the output brightness determined as a result of applying the color conversion to the input brightness of the pixel color is equal to the sum of the anchor brightness and the brightness offset. The device according to any one of the above. The color conversion unit reads at least one second source anchor luminance obtained from a second source that supplies the second image or video of the two images or videos, and the deviation determination unit is at least one. One of the determining the luminance offset based also on the second source anchor luminance apparatus according to claim 4. The color conversion unit said to map the first image or video from the dynamic range associated with the coding of the first image or video to a dynamic range having a peak brightness different by at least a multiplication factor of 2. The color conversion applied to the first image or video is also established or the color conversion depends on the first color remapping function that specifies the change in the brightness distribution of the object in the first image or video. unit, the second image or video, from said second image or dynamic range associated with the encoding of video, the second image to be mapped to a dynamic range of at least multiplication factor 2 by different peak brightness Alternatively, any one of claims 1 to 5, which also relies on a second color remapping function that specifies a change in the brightness distribution of the object in the video to establish a color transformation applied to the second image or video. The device described in the section. The dynamic range establishment unit according to any one of claims 1 to 6, wherein the dynamic range establishment unit establishes the combined luminance dynamic range depending on the luminance present in at least one of the two images or videos. apparatus. The dynamic range establishment unit establishes the combined luminance dynamic range depending on the peak brightness of the display on which at least one output image is rendered, and preferably also on the brightness characteristics of the viewing environment. Item 7. The apparatus according to item 7. The brightness anchor determination unit comprises the combined brightness dynamic range, the brightness of at least one image or video of the two images or videos, the peak brightness of the display on which the at least one output image is rendered, and the brightness characteristics of the viewing environment. The device according to any one of claims 1 to 8, wherein the anchor brightness is determined depending on at least one of the two. A method for synthesizing the two images or two videos of a plurality of images, wherein one of the two images or two videos is an image or video with a high dynamic range.
For rendering the maximum brightness of the two images of the plurality of images or at least one image or video of the two videos, and the two images of the plurality of images or the two videos. A step of establishing a composite luminance dynamic range characterized by at least a maximum brightness determined based on at least one of the peak brightness of the display, wherein the establishing step determines the anchor luminance in the composite luminance dynamic range. Steps to establish, including more steps,
A step of applying a color conversion consisting of at least a luminance conversion to at least one of the two images or videos, wherein the color conversion supplies a first image or video of the two images or videos. Established based on the value of the source anchor brightness read from the metadata from one source, the color transformation applies, which has the property that the source anchor brightness is mapped to an output brightness in the vicinity of the anchor brightness. Steps and
A method comprising the steps of synthesizing the two images or two videos of a plurality of images using their pixel colors in the composite luminance dynamic range to form at least one output image. In the color conversion, the output brightness determined as a result of applying the color conversion to the input brightness of the color of the pixel of the first image or video equal to at least one source anchor brightness becomes equal to the anchor brightness. 10. The method of claim 10, wherein the method is determined so as to be equal to or equal to the sum of the anchor brightness and the determined difference. In the color conversion, the brightness ratio obtained by dividing the second output brightness determined as a result of applying the color conversion to the second input brightness by the output brightness is the brightness ratio of the second input brightness by the source anchor brightness. The method of claim 10 or 11, wherein the method is determined to be a constant multiple of the divided ratio. The color conversion of the second image or video of the plurality of images is at least one second source anchor luminance obtained from a second source that supplies the second image or video of the two images or videos. The method according to any one of claims 10 to 12, which is determined based on the above. At least one of the color conversions applied to each of the at least two images or videos is at least twice or less than the corresponding dynamic range in which each image or video is encoded. The method of any one of claims 10-13, which is determined at least in part based on at least one receiving function indicating how it is transformed into the dynamic range of. At least one of the composite brightness dynamic range and the anchor brightness determines the brightness distribution-dependent property of the object in at least one of the two images, and the brightness characteristic of at least one of the two images. Determined based on information to be summarized, the dynamic range of the display on which at least one composited output image is rendered, and at least one factor from a set of brightness measurements for the viewing environment in which the output image is viewed. The method according to any one of claims 10 to 14.

Images