That State of HDR Video in 2019

View fullsize

That State of HDR Video in 2019

April 19, 2019

If you want to feel like a real underachiever, read some of what’s been published lately in the SMPTE Motion Imaging Journal on the subject of High Dynamic Range (HDR) video. These writers are brilliant engineers and academics who’s work comes together to standardize all these manufacturer’s incongruous format madness; the confounding mathematics at the crux of it I have my head around just enough for a cursory understanding. I am but a mere, end-user ‘splainer. I understand these things well enough to read a white paper or two and then perhaps break them down into slightly more digestible laymen’s terms. Or do I? We shall see.

First, there’s HDR video which is easily conflated with HDR imaging. Both have actually been around for quite some time, decades; but HDR imaging has in my opinion, come quite close to ruining digital photography, whereas HDR video, with five “competing” formats (don’t call it a format war) and some very nice, affordable consumer TVs, is now officially a “thing.” Though an oft misunderstood thing, hence this attempt at clarification.

High Dynamic Range video, herein referred to as HDR, at the quickest glance is brighter brights, deeper blacks, and richer colors—at least 10-bit color depth as opposed to 8-bit in Standard Dynamic Range video, SDR, so less banding and artifacting, along with a wider color gamut (more reproducible colors.) In other words, “prettier pictures.” This is true today but it certainly didn’t begin this way.

My first experience with HDR was let’s see? Perhaps five NABs ago? 2014? Must have been. They all blur together in a haze of gear, jargon, people who’s names I will never remember, booze, steak, lack of sleep, and madness. At Dolby’s booth that year I saw perhaps the brightest monitor I had ever seen. Next to it was a placard that read something to the effect of, “Dolby Vision Test Display High Dynamic Range Imaging 2000 nits, display technology capable up to 10,000 nits.” I’m assuming you know that nits are shorthand for candelas per meter squared, a measurement of display brightness, but if you need refresher, read this.

The monitor above is not the Prototype HDR monitor Dolby had on display in 2014, rather the PRM-4220 Reference monitor which became available the following year. 42”, 1920x1080, 600 nit LCD, 48 nit film projection emulation mode, 100 nit CRT emulation mode, and tons of other bells and whistles. This information was lifted right off their current website. If this monitor is actually the 4000 nit “Pulsar” then their website is badly in need of an update. All Google searches for “Dolby 4000 nit Pulsar” lead to this page but I know for a fact that the monitor exists because I’ve seen one. Not super helpful, Dolby, especially if you’re trying to sell this insanely expensive piece of hardware.

For context, a SDR reference monitor calibrated at “studio levels” is 100 nits. Literally the brightness of everything you’ve been watching your entire life prior to HDR has fallen somewhere between 48 nits (theatrical film projection), 120 nits (a very bright TV), and 300-500+ nits (LCD advertising displays.) That said, digital images displayed at 1000 nits are 20x brighter than film projection, 10x brighter than your average TV. And with Dolby Vision’s theoretical peak brightness of 10,000 nits, that’s 100x brighter than most of the digital images that currently surround us.

The wide spread proliferation of HDR, in my opinion, constitutes nothing short of a major paradigm shift in digital imaging.

For the initial demo in 2014, Dolby was showing some very weird and unnatural nature videography on their 2000 nit prototype display. I remember it hurting my eyes a bit. It just looked plasticky and wrong. Like bad HDR photography in motion.

View fullsize

Lifted from Google Images. Search criteria “bad HDR photography” No credit. There is nothing in our human visual system that looks even remotely similar to this.

2000 nits was bright so 10,000 would surely melt your face off. That’s enough lumens to light up your street from your living room window. So from this initial offering, I was not only skeptical but a little disturbed.

Cut to a few years later and HDR was the “new 3D,” literally everywhere at the NAB show, but already looking much better. There were tons of new consumer TVs on display at more “comfortable” levels between 400 and 600 nits. By now Sony had released the BVM-X300 UHD HDR reference grade monitor so colorists had a viable facility-level tool to work with and were quickly getting the hang of the subtleties of the HDR grade. Turns out most of the time, less is more with HDR. The images I saw at the 2016 show appeared neither hyper-real nor unnatural but instead—vibrant, punchy, clear, and dynamic. Like life. But on TV. Yeah, LG marketing. Suddenly, I was extremely interested.

Early in 2017, I picked up a Playstation 4 Pro which can output both UHD and HDR to a capable display. After playing a few titles on my sad, tired, old 3D LCD which was losing contrast after 5 years of use, I tossed it in my building’s dumpster and picked up the LG OLED55C6P, the last TV LG will ever make that is passive 3D, UHD, and HDR. All boxes ticked. If you weren’t already aware, in North America, digital passive 3D is effectively dead. No TV’s sold here include this feature anymore. I still love it and had to get a TV while I still could. Wasn’t cheap but totally worth it.

A related side note. And a word of caution for OLED TV owners. To satisfy my Trump Derangement Syndrome (TDS), I watch about 30 mins of MSNBC every morning while drinking my coffee. That’s not much, but enough that in just over a year, MSNBC’s bright red lower third burned into my TV. I did what I’ve discovered to be the most effective method of retaliation and railed against the manufacturer on Twitter. Much to LG’s credit, I had a brand new TV in a week. Way out of warranty. Needless to say, I was shocked. And extremely pleased. Thank you for doing the right thing, LG.

I was pretty skeptical about the curved OLED but in a small living room where you’re close to the TV, it’s actually quite a cool effect.

I must say, the PS4 Pro Enhanced (Sony garble for UHD HDR) gaming experience is nothing short of revelatory. This of course led to the purchase of a UHD Blu-ray player, which led to a collection of UHD Blu-ray’s (Blade Runner 2049...so so good), which led to way more movie nights at my place. The difference between viewing any kind of content in muddy 100 nit HD SDR and bright, vibrant 600 nit UHD HDR with those rich OLED blacks is like the difference between Dunkin Donuts House Brew and the most expensive pour-over at Blue Bottle. There’s no going back. Even 1000 nit smart phones are starting to show up. The HDR flood gates are open. And that’s a great thing for both content creators and end users alike.

God of War in UHD HDR. Stunning imagery. Surprisingly boring game.

So, how do I know if my TV is displaying HDR correctly? First, turn the feature on for the correct HDMI input (it’s off by default on most TVs). On LG TVs it’s intuitively called “HDMI ULTRA HD Deep Color” (why?), resist the urge to buy that “high speed 4K” HDMI cable. You don’t need it, any old HDMI 1.4 cable is good enough. Plug in your HDR device, turn it on, and you’ll know you’re good if you get a little pop up bubble in the upper right-hand corner that says “HDR” or “Dolby Vision.” Like this—

At this point, I’m completely sold on this technology and see HDR soon becoming synonymous with UHD and all higher than HD resolution video formats. I can say that in the post delivery of the media business, there’s still a huge distinction between SDR (delivered both in HD and UHD) and HDR (UHD deliveries being mostly universal.) I see this distinction going away in the not too distant future. That said, we can look forward to even more resolution, more nits, more eye ball busting content, thinner displays, wearable displays, holographic displays, AI’s that treat you better than your partner, thousand year storms, dust bowl belts, and beachside properties in Vermont. Go read The Uninhabitable Earth. Actually don’t. Unless you want to never stop being depressed for the rest of your life.

Let’s start at the top—keeping in mind the old adage, “Garbage In, Garbage Out.” That said, let’s frame this as a matter of Source and Destination.

A SDR TV (aka normal HD TV) can display about 6 stops between the darkest dark and the brightest white. That’s not much. A HDR TV can display 13 stops or more. If your source imagery is low quality, low dynamic range, you can’t magically turn it into HDR. However, with this wonderful up/down/cross/SDR/HDR device—the AJA FS-HDR with Colorfront Engine Video Processing, it’s shocking what can be done.

View fullsize

This device can’t polish a turd into a gold ingot but a sharp, well exposed, log-encoded 10-bit HD image can be upscaled and its dynamic range expanded into an extremely convincing UHD HDR picture. The side-by-side demo at AJA’s booth I saw this year at NAB on the 31” FSI HDR monitors was very impressive. AJA has the hardware expertise and Colorfront has the color science down. A very smart and effective partnership.

So in order to make viewable HDR video for HDR TVs, you must start with a High Dynamic Range video source. Fortunately for us, all these digital cinema cameras that have been capturing rich log-encoded images for nigh on the past decade, with their 14 stops or more of dynamic range, are already by default, HDR. So all this time we’ve had HDR, just no way to display it properly. Until now.

Whereas before, all this lovely range had to be compressed into the tiny 6 stop SDR 8-bit bucket for display on crummy Rec. 709 HD TV’s. Now, these modern camera’s 14-18 stops can be utilized and displayed correctly on UHD HDR TVs, with the assistance of a DIT who understands HDR exposure and a colorist who knows how to grade for HDR of course. Forget the DP. What do they do? Bad joke. The truth—if the producers will pay their rate they get to sit in on the HDR grade and finesse their work. If not, the producers, director, and colorist will finesse their work.

I should now also mention that well-exposed color negative qualifies as an HDR source; however, its inherent grain has proven to pose an aesthetic problem in HDR so is not an ideal source for this medium. That hasn’t stopped people from trying though. And still trying.

Format 1—Dolby Vision

View fullsize

Very “enter the Void.” One of my favorite films of all time. What does that say about me?

I really like Dolby Vision. In my opinion, it’s a better reproduction of human vision and is far more aesthetically pleasing than any of the other available formats. There are several reasons for this. First, Dolby makes substantial investments in R&D. They became the premium brand in cinema audio through R&D and very cleverly realized they could extend this brand equity into picture as well. The other reason is the driving force behind the imaging technology—scene-specific dynamic metadata. One size does not fit all with HDR and in Dolby Vision’s EDR (Extended Dynamic Range) metadata, luminance levels are set to the creative’s desired output for literally every cut. Is this a lot of work for the colorist? Absolutely. High and low luminance levels must be set for every single shot in the edit. But does it result in a product that producers and creatives are more excited about? Definitely. Not only is Dolby Vision more focused on the creatives but from the resulting Dolby Vision deliverable’s ancillary metadata, 8-bit SDR, 10-bit HDR (aka HDR10,) and potentially more extractions can be seamlessly derived. More on this in a bit.

Not to come off as a total cheerleader but I’m extremely impressed with the technology because it’s smart and focused on both the experience of the content creator and the end user. And when I’m impressed you’ll know, just like when I’m not impressed. If you’ve never seen a film at a Dolby Cinema, what are you waiting for? There is literally no better way to see a tentpole movie in 2019 than on a Dolby screen. It’s like a 60’ 4K HDR OLED complete with your own almost completely horizontal comfy recliner.

For the rest of the of the article I’ll be focusing most on Dolby Vision mostly because I like it and I think it has the brightest future (no pun intended.) Reason being, as I just mentioned, it’s focused on creative intent instead of the profits and ease of implementation of electronics manufacturer’s consortiums. Dolby needs to make their money too and good work doesn’t come free so the Dolby Vision licensing fees are the impetus of all the other HDR formats. More on this in a moment.

Dolby Vision aka Perceptual Quantizer aka PQ aka SMPTE ST 2084. From the truncated wikipedia article— “is a transfer function that allows for the display of high dynamic range (HDR) video with a luminance level of up to 10,000 cd/m2 (nits) and can be used with the Rec. 2020 color space. PQ is a non-linear electro-optical transfer function (EOTF). On April 18, 2016, the Ultra HD Forum announced industry guidelines for UHD Phase A, which uses Hybrid Log-Gamma (HLG) and PQ transfer functions with a bit depth of 10-bits and the Rec. 2020 color space. On July 6, 2016, the ITU announced Rec. 2100, which uses HLG or PQ as transfer functions with a Rec. 2020 color space.”

Words, words, words. Not particularly helpful. Here’s some more—MaxCLL (Maximum Content Light Level), MaxFALL (Maximum Frame Average Light Level), 12-bit quantization, Barten Threshold, etc. How familiar do you need to be with these terms? Not at all unless you’re mastering in Dolby Vision.

Here’s really all you need to know—

Dolby Vision is a delivery format both for broadcast (mezzanine/IMF) and mastering and NOT an acquisition format. It is Dolby’s proprietary implementation of the PQ Electro-Optical Transfer Curve (EOTF). It supports both BT. Rec. 2020 and DCI-P3 primaries, color spaces, and white points, as well as 12 bits per color channel for both home and theater.

Though it mentions a target of 10,000 nits, that’s more of a hypothetical with a current target of 1000 nits as standard practice. Most consumer HDR TVs are 400-750 nits. As I previously mentioned, what makes the format special is the inclusion of dynamic metadata which is output at the time of the HDR grade in the form of a sidecar XML file, without which the scene-specific HDR (aka EDR Extended Dynamic Range) cannot be properly decoded and displayed. Included in additional metadata are instructions on how to extrapolate 8-bit SDR video, generic 10-bit HDR video at various nit levels 400, 600, and 1000 for example so that the video correctly maps to any consumer device. This is the big deal for content creators who would like their work to appear as intended.

And so this is why there’s a licensing fee in order author content in Dolby Vision. The manufacturer of the display or device or the finishing software utilizing Dolby’s proprietary technology pays this licensing fee. Not the finishing facility, not the creative, not the end user. Very important to draw this distinction.

Previously in finishing you had to use the Dolby CMU (Content Mapping Unit), a standalone piece of hardware that stood between your grading software and HDR display. This functionality is now rolled into most, if not all high-end finishing softwares. For example, the current version of Baselight has multiple versions of the software CMU for multiple nit targets. In the earliest days you also had to finish on Dolby’s monitor, the Dolby PRM-4220 but they’ve now qualified the Sony BVM-X300, the newly announced Sony BVM HX310, and the Canon PD-V2420 displays. This information could be dated. There are likely more by now.

So for the content creator, in my opinion the advantages of authoring in Dolby Vision are fairly obvious given its flexibility and with its luminance mapping metadata, your content should appear as intended across most devices which is extremely appealing. There are some peculiarities about the HDR grade in that it’s all about the “trim passes.”

Ideally, you first grade in HDR at 1000 nits, most likely on a Sony BVM X300. I’m not a colorist or a DI producer so I’m merely relaying information that I’ve heard in speaking with these people. You get your HDR grade looking really good, export your HDR deliverables and ancillary metadata, STOP. Switch the BVM X300 over to 100 nits, check that your RGB primaries and white point are identical to that of the HDR grade and then go back do your your SDR “trim”, essentially grading out any part of the picture that exceeds 100 nits, matching it as closely as possible to your HDR grade, the two are sometimes done in tandem, with a SDR and HDR monitor side-by-side, though I’ve heard this can blow the creative’s minds because the SDR will always look terrible compared to the HDR and for now, 95% of the viewing public will sadly never have the treat of the seeing the HDR ;(

Here’s Netflix far less verbose take—

Do I grade HDR first or SDR first?

The HDR should be graded first, followed by an SDR trim pass. Since the SDR will depend on the HDR grade, the creative approval of these should happen in parallel, rather than one after the other, so that both can be simultaneously adjusted and approved.

But instead what often happens is the opposite, which again, I’m not a facility person, I’ve just heard this. Because the audience for HDR right now is so small, the focus should really be on the SDR grade so do that first and then build off that to create the HDR grade. In this case there is no trim pass because there’s nothing to trim. You’re starting small and then going big. In this case, HDR while still important, is more of an afterthought. I can see the logic in this because a bunch of creatives huddled around a 30” HDR monitor fawning over how awesome the HDR looks and then realizing that hardly anyone but them will get to see it that way. Makes sense to focus on how the bulk of your audience will experience work. However, there is huge problem with this in that if there’s no trim metadata, there’s no way to extract the SDR from from the Dolby Vision XML. This process would essentially leave you with broken metadata. I can’t imagine any studio accepting a deliverable generated from this workflow. This would only work for HDR10.

For authoring HDR IMF bundles for streaming (to my knowledge no one is doing over the air, satellite, or linear cablecast HDR), Colofront Transkoder is the best solution right now (also a phenomenal software upscaler.) I spoke with an engineer from Filmlight at NAB this year and proper IMF output is coming to Baselight soon but if your finishing software can’t make a Dolby Vision IMF, this is one way to make an intermediary (or a master for that matter.) Here’s another way.

Dolby Vision file sets are PQ EOTF, Rec. 2020 color space, 12-bits per color channel encoded in padded 16-bit Tiff image sequences. And of course, being Tiff’s with no file headers, these image sequences cannot include their own metadata so frame numbering must correspond to program timebase/timecode precisely. And without the sidecar XML file, you don’t have Dolby Vision. Take these Tiff sequences and XML into a Clipster or Transkoder and voila, Dolby Vision IMF ready for streaming. Or just online in those softwares. Many ways to skin the cat.

Where can you see Dolby Vision content? Currently on Netflix, Amazon Prime Video, iTunes, Vudu, and some UHD Blu-ray titles. And of course in Dolby Cinemas.

Netflix being Netflix with their open source white papers and delivery specs has some interesting content on the topic at https://research.netflix.com/ that’s worth digging around on.

If you want more, here’s Dolby’s rather thin white paper on the technology.

And the Dolby Vision Workflow Guides for content creators.

Format 2—HDR10

Horizon Zero Dawn displayed in HDR 10. This is a truly phenomenal game that comes to life in UHD HDR. Hard to put down once you get going.

HDR10 aka “Generic 10-bit HDR” aka “Static HDR” aka “One-Size-Fits-All-HDR” aka “Only-Looks-Good-On-Video-Games-HDR” aka in LG parlance “HDR Pro” (why?). HDR10 is the product of an Asian electronics manufacturer’s consortium that didn’t recognize the advantages of Dolby’s dynamic scene-specific metadata and didn’t want to pay the licensing fee. Guess what? Most of them are on-board now and the new crop of LG and Sony TVs include both Dolby Vision and HDR10. There isn’t a whole lot to say about HDR10. It utilizes PQ EOTF and decoding metadata can be either embedded or in a sidecar file depending on the file type. It’s a rather one size fits all approach that spreads however much dynamic much range is encoded in the HDR video signal across however many nits your TV can display at a fixed level. The darkest dark and brightest bright in whatever scene will always be displayed 0 nits and 600 nits respectively. For video games, this doesn’t seem to matter much. For example, PS4 Pro Enhanced games on my 600 nit HDR LG I’d say reproduce about 12 stops, twice as many as a conventional HD TV. Sometimes HDR10 looks good, sometimes not so good. It can be particularly brutal in the highlights. For HDR10 metadata all you really need is the colorspace, and aforementioned static MaxCLL and MacFALL information. That’s all I’m going to say about it. But it does lead to an interesting segue…

Format 3—HDR10+

WTF is HDR10+? Ask Samsung. In a nutshell, it’s HDR 10 with the inclusion of dynamic metadata (à la Dolby Vision) that they developed for Amazon Prime Video to get around paying the Dolby licensing fees. It’s 10-bit as opposed to 12 and I don’t know if its authoring potential is as powerful and flexible as Dolby’s. For some reason I kind of doubt it.

Format 4—Sony HLG (Hybrid Log Gamma)

Hybrid Log Gamma aka HLG HDR is an odd duck in that it is both an acquisition and delivery format. It was developed by the BBC and NHK and implemented by Sony to address the problem of live broadcasting HDR over existing infrastructure. On paper HLG HDR makes sense but there a lot of limitations and probably the only place you’re ever going to find it is in Sony TVs and in Sony products. The company touts it as “Instant HDR” which makes it similar to HDR 10 in its one-size-fits-all approach. What makes it hybrid though, from Wikipedia—HLG defines a nonlinear transfer function in which the lower half of the signal values use a gamma curve and the upper half of the signal values use a logarithmic curve. In other words, the bottom half of the signal is SDR, the top half is extended range. If your display can pick up that portion of the signal, you’ll see HDR, if not, you’ll see SDR. Very pragmatic. Very Sony.

Format 5—Technicolor Advanced HDR

Oh how I love a baffling flow diagram. From CNET / https://www.cnet.com/news/what-is-advanced-hdr-by-technicolor/

Well if Dolby has an HDR format then we have to too (sticks tongue out.) The newest and actually yet to be implemented HDR format, it apparently comes in three flavors—SL-HDR1 (Single Layer HDR1) SDR with HDR picture data hidden inside (has been approved by ATSC 3.0), SL-HDR2 (Single Layer HDR2) basically HDR10+, SL-HDR3 (Single Layer HDR3) uses Sony Hybrid Log Gamma (did I just say probably?) as a base but with metadata on top and the rest is TBD special sauce! In other words, who the hell knows. This format seems to be still very much in its infancy. Though Tech might strongly disagree with that statement.

And a bit more—

This was meant to be a short article and ended up being about 5000 words. This isn’t really a format war just a lot of crap for consumers to sort through. My two cents—Dolby Vision is great for both content creators and consumers and neither have to pay the licensing fees. This format is by far the most aesthetically pleasing, flexible, and has the most room to grow. I hope it doesn’t get elbowed out by something inferior that was designed to get around it. I’m looking at your HDR10+.

SMPTE and the ITU have done a ton of work bringing order to this chaos and there are a suite of specs worth a glance—

SMPTE ST 2084

Rec. 2100

SMPTE ST 2094-40 (HDR10+)

SMPTE ST 2086 (static HDR metadata)

SMPTE ST 2094-10 (Dolby dynamic metadata)

One last point to make and that is HDR monitoring on the set. I’ve spoken with a lot of DITs about this. Some do it, some don’t. Some say it confuses the DP’s brain so much that they set up a separate HDR viewing tent (I can totally see why, side by side, your SDR will only disappoint.) The sweet spot for on-set monitors is 24” and prior to NAB 2019, I didn’t see any HDR monitors smaller than 30 inches. However, Sony is now making smaller HDR monitors for the purpose of use on the set. I do know DITs who’ve hauled around Sony BVM X300’s on movie sets even though those are facility-level displays and were never intended to be strapped to a DIT cart. I’ve been out of the on-set game since 2014 so am not really the most qualified to speak to this. If anyone reading this is doing HDR on-set and would like to write a post on the topic, be my guest. Art Adams wrote one not too long ago, probably still relevant.

Also TPN is the new MPAA. Did you know that? I didn’t.

As always, I rely on reader feedback for accuracy so if there are any errors or omissions, please let me know. Thanks.