Request: Please add support for API-4 to the ultrahdr_app demo. about libultrahdr

[ram-mohan]

@pmonck Thank you for raising this request. Currently we are working on few changes that would cause modifications to the sample application. Along with these changes we will add api-4 support

[DichenZhang1]

Hi @gregbenz @pmonck ,

Encode API-4 serves the use case where user wants to generate the gain map and gain map metadata from outside (eg. from hardware), and use the library to assemble everything together into the correct JPEG/R image format. Unfortunately the library currently only recognize the libultrahdr format of gain map / gain map metadata. With that said, if you pass in an Apple format gain map and gain map metadata into API-4, the encoding process should not crash but the output image could be wrong, and when you decode that image the recon could be wrong.

We know this limitation that currently libultrahdr cannot recognize other gain map formats, and we do have that planned on our roadmap.

Thank you!

[DichenZhang1]

Hi @gregbenz

I'm sorry the there are quite a few confusion terminologies, let me try to explain.

Gain maps are always bundled with gain map metadata, which contains gain map specific parameters such as min/max content boost, gamma, etc, and these parameters are mandatory for gain map calculation and reconstruction. So in our terminology we say "gain map metadata" instead of "metadata".

For general "metadata" like EXIF, XML, the encode API-4 can't take them from outside. However, they can be passed through the SDR JPEG. I.e., metadata in the SDR JPEG will be preserved in the output image.

We do have plan (actually it's working in progress and will land soon) to take more types of HDR / SDR input such as HDR HEIC/AVIF, and from there the library will generate the JPEG/R.

Regarding the SDR image quality, and I assume this is referring to encode API-0. We don't encourage API-0 to be used directly so we marked this API "experimental". For better SDR quality, it is suggested to tone map the HDR from outside.

[DichenZhang1]

API-4 libultrahdr would just be offering a mechanism to encode MPF /GContainer and and the SDR, but gain map pixel content and gain map metadata would all be created externally as source inputs to libultrahdr.

Excatly

[gregbenz]

If this could use the Apple gain map format as an input, this would be incredibly useful. I would assume that is the intention for API-4, as it does not clarify where the required metadata would come from to interpret the base + gain map images.

[gregbenz]

@ram-mohan What is the intended use for API-4? I don't see any detail on the metadata requirements for the source there. Is this just to process Ultra HDR source files, or might this support conversion between the Apple and Ultra HDR / ISO gain map formats?

[pmonck]

Thanks @DichenZhang1,

We know this limitation that currently libultrahdr cannot recognize other gain map formats, and we do have that planned on our roadmap.

Does it recognize and output 3-layer gain maps?

[DichenZhang1]

Does it recognize and output 3-layer gain maps?

Not now, but on roadmap

[gregbenz]

@DichenZhang1 Thank you for that detail. Perhaps I'm unclear, but as I read the table for API4, it sounds to me like the input is an SDR JPG and a gain map, but no metadata. I'm unclear how those inputs would be sufficient to produce a valid gain map as there would be no reference HDR image to create the missing metadata.

For artists like myself, it would be very helpful if there were a way to provide an SDR and HDR image and have a tool to generate the gain map. I read APIs 1-3 as being options to do that, but requiring an HDR format I do not know how to produce (HDR yuv encoded in P010). I posted a discussion item on this asking for suggestions on how to convert a more common HDR format (such as an HDR AVIF or PNG) to that input format. It would be amazing if libultrahdr could do that, but that strikes me as potentially out of scope for the project to support other input formats for the HDR image.

In my testing with Instagram on Apple and Android devices (including using gain maps captured on those devices as well as created from Adobe ACR / LR), I'm seeing interoperability issues between the Apple / Google / Adobe formats. The Adobe to Ultra HDR challenges I've seen I suspect are probably related to RGB vs luminosity only gain maps. Great to hear support for 3-channel & Apple gain maps is on the road map. Thank you so much for your contributions to the HDR community!

[gregbenz]

@DichenZhang1 A little more detail on the value of full SDR / HDR control (if expanded support for other HDR formats is in scope):

• The final SDR result is visually superior when a photographer has full control over the SDR edit. Automatically generated SDR images and even the ones Adobe offers with a 7-slider "preview for SDR display" are not of the same quality. I have received several questions from other photographers asking to solve this issue as they feel the quality of the SDR is insufficient in their gain maps
• Many photographers pursuing HDR also want to print their work. When you've put in the SDR work to create a high-quality print, it would be ideal to use that as your base image in a gain map
• Similarly, many may upgrade existing SDR edits. The gain maps we can produce now show inferior results after the SDR has gone through a round trip to HDR and back. Simply encoding the SDR as is would be ideal.

Again, I realize this is probably outside the intended scope for libultrahdr, but if it could support other inputs or guidance might be shared on how to generate the required hdr yuv file using other CLI tools, that would be very useful.

[gregbenz]

@DichenZhang1 That's fantastic to hear that HEIC/AVIF input support is planned! That will be incredibly valuable to producing the highest quality gain maps.

If I'm understanding correctly on API-4, it will then require the gain map source file also has gain map metadata such as the example below (or something else within spec, such as 3-channel metadata). ie, in API-4 libultrahdr would just be offering a mechanism to encode MPF /GContainer and and the SDR, but gain map pixel content and gain map metadata would all be created externally as source inputs to libultrahdr.

---- XMP-hdrgm:Main ----
Version : 1.0
Base Rendition Is HDR : False
Gain Map Min : 0.000000
Gain Map Max : 4.636625
Gamma : 1.000000
Offset SDR : 0.015625
Offset HDR : 0.015625
HDR Capacity Min : 0.000000
HDR Capacity Max : 4.636625

Agree on the SDR source quality, best that the SDR is provided to libultrahdr. That's quite complicated and requires artistic input for best results.