How to create .avifs (image/avif-sequence) files? about libavif

[cconcolato]

Here is a script that I used to generate AVIFS with MP4Box:

MP4Box -add-image Chimera-AV1-10bit-480x270-531kbps_44_48.mp4:id=1:primary -new test.avifs
MP4Box -ab avis -ab msf1 -ab miaf -ab MA1B -rb mif1 -brand avis test.avifs
MP4Box -add Chimera-AV1-10bit-480x270-531kbps_44_48.mp4:hdlr=pict:ccst:name="GPAC avifs" test.avifs

The first line creates a new file with an image item from the first frame of the video.
The second line handles brands.
The thirds one finally adds the video to the output file.

[joedrago]

libavif currently doesn't support encoding image sequences, but the guts of most .avifs files are just regular AV1-encoded .mp4 files with a different file type box in the front. They even use moov tracks instead of items, etc. The line between an an "image sequence" and a regular MP4 video was never thinner!

I haven't messed around with this much, but if your goal is just to make some more image sequences to read, you can probably just take any AV1-encoded MP4 you want and add avis as a compatible brand to it, and rename it .avifs:

cp some_av1_video.mp4 image.avifs
mp4box -ab avis image.avifs

You can make your own AV1 videos with FFMpeg:

https://trac.ffmpeg.org/wiki/Encode/AV1

Obviously you'd want to trim them to some reasonable frame count and probably remove the audio, but neither of these are requirements; libavif will probably handle it just fine.

[joedrago]

The next version of avifenc should be able to encode image sequences, including from multi-frame y4m sources (even piped from stdin!). Someone is still reviewing various changes I've made, but I'm likely to put out the new version this week, barring any serious issues.

[cconcolato]

Here are the command lines I used for generating the alpha sequence:

MP4Box -add-image alpha.obu:id=3:ref=auxl,4:alpha:name=Alpha -add-image video.obu:id=4:name=Color -set-primary 4 -ab avif -new test.avifs
MP4Box -add video.obu:hdlr=pict:ccst:name="GPAC avifs" -add alpha.obu:hdlr=auxv:ccst:alpha:name="GPAC avifs alpha" -ref 2:auxl:1 -ab msf1 -ab miaf -ab MA1B -brand avis test.avifs

The first one adds the images (one for the alpha item, one for the master item), sets an item reference between them, sets a brand and the primary item.
The second one adds the video and alpha video tracks, the brands and track references.

[joedrago]

Yep. AVIFs, HEIFs, and MP4s all use the same file format (Base Media File Format), they just use different subsets of "boxes" inside. The "File Type" box is always first in the file and lists all of the types ("brands") this particular file is a legal version of. It is quite simple and legal to have a file that is a valid MP4 movie and a valid AVIF, by just offering both brands in that box.

[joedrago]

I'd like to eventually support them, yes.

There are some oddities in the standard with regards to storing various pieces of metadata (like imir, irot, XMP), as storing this information in tracks (instead of items) is an entirely different path. Also, adding support for this is definitely going to add quite a bit of complexity in various places: properly feeding frame timing data, choosing new input formats for avifenc, the need to surface new codec parameters, etc. I'd prefer to not reinvent ffmpeg, if possible. I also have no idea what additional memory footprint or encode time this will cause.

That said, there's no reason I can't offer something. I've got some local changes for the underpinnings of the codec layers and write.c which should pave the way for this functionality. If I can get more of the "core" metadata/features working with sequences (ICC, alpha), we can slowly iron out the weirder issues as we go.

[novomesk]

Thank you, this is very interesting. I use ffmpeg often and aomenc sometimes but I never tried MP4Box (my favourite container is webm/mkv).
How did you make the alpha_video.avifs which has transparency? Is there another video stream for the alpha channel?

[novomesk]

I observed that ffprobe recognized this avifs file:

ffprobe -hide_banner alpha_video.avifs 
[libaom-av1 @ 0x564c4aeb29c0] 1.0.0-errata1-avif-387-gc179851d9
[libaom-av1 @ 0x564c4aeb84c0] 1.0.0-errata1-avif-387-gc179851d9
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'alpha_video.avifs':
  Metadata:
    major_brand     : avis
    minor_version   : 0
    compatible_brands: mif1avifiso4av01avismsf1miafMA1B
    creation_time   : 2019-02-04T21:19:42.000000Z
  Duration: 00:00:01.92, start: 0.000000, bitrate: 44 kb/s
    Stream #0:0(und): Video: av1 (Main) (av01 / 0x31307661), yuv420p(tv, bt709/bt709/iec61966-2-1), 640x480, 14 kb/s, 25 fps, 25 tbr, 25k tbn, 25k tbc (default)
    Metadata:
      creation_time   : 2019-02-04T21:19:42.000000Z
      handler_name    : GPAC avifs
    Stream #0:1(und): Video: av1 (Main) (av01 / 0x31307661), gray(tv, bt709/bt709/iec61966-2-1), 640x480, 19 kb/s, 25 fps, 25 tbr, 25k tbn, 25k tbc (default)
    Metadata:
      creation_time   : 2019-02-04T21:19:42.000000Z
      handler_name    : GPAC avifs alpha
[libaom-av1 @ 0x564c4aeb5dc0] 1.0.0-errata1-avif-387-gc179851d9
[libaom-av1 @ 0x564c4af74340] 1.0.0-errata1-avif-387-gc179851d9

[ledyba-z]

To make monochrome AV1 video, you can't use ffmpeg now.

I am sending a patch to enable it.

http://ffmpeg.org/pipermail/ffmpeg-devel/2020-April/260007.html

[EwoutH]

Do you plan to support image sequences eventually in libavif?

[novomesk]

I am also interested in making image sequences without other tools (only via libavif).
I am worried about memory requirements - when we have a single image, libaom allocate several GB sometimes. We can't keep very long uncompressed sequences in memory. It's like video encoding, we will need some temporary files on disk during the process.

[novomesk]

I want to ask. When we will encode sequences, is it necessary to set NCLX/ICC profile on each image in the sequence or just for the first one?

[joedrago]

The API takes all metadata (Exif/XMP/ICC/transforms) from the first provided avifImage. As long as the subsequent avifImages correspond in dimensions/depth/format/range/alpha, it should work just fine.

[gitoss]

libavif currently doesn't support encoding image sequences, but the guts of most .avifs files are just regular AV1-encoded .mp4 files with a different file type box in the front. They even use moov tracks instead of items, etc. The line between an an "image sequence" and a regular MP4 video was never thinner!

This is ancient news, but fyi here's the reasoning why Google created webp and didn't just stuff everything into webm ... I don't know how much different image sequences in avif will be:

"Why not simply support WebM in ?" https://developers.google.com/speed/webp/faq