crash when encoding large image with AOM codec at speed 9 and 10 about libavif

[bnason-nf]

@novomesk: maybe this is a stack overflow? Could you try increasing the thread stack size where you call avifEncoderWrite() from save_layer() to see if that helps?

[joedrago]

I think Ben's guess is great. That looks like a particularly deep/heavy stack.

Answering from a libavif perspective, the speed choice between 8-10 (say) are basically identical codepaths for libavif's codec_aom implementation, so a crash in libaom here (but only with speed==10) is unlikely to be specific to libavif guts. We're just passing through a different value into the codec control.

[novomesk]

I tried to increase stack 4 times using following code before calling avifEncoderWrite but it doesn't help.
Old Stack size: 8376320
New Stack size: 33505280

  pthread_attr_t  tattr;
  size_t          stack_size;
  int ret;

  ret = pthread_getattr_np ( pthread_self(), &tattr );
  if ( ret == 0 )
    {
      ret = pthread_attr_getstacksize ( &tattr, &stack_size );
      printf ( "Old Stack size: %ld\n", stack_size );

      ret = pthread_attr_setstacksize ( &tattr, 4*stack_size );
      ret = pthread_attr_getstacksize ( &tattr, &stack_size );
      printf ( "New Stack size: %ld\n", stack_size );
    }

Do you have some other ideas?

[novomesk]

avif_crash.c.txt

I am attaching a small C program to demonstrate crash.
I shared the crash demo in https://bugs.chromium.org/p/aomedia/issues/detail?id=2584

[joedrago]

I can repro with your example as well, and it doesn't seem to be image size related; I can set the image to 256x256 and still get a crash. I'll see if I'm sending libaom something odd here, but it is starting to just feel like a libaom bug at this point.

[joedrago]

Upon further digging, it appears to be related to AOM_USAGE_REALTIME. If I avoid that, libaom will handle any size. I'll shoot them an email and see what's going on.

[joedrago]

I can repro this without libavif being involved.

256.zip

aomenc 256.y4m --cpu-used=8 --usage=1 -o 256.ivf

I'm still in discussions with the libaom people.

[joedrago]

https://bugs.chromium.org/p/aomedia/issues/detail?id=2587

[joedrago]

It looks like libaom have found the bug and have a tentative fix. I'll have to see what to do about getting an updated tag pushed that includes this fix.

[joedrago]

Alright, this has been worked around in the new version v0.5.6. libaom has a proper fix for this, but aren't planning to make a new tagged release for another month or two, so we've worked out a clamp on libaom's cpu-used when encoding >8bpc images for now.

I'm tracking the re-enabling of this speed in another github issue here:

#56