cubanismo / allocator Goto Github PK

One of the resounding things that came out of the discussions at XDC was the mobile guys saying that they don't want to have to do contiguous memory allocation in their GPU driver. Instead, they wanted a central allocator (such as ION) that would understand all the constraints and be able to allocate an image. Both NVIDIA and Intel badly wanted to do the allocation inside their driver stack in the case of single-vendor interactions so that they could handle all their "magic".

Two options were discussed for resolving this:

• Require the client to go around to each of the drivers until it finds one that can allocate its image
• Have a central allocator (similar to gralloc) that understands everything

In the end, we settled on the second option (central allocator) with the understanding that it would have a bunch of pluggable back-ends and would internally walk around to the different back-ends until it found one that could allocate.

This answers the question for the final allocation step, but there are several more steps involved:

• Convert GL/Vulkan/CL/v4l image specification to allocator spec (VkFormat -> allocator format, etc.)
• Get allocation capabilities
• Intersect capabilities and union constraints
• Allocate memory (central)
• Lay out the image (highly vendor-specific except for the 2D case)

There are a few possibilites here. (Thanks to @robclark for the write-up. I'm mostly just moving it to github.):

• fourcc:
  • Simple and already used in many places (drm/kms, egl dmabuf extensions, v4l, etc)
  • Should be sufficient to describe anything that can be shared between multiple different gbm2_backends (ie. between scanout, gl/vk, camera, vidc, etc)
• Khronos dataformat spec:
  • is more expressive than fourcc.. fourcc is not sufficient for describing “advanced” textures (mipmap levels + array and/or 3d, for example)..
  • This case is less likely to be involved in cross-device sharing, but could be used in cross-api sharing (ie. interop between $vendor’s gl and vk driver)
  • note that the interop case includes sharing between different versions of $vendor’s gl/vk driver (such as different version in a container/chroot) where layout of layers/levels might have changed)?

Straw man suggestions:

1. Use fourcc plus additional capabilities/constraints to fill in the gaps for gl/vk interop?
2. Use dataformat plus helper lib which can be used by backend to map from
3. Use fourcc together with (possibly device-specific) metadata. Types of metadata supported would be handled by constraints

1. gbm has some extra stuff, like surfaces, for gluing together with egl.. but gbm_surface perhaps doesn't make as much sense for other APIs like v4l
2. gbm is (mostly) using fourcc, but if "liballoc" uses dataformat this might not fit so nicely as a (backward compatible) extension to gbm

Possibly this new thing just ends up sitting on the side as a separate API which only deals with allocation, with some small extension to gbm to get from a alloc_bo_t to a struct gbm_bo? You could also maybe just do that via importing the dma-buf fd into gbm, but it would be convenient for buffers allocated via the gl/vk driver's "liballoc" backend, to be able to recover the original pipe_resource or equivalent.

The other possibility is new set of EGL extension, plus maybe some optional (ie. not supported by all backends) features like surfaces, so this new thing replaces gbm. Although there are existing users who use gbm to do gl apps on "bare metal" so this would mean that mesa has to support both gbm and new thing as potential args to eglGetDevice()..

Capbailities

My initial vision for memory allocation was to have independent blocks of capabilities where each block was fairly self-contained but could have complex interactions within it. This has the down-side that the driver has to be involved in the intersection process. The directions that @cubanismo was pushing was to have everything as small atomic capabilities. While this is less expressive, it's much simpler to perform an intersection operation. We need to double-check that everything needed is expressible this way.

Tentatively, we have said that the capabilities will be relative to the surface allocation parameters: width, weight, format, sample count, mip levels, array slices, etc. In other words, you don't have to worry about expressing every possible set of capabilities which could grow very large.

List of common capabilities (update as more common capabilities are found):

• linear (pretty much needed for cross-vendor sharing)

Constraints:

Constraints are a bit special because it will take more care to union them properly. We've determined that these constraints will have to be understood by the allocation library (and therefore common) so that they can be combined properly.

List of common constraints (update if more constraints are needed):

• Max pitch
• Min pitch alignment
• Start address alignment
• Max width/height (Maybe handled by the app?)
• dma mask/address range
• YUV multiplanar - planar, where U and V need to be positive offsets from Y < 64MB
• Need to have U and V in separate RAM banks.

Because our strategy for allocation involves going to the different back-ends and asking them to allocate, we need some strategy for knowing which back-ends care to be involved at any given point. I really would like to avoid the Vulkan loader solution of "just load everything". Some companies are known to ship rather large drivers and are liable to make libdrm2_vendor.so a symlink to their libGL.so. I don't want doing something with the allocator to mean you always pay the cost of loading ~1GiB worth of shared libraries into your address space just because you have lots of big heavy GL drivers installed.

This means that you need some way of determining when a particular back-end will be needed. We have a few different options:

1. The client specifically asks us to load back-ends
2. We figure out, based on capabilities, which back-ends are relevant.

The prototype code doesn't allow devices to expose their maximum image width and height as a constraint. Support for this constraint should be added.

While reviewing the current prototype implementation, @mvicomoya pointed out the way I'd used length_in_words was more consistent with a field called extra_words or something, since it excludes the length of the header structure. Now I'm unclear which is preferable: Should length_in_words include the header structure or not? If not, should we change the name?