Priority inheritance (Binder-like) about bus1

[dvdhrm]

We never implemented priority inheritance (PI), because the kernel lacks suitable infrastructure for it. The only user visible PI interface is FUTEX_LOCK_PI and friends, and thus ends up being struct rtmutex internally. This interface, though, is not directly applicable to IPC and we'd have to grossly mis-use it to get the desired PI effect. This is also why binder uses nice-levels, which were not particularly well received by upstream maintainers. Using nice-levels is a hack.

We did, however, discuss several PI setups for bus1. They are all modeled around blocking unicast transactions, that temporarily boost the receiver until its reply is queued. For instance, imagine a u64 reply_handle parameter to SEND, which, if requested by the caller via a flag, will block the call until a reply has been queued to the specified handle. Internally, we can thus make the receiver of the message inherit the priority of the caller.

Now with this you still end up with a bunch of open issues. What to do with multicasts? Which process do you boost while a message has not been dequeued, yet? Which kernel infrastructure do you use to boost processes? (e.g., how to avoid nice-level conflicts with user-space administration?)
Some of those are easier to solve than others. However, the big issue I still have with PI is its effectiveness. Why is PI so important, but other controllers are ignored? For instance, a nice level raises CPU priority, but it does not grant IO priority. It does not elevate the resource limits, nor does it modify any other controllers of the destination.

I am not saying PI does not work. The issue I have is the lack of proper kernel infrastructure to do this. We do have lots of knobs to control process priorities, resources, etc. via cgroups. Using nice-levels as the magic solution for IPC sounds wrong to me.
If bus1 gets merged and somebody convinced upstream that a binder-like nice-level hack is fine, we will definitely not block this. I also think that the flag-fields of SEND and nodes are enough to implement this. However, I would prefer a wider discussion about this issue. What are the problems we are solving here? Does this really apply to real world scenarios? How do other systems solve it? How do systems with non-blocking IPC deal with this? Who are the users that want this feature?

Now, regarding your second question (if this turns into a longer discussion, I'd prefer doing it as a separate issue): Care to elaborate how PEEK is useful against FD exhaustion attacks? In a single-threaded dispatcher, you can simply close the FDs once received. If you don't want FDs, don't use BUS1_RECV_FLAG_INSTALL_FDS.
Looking at a message to decide whether you want those FDs is a hack, IMO. If the existing API does not serve your security concerns, I'd appreciate if you can elaborate on the exact attack vectors, so we can comment on the issue, rather than on the solution.

Thanks
David

[l0kod]

I don't think that the FUTEX_LOCK_PI trick can be used to force a caller to use it and delegate its priority to the callee. Moreover, this seems to only be able to raise a priority but can not lower it.

This is also why binder uses nice-levels, which were not particularly well received by upstream maintainers. Using nice-levels is a hack.

Probably, but it seems to work fine for Android, though.

I like the reply_handle with a flag, but I'm not sure that the caller thread must be blocked. Why not just push its priority to the callee while keeping an async IPC, at least with the nice-level trick?

I agree that other controllers should be taken into account. The nice-level should definitely not be the sole player here.

The PI is not easy to get right, but I think every service should use it to protect themselves against abuses from their (potentially malicious) clients. Being able to charge a request should be an available feature in a sane IPC. I don't know other IPCs that solve this issue but some inspiration may be gained from Ethereum and their gas, though. :)

[bl33pbl0p]

There was recently a paper by the seL4 people on exposing scheduling context as capabilities, putting this here if people want to look into it to get some ideas on how to solve this: http://ts.data61.csiro.au/publications/csiro_full_text//Lyons_MAH_18.pdf

Clients transferring SC as caps to the server probably has more to do with accounting, but it is, as I see it, also a PI mechanism. It is similar to the flags approach in that it is transferred during the send (and supports both active and passive models for the server).

[dvdhrm]

@bl33pbl0p, Thanks! Definitely an interesting read. This references several techniques we already use in bus1. Eg., the reply-handle is the only way to get replies in bus1, and would allow attaching SC in the same way as is mentioned in the paper. We definitely envision something similar with bus1 as well. However, reading that article, it becomes even more clear that we need tight scheduler integration for that, and so far the linux scheduler does not expose these features to us.