cruise-automation / rbacsync Goto Github PK

Need a better way of ascertaining whether RBACSync is healthy. Export metrics for number of fulfilled rbacsyncconfigs vs errored. This will allow detection of upstream issues with monitoring.

Current Behavior:

When controller fails to obtain group members from group.Grouper due an error (e.g G Suite is down or unreachable), it will remove all associated rolebindings from all namespaces. Relevant line is

Expected Behavior:

G Suite API errors should NOT cause removal of rolebindings that already exist for the members. They should only be removed if we can positively confirm the member is no longer in the group. Otherwise this may cause production outages when G Suite API suddenly becomes unavailable due to misconfiguration/connectivity issue/etc.

@stevvooe raised the concern that this may prevent self-healing in case attacker somehow escalates their privileges and creates an "out-of-band" rolebinding. Then they can disable G Suite API and prevent RBACSync from restoring order. I'd argue that we should tradeoff potentially heightened security for reliability because:

• DoS attack is still a valid security concern. Currently an attacker might break G Suite connection and will cause an entire cluster's user base to lose access. The blast radius is potentially huge on this one.
• We can quite easily mitigate the scenario described above by auditing/alerting for "out-of-band" bindings and also alerting on G Suite unavailability.
• Even leaving things as-is, this "vulnerability" still exists because attacker may just repeatedly re-create malicious rolebinding in-between RBACSync re-sync intervals.

Proposed Solution:

Only evict rolebindings when we can positively confirm the associated members aren't in the group members list anymore (by issuing successful API call and examining membership list). Export Prometheus metrics for G Suite errors.

Hi,

Unfortunately I wasn't able to set this up. Since the whole GSuite delegation account and GCP service account and thier permission are quite complex to setup, some screenshots might be helpful. I know it's a very young project but it may raise adoption and people may be willing to maintain documention as well.

Hence, the linked Fin.com is not accessible outside the US but Google cache has it archived (not sure how long)

https://webcache.googleusercontent.com/search?q=cache:Q5VZUOE3cNgJ:https://www.fin.com/post/2017/10/navigating-google-suite-directory-api+&cd=1&hl=en&ct=clnk&gl=de

Cheers,
Peter

While we do provide events and detailed logs of actions surrounding RBACSync, it might be good to improve the audit logging with mind of understanding the complete operation of all configurations used by rbacsync.

This may be solved with software improvements, through better status reporting or through better debugging and auditing documentation.

Hey, I was trying out this project and had to make some changes to remove all the deprecation warnings and get it to working on k8s version 1.20

Changes I made:

• Updated CRDs with OpenAPIV3 schema validation
• Updated CRD API
• Update RoleBindings API

I am not sure whether this project is actively being maintained or not, but if you are interested I can open a PR with the updated manifests.

It is possible Keycloak/LDAP support as idp to sync groups with any schema?

To prevent privilege escalations, we had limited RBACSyncConfig to only reference roles. However, there are valid use cases in which a user may want to bind a namespaced group or user to a ClusterRole. We have decided to support the full abilities documented in https://kubernetes.io/docs/reference/access-authn-authz/rbac/.

This ticket will likely only need the removal of a validation and include some additional test cases.

The https://www.googleapis.com/auth/admin.directory.group.member.readonly scope is not needed to query group membership. Remove from the doco?

Hi, we (Spotify) are currently using RBACSync on each GKE/K8s cluster that we run (over 30 clusters). The aggregate querying of all these RBACSync instances frequently hit the limits for the Google Groups API.

We understand that there are many different configuration options that can change the RBACSync query behavior/frequency of the Google Groups API querying. However, these configuration options aren't able to mitigate the large amount of querying done by RBACSync when you have a lot of Kubernetes clusters running that each have a copy of RBACSync running on them. The problem is especially exacerbated when a new version of RBACSync is deployed as the RBACSync workloads on each cluster will restart/be updated at once and result in a lot of simultaneous querying.

We're curious if there's anything that's recommended for this case to avoid the aggregate of these RBACSync instances exceeding the Google Groups API. We're curious how Cruise/other RBACSync users are running, operating, and deploying RBACSync and if anyone else is encountering these kinds of issues.