Proposal: Deprecate and remove native secret store support about crossplane

[balorette]

I understand the desire to streamline development efforts, I believe that maintaining this feature is crucial for secure use cases, especially when integrating solutions like HashiCorp Vault.

Couple quick points on it:

Using an external secret store like Vault, we can add an additional layer of protection to sensitive connection details. This includes features such as secret injection and secret writing, which significantly enhance the security of our infrastructure. With native support, we can seamlessly integrate these security measures into our workflows, ensuring that our secrets are well-protected.

Running the control plane as a solution for end users to consume, we can conveniently write their secrets to Vault for consumption. This eliminates the need for users to manage multiple solutions and simplifies their experience. By maintaining native support, it allows customers to utilize a trusted and comprehensive solution for their security needs.

Managing multiple solutions for a single domain (IaC provisioning) to handle external secrets can be cumbersome and inefficient. While using another solution (External Secret Store Operation) may provide additional integration, it is more of a pattern to consider. By maintaining native external secret store support, we eliminate the need for an additional product solely to secure Crossplane. This reduction in the number of products needed not only simplifies our infrastructure but also reduces the potential for misconfigurations and security vulnerabilities.

Furthermore, as more customers, particularly those in areas with a high focus on security, consider using our product, the integration of external secrets into solutions like HashiCorp Vault will become an imperative feature. Customers in highly regulated businesses seek a first-class integration that allows them to leverage the advanced security capabilities of Vault without the need to manage multiple solutions.

[shuknk8s]

We have strict security requirements to push the generated secrets directly to external secret manager like vault NOT to create insecure kubernetes secrets for connection details and passwords(ex rds , aurora db) in crossplane cluster and then push. Hence not supporting ESS is going to negatively impact us and we may have to discard crossplane/upbound as solution. ESO can only push existing insecure k8s secret to external secret manager store so it is not at all a secure solution, I would urge you to support ESS, ESO pushsecret is not an equivalent solution to ESS. ESO requires an existing kubernetes secrets in the cluster to push to externakl secret store , ESS does not need that by design which we really appreciate from cyber security stand point

[haarchri]

@negz @turkenh think we need to work on this Deprecation Notice when we want to drop ESS support https://github.com/crossplane/crossplane/pull/2940/files#diff-40b5c705a7ba08a514b14b156d26120fcd0c6a8b4326096ceecbb7d3bd971f29R47

// This field is planned to be replaced in a future release in favor of
// PublishConnectionDetailsWithStoreConfigRef. Currently, both could be
// set independently and connection details would be published to both
// without affecting each other as long as related fields at MR level
// specified.
// +optional
WriteConnectionSecretsToNamespace *string json:"writeConnectionSecretsToNamespace,omitempty"

// PublishConnectionDetailsWithStoreConfig specifies the secret store config
// with which the connection secrets of composite resource dynamically
// provisioned using this composition will be published.
turkenh marked this conversation as resolved.
// +optional
// +kubebuilder:default={"name": "default"}
PublishConnectionDetailsWithStoreConfigRef *xpv1.Reference json:"publishConnectionDetailsWithStoreConfigRef,omitempty"

cc: @ytsarev

[negz]

@balorette I totally get the desire to integrate with external systems like Vault.

I think this is clear to you already given what you wrote, but I want to reiterate that that would still be possible. This would change how that's done, not that it can be done.

In fact, ESO supports integrating with many more stores than Crossplane. Today our alpha ESS supports only Vault, while ESO supports Vault while ESO lists 26 external secret stores in their documentation, including AWS Secrets Manager and Azure Key Vault. ESO also supports pulling secrets from external stores like Vault, not just pushing them.

Managing multiple solutions for a single domain (IaC provisioning) to handle external secrets can be cumbersome and inefficient. While using another solution (External Secret Store Operation) may provide additional integration, it is more of a pattern to consider.

I wonder if this would incur meaningfully more operational burden relative to our current ESS integration? I don't think what we offer at the moment is very streamlined. e.g. Compare:

• https://docs.crossplane.io/knowledge-base/integrations/vault-as-secret-store/
• https://external-secrets.io/latest/provider/hashicorp-vault/

Today with ESS I believe the flow is:

1. Use Helm to install the Vault agent sidecar injector
2. Use Helm to install Crossplane's Vault ESS Plugin
3. Update Crossplane to set the --enable-external-secret-stores flag
4. Add a DeploymentRuntimeConfig for each Provider to set the --enable-external-secret-stores flag
5. Create a VaultConfig to configure Crossplane's Vault ESS Plugin to talk to Vault
6. Create a StoreConfig to configure Crossplane to talk to its ESS Plugin
7. Create a StoreConfig for each provider, to configure it to talk to Crossplane's ESS Plugin
8. Create claims/XRs/MRs that write connection details to Vault by setting spec.publishConnectionDetailsTo

With ESO there's more (and simpler) authentication options, but assuming you use the Vault agent sidecar, the flow is:

1. Use Helm to install the Vault agent sidecar injector
2. Use Helm to install ESO
3. Create a ClusterSecretStore to configure ESO to talk to Vault ¹
4. Create claims/XRs/MRs that write connection details to a Secret by setting spec.writeConnectionSecretToRef
5. Push secrets to Vault by creating a PushSecret.

Footnotes

1. ESO doesn't use the Vault agent sidecar injector. Instead it supports various ways of authenticating to Vault, including Kubernetes service accounts, AWS IRSA, and reading a Vault token from a secret. ↩

[negz]

Push secrets to Vault by creating a PushSecret.

One shortcoming of ESO is that you currently need to create a PushSecret for each secret you want to push to an external secret store like Vault.

There's a desire to support pushing all secrets from a namespace, per external-secrets/external-secrets#3183.

I think pushing based on label and type would be convenient too. All Crossplane connection secrets are of type: connection.crossplane.io/v1alpha1. So ideally you could easily configure "push all Crossplane connection secrets (in this namespace)" to Vault.

https://github.com/crossplane/crossplane-runtime/blob/v1.15.1/pkg/resource/resource.go#L41

I'd be happy to contribute this to ESO if it was the difference between having to maintain our own ESS implementation or not.