snapp-incubator / ceph-s3-operator

A Kubernetes operator for efficient management of S3 users and buckets in S3 object storage created with Ceph RGW.

Ceph S3 Operator

Introduction

The Ceph S3 Operator, an open-source endeavor, is crafted to streamline the management of S3 users and buckets within a Ceph cluster environment. It enhances efficiency and simplifies processes, rendering S3 usage on Ceph clusters more straightforward and user-friendly.

Features

• S3 User Management
• Bucket Management
• Subuser Support
• Bucket policy Support
• Quota Management
• Webhook Integration
• E2E Testing
• Helm Chart and OLM Support

Installation

Prerequisites

• Kubernetes v1.23.0+
• Ceph v14.2.10+

  Note: prior Ceph versions don't support the subuser bucket policy. Nevertheless, other features are expected to work properly within those earlier releases.

• ClusterResourceQuota CRD: kubectl apply -f config/external-crd

Using OLM

You can find the operator on OperatorHub and install it using OLM.

Using Helm

Deploy using Helm (version 3.8.0 or later), which supports OCI charts. To use the helm chart, edit the values.yaml file and set controllerManagerConfig.configYaml to your Ceph cluster configuration like secret.yaml.

helm upgrade --install ceph-s3-operator oci://ghcr.io/snapp-incubator/ceph-s3-operator/helm-charts/ceph-s3-operator --version v0.3.7

Using Makefile

Deploy using a simple command:

make deploy

Usage and Documentation

• CRD Examples: Located in the samples folder.
• Detailed Documentation: Available on the wiki.
• Design and Decision Insights: Refer to our design doc for an in-depth understanding.

Versioning and Release

A new docker image, bundle and helm chart will be created each time a tag starting with v is pushed to the main branch.

Development

We follow Kubebuilder development principles, Specifically about testing in an environment similar to the real world and avoiding mocks as much as possible.

For example, we don't mock RGW API. Instead, we use a similar approach to what go-ceph does.

Building the testing image

TESTING_IMAGE_TAG=<desired_tag> make build-testing-image

Don't forget to update the tag in Makefile!

Building the helm chart

We use helmify to generate Helm chart from kustomize rendered manifests. To update the chart run:

make helm

The chart will be created/updated in charts/ceph-s3-operator path

Run locally

If you want to test the operator on your local environment, run the below instructions:

First setup the local Ceph cluster:

make setup-dev-env

Then run the operator either with or without webhook:

make run  # Without webhook
make run-with-webhook # With webhook

At the end, you can tear down the operator and the Ceph cluster:

make teardown-operator teardown-dev-env

Test

To test the project via the operator-sdk envtest:

make test

And to run the e2e tests with KUTTL performing the tests on a KIND cluster:

kubectl-kuttl test

Contributing

Contributions are warmly welcomed. Feel free to submit issues or pull requests.

License

This project is licensed under the GPL 3.0.

ceph-s3-operator's Issues

I am using a cluster installed by kubeadm, can I use this project

When I went to create the user, I reported an error that was not recognized by the api

createuser.yaml
apiVersion: s3.snappcloud.io/v1alpha1
kind: S3UserClaim
metadata:
  name: s3userclaim-sample
  namespace: s3-operator-test
spec:
  s3UserClass: ceph-dashboard
  readonlySecret: s3-sample-readonly-secret
  adminSecret: s3-sampdashboardn-secret
  subusers:
    - subuser1
    - subuser2

2024-02-29T07:36:12Z	ERROR	s3userclaim-resource	failed to validate against cluster quota	{"error": "failed to get clusterQuota, no matches for kind \"ClusterResourceQuota\" in version \"quota.openshift.io/v1\""}

[Feature]: Modify status section of the s3bucket CRD

I suggest modifying the s3bucket status section:
Current status:

type S3BucketStatus struct {
	// +kubebuilder:validation:Optional
	// +kubebuilder:default=false
	Ready bool `json:"ready,omitempty"`

	// +kubebuilder:validation:Optional
	Reason string `json:"reason,omitempty"`

	// +kubebuilder:validation:Optional
	S3SubUserBinding []SubUserBinding `json:"s3SubUserBinding,omitempty"`
}

Suggested:

type S3BucketStatus struct {
	// +kubebuilder:validation:Optional
	// +kubebuilder:default=false
	Created bool `json:"created,omitempty"`

	// +kubebuilder:validation:Optional
	Reason string `json:"reason,omitempty"`

	// +kubebuilder:validation:Optional
	Policy string `json:"policy,omitempty"`
}

I believe that using the Created term is more meaningful. Additionally, brining the whole Bucket Policy in the status would be more helpful specially for further debugging purposes.

[Feature]: Adding bucket policy and sub-user support to the operator

Feature Proposal

Objective

At the moment, the operator is only able to give admin and read-only access with two separate credentials to the users. However, in order to remove the need for Minio users, the operator must be able to create specific credentials with customizable access levels for different users.

Proposed CRDs

S3UserClaim

apiVersion: s3.snappcloud.io/v1alpha1
kind: S3UserClaim
metadata:
  name: s3userclaim-sample
  namespace: s3-operator-test
spec:
  s3UserClass: ceph-default
  readonlySecret: s3-sample-readonly-secret
  adminSecret: s3-sample-admin-secret
  quota:
    maxSize: 1000
    maxObjects: 1000
    maxBuckets: 5
  subusers:
    - myUser1
    - myUser2

This results in creating different credentials for every subuser. The credentials key type can be s3 instead of swift.

S3User

It's the same since s3User instances are created only for every s3UserClaim.

S3Bucket

apiVersion: s3.snappcloud.io/v1alpha1
kind: S3Bucket
metadata:
  name: s3bucket-sample
spec:
  # Only a S3User can be the owner. i.e. s3SubUser can't be.
  s3UserRef: s3userclaim-sample
  s3DeletionPolicy: delete
  s3SubUserBinding:
    - name: myUser1
      access: read
    - name: myUser2
      access: write

Access levers on the subusers:

• read
• write

Reconciliation Flow

When the S3User adds a new subuser, the operator would create a new subuser with access=SubuserAccessNone in S3SubUserSpec which leads to creating credentials (access and secret key) for the new subuser.

When the S3User adds the new Sub3User to the s3Bucket, a bucketPolicy would be added with the defined access level to the bucket for the s3SubUser.

Team label error is not clear

Documentation

• The requirement of the ClusterResourceQuota CRD must be mentioned in the Readme and CSV description.

Code

• When a user applies a S3UserClaim object he faces this error: The S3UserClaim "s3userclaim-sample" is invalid: spec.quota: Internal error: please contact the cloud team. This error is related to the namespace which doesn't have the team label. The error should be rewritten to be more clear.

[FEATURE]: User should be able to disable resource quota validation

Is your feature request related to a problem? Please describe.
At the moment, the cluster resource quota and namespace resource quota are validated through validateQuota function in the webhook. The user might not need any kind of validation (against cluster or namespace). Hence, he/she should be able to disable the validation.

Describe the solution you'd like
Two environment variables must be defined that disable the quota validations when they're false:

• CLUSTER_RESOURCE_QUOTA_VALIDATION
• NAMESPACE_RESOURCE_QUOTA_VALIDATION

Their default value must be True and the user must not be needed to install ClusterResourceQuota CRD if its corresponding env variable is set to False.

Describe alternatives you've considered
None.

Additional context
None.

[FEATURE]: Order of apply between S3Bucket and S3UserClaim

Is your feature request related to a problem? Please describe.
If a user wants to create S3UserClaim and S3Bucket manifests, S3Bucket might be applied before S3Userclaim and it will be denied by the webhook.

Describe the solution you'd like
If the required S3UserClaimRef is not available yet, the S3Bucket manifest must be created but be in the pending phase till the S3UserClaim is created and operator must handle it with reconcilation.

Describe alternatives you've considered
None

Additional context
None