openebs / rawfile-localpv

Dynamically deploy Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is provisioned from RAW-device file loop mounted Local-Hostpath storage.

Welcome to OpenEBS

We are an ultra-modern Block Mode storage platform, a Hyper-Converged software Storage System and an Enterprise Grade virtual NVMe-oF SAN (vSAN) Fabric that is natively & very tightly integrated into Kubernetes.

Important

OpenEBS provides...

• Stateful persistent Dynamically provisioned storage volumes for Kubernetes
• High Performance NVMe-oF storage access optimized for All-Flash Solid State storage media
• is a 100% Cloud-Native storage platform
• Delivers a Kubernetes cluster-wide vSAN fabric that provides containers/Pods with resilient access to storage across the entire cluster.
• Enterprise Grade data management capabilities such as snapshots, clones, replicated volumes, DiskGroups, Volume Groups, Aggregates, RAID
  

Multiple Storage Engines

OpenEBS is a Kubernetes provides Persistent Stateful Storage Platform that has 5 core Data-Engines
Each Storage engine provides different Capabilities, Flexibility, Resilience, Data Protections, and Performance features.

ID	Data-Eegines	Where to create your issues
	Replicated PV	Replicated storage and data volumes
1	Replicated PV Mayastor	Distributed vSAN Fabric attached volumes that are replicated
	Local PV	Non-replicated node local storage and volumes
2.	Local PV HostPath	Dynamically provisioned Node-local volumes with Hostpath resident backend data
3.	Local PV ZFS	Dynamically provisioned Node-local volumes with an integrated ZFS storage backend
4.	Local PV LVM	Dynamically provisioned Node-local volumes with an integrated LVM storage backend
5.	Local PV Raw-device-File	Dynamically provisioned Node-local volumes via soft-lun RAW Device files on Hostpath resident backend filesystem

OpenEBS is very popular : Live OpenEBS systems actively report back product metrics every day, to our Global Anaytics metrics engine (unless disabled by the user). Here are our key project popularity metrics as of: 01 Feb 2024

🚀   OpenEBS is the #1 deployed Storage Platform for Kubernetes
⭐   We are the #1 GitHub Star ranked K8s Data Storage platform
💾   We have +49 Million Volumes deployed globally
📺   We have +8 Million Global installations
⚡   1 Million OpenEBS K8s Containers are spawned per week
😎   1.1 Million global users

We have a very large active community, and many storage users contribute to our product with discussions, ideas, Issues, Feature requests and even code contribnutions.

There are many ways to get in touch with our team.

• EMAIL us: [email protected]
  
• Contact the Maintainers : The up to date Maintainers list is here

Reach out via GitHuib to the OpenEBS core leadership team.
🚀   Ed Robinson | @edrob999
⭐   David Brace | @orville-wright
⚡   Vishnu Attur | @avishnu
😎   Tiago Castro | @tiagolobocastro

	Try our Slack channel If you have questions about using OpenEBS, please use the CNCF Kubernetes OpenEBS slack channel, it is open for anyone to ask a question

---

Current status

Activity dashbaord

https://openebs.io/

Read this in 🇩🇪 🇷🇺 🇹🇷 🇺🇦 🇨🇳 🇫🇷 🇧🇷 🇪🇸 🇵🇱 🇰🇷 other languages.

OpenEBS is the most widely deployed and easy to use open-source storage solution for Kubernetes.

OpenEBS is the leading open-source example of a category of cloud native storage solutions sometimes called Container Attached Storage. OpenEBS is listed as an open-source example in the CNCF Storage White Paper under the hyperconverged storage solutions.

Some key aspects that make OpenEBS different compared to other traditional storage solutions:

• Built using the micro-services architecture like the applications it serves. OpenEBS is itself deployed as a set of containers on Kubernetes worker nodes. Uses Kubernetes itself to orchestrate and manage OpenEBS components.
• Built completely in userspace making it highly portable to run across any OS/platform.
• Completely intent-driven, inheriting the same principles that drive the ease of use with Kubernetes.
• OpenEBS supports a range of storage engines so that developers can deploy the storage technology appropriate to their application design objectives. Distributed applications like Cassandra can use the LocalPV engine for lowest latency writes. Monolithic applications like MySQL and PostgreSQL can use the ZFS engine (cStor) for resilience. Streaming applications like Kafka can use the NVMe engine Mayastor for best performance in edge environments. Across engine types, OpenEBS provides a consistent framework for high availability, snapshots, clones and manageability.

Deployment

OpenEBS itself is deployed as just another container on your host and enables storage services that can be designated on a per pod, application, cluster or container level, including:

• Automate the management of storage attached to the Kubernetes worker nodes and allow the storage to be used for Dynamically provisioning OpenEBS Replicated or Local PVs.
• Data persistence across nodes, dramatically reducing time spent rebuilding Cassandra rings for example.
• Synchronous replication of volume data across availability zones improving availability and decreasing attach/detach times for example.
• A common layer so whether you are running on AKS, or your bare metal, or GKE, or AWS - your wiring and developer experience for storage services is as similar as possible.
• Backup and Restore of volume data to and from S3 and other targets.

An added advantage of being a completely Kubernetes native solution is that administrators and developers can interact and manage OpenEBS using all the wonderful tooling that is available for Kubernetes like kubectl, Helm, Prometheus, Grafana, Weave Scope, etc.

Our vision is simple: let storage and storage services for persistent workloads be fully integrated into the environment so that each team and workload benefits from the granularity of control and Kubernetes native behaviour.

Roadmap (as of Jan 2024)

OpenEBS is 100% open source software. The project source code is spread across multiple repos and covers multiple projects:

Our main Roadmap is focused exclusively on the modern (STANDARD Edition) Data-Engine Mayastor. It does not define any net-new features or capabilities for any LEGACY projects or projects that are tagged & defined as DEPRECATED or ARCHIVED. Currently those projects are defined as the follows (see references above for the details on the projects DEPRECATED and ARCHIVAL strategy)...

• Jiva
• cStor
• NFS-Provisioner

**MayaStor Roadmap 2024 Roadmap

Scalability

OpenEBS can scale to include an arbitrarily large number of containerized storage controllers. Kubernetes is used to provide fundamental pieces such as using etcd for inventory. OpenEBS scales to the extent your Kubernetes scales.

Installation and Getting Started

OpenEBS can be set up in a few easy steps. You can get going on your choice of Kubernetes cluster by having open-iscsi installed on the Kubernetes nodes and running the openebs-operator using kubectl.

Start the OpenEBS Services using operator

# apply this yaml
kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

Start the OpenEBS Services using helm

helm repo update
helm install --namespace openebs --name openebs stable/openebs

You could also follow our QuickStart Guide.

OpenEBS can be deployed on any Kubernetes cluster - either in the cloud, on-premise or developer laptop (minikube). Note that there are no changes to the underlying kernel that are required as OpenEBS operates in userspace. Please follow our OpenEBS Setup documentation.

Status

OpenEBS is one of the most widely used and tested Kubernetes storage infrastructures in the industry. A CNCF Sandbox project since May 2019, OpenEBS is the first and only storage system to provide a consistent set of software-defined storage capabilities on multiple backends (local, nfs, zfs, nvme) across both on-premise and cloud systems, and was the first to open source its own Chaos Engineering Framework for Stateful Workloads, the Litmus Project, which the community relies on to automatically readiness assess the monthly cadence of OpenEBS versions. Enterprise customers have been using OpenEBS in production since 2018.

The status of various storage engines that power the OpenEBS Persistent Volumes are provided below. The key difference between the statuses are summarized below:

• alpha: The API may change in incompatible ways in a later software release without notice, recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
• beta: Support for the overall features will not be dropped, though details may change. Support for upgrading or migrating between versions will be provided, either through automation or manual steps.
• stable: Features will appear in released software for many subsequent versions and support for upgrading between versions will be provided with software automation in the vast majority of scenarios.

Storage Engine	Status	Details
Jiva	stable	Best suited for running Replicated Block Storage on nodes that make use of ephemeral storage on the Kubernetes worker nodes
cStor	stable	A preferred option for running on nodes that have Block Devices. Recommended option if Snapshot and Clones are required
Local Volumes	stable	Best suited for Distributed Application that need low latency storage - direct-attached storage from the Kubernetes nodes.
Mayastor	stable	Persistent storage solution for Kubernetes, with near-native NVMe performance and advanced data services.

For more details, please refer to OpenEBS Documentation.

Contributing

OpenEBS welcomes your feedback and contributions in any form possible.

• Join OpenEBS community on Kubernetes Slack
  • Already signed up? Head to our discussions at #openebs
• Want to raise an issue or help with fixes and features?
  • See open issues
  • See contributing guide
  • Want to join our contributor community meetings, check this out.
• Join our OpenEBS CNCF Mailing lists
  • For OpenEBS project updates, subscribe to OpenEBS Announcements
  • For interacting with other OpenEBS users, subscribe to OpenEBS Users

Show me the Code

This is a meta-repository for OpenEBS. Please start with the pinned repositories or with OpenEBS Architecture document.

License

OpenEBS is developed under Apache License 2.0 license at the project level. Some components of the project are derived from other open source projects and are distributed under their respective licenses.

OpenEBS is part of the CNCF Projects. OpenEBS is a CNCF project and DataCore, Inc is a CNCF Silver member. DataCore support's CNCF extensively and has funded OpenEBS participating in every KubeCon event since 2020. Our project team is managed under the CNCF Storage Landscape and we contribute to the CNCF CSI and TAG Storage project initiatives. We proudly support CNCF Cloud Native Community Groups initatives.

Thanks for dropping by.

Commercial Offerings

This is a list of third-party companies and individuals who provide products or services related to OpenEBS. OpenEBS is a CNCF project which does not endorse any company. The list is provided in alphabetical order.

• Clouds Sky GmbH
• CodeWave
• DataCore
• Gridworkz Cloud Services

rawfile-localpv's Issues

Filesystem resizing broken

Tried on both v1.23 and v1.25 clusters. Controller pod's external-resizer times out with following:

I1129 05:03:37.766216       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-volume-mongodb-0", UID:"6fd2e208-0b93-47ea-8396-f2868b717ffd", APIVersion:"v1", ResourceVersion:"130660835", FieldPath:""}): type: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd
E1129 05:03:47.765704       1 controller.go:286] Error syncing PVC: resize volume "pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd" by resizer "rawfile.csi.openebs.io" failed: failed to get capabilities of volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd with error getting controller capabilities: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1129 05:03:47.766862       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-volume-mongodb-0", UID:"6fd2e208-0b93-47ea-8396-f2868b717ffd", APIVersion:"v1", ResourceVersion:"130660835", FieldPath:""}): type: 'Warning' reason: 'VolumeResizeFailed' resize volume "pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd" by resizer "rawfile.csi.openebs.io" failed: failed to get capabilities of volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd with error getting controller capabilities: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1129 05:03:47.806703       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-volume-mongodb-0", UID:"6fd2e208-0b93-47ea-8396-f2868b717ffd", APIVersion:"v1", ResourceVersion:"130660924", FieldPath:""}): type: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd
E1129 05:03:57.807200       1 controller.go:286] Error syncing PVC: resize volume "pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd" by resizer "rawfile.csi.openebs.io" failed: failed to get capabilities of volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd with error getting controller capabilities: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I1129 05:03:57.807354       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"data-volume-mongodb-0", UID:"6fd2e208-0b93-47ea-8396-f2868b717ffd", APIVersion:"v1", ResourceVersion:"130660924", FieldPath:""}): type: 'Warning' reason: 'VolumeResizeFailed' resize volume "pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd" by resizer "rawfile.csi.openebs.io" failed: failed to get capabilities of volume pvc-6fd2e208-0b93-47ea-8396-f2868b717ffd with error getting controller capabilities: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Add XFS snapshot support via reflink

Looks lke XFS added reflink support, this can be leveraged to support snapshots for XFS

https://blogs.oracle.com/linux/post/xfs-data-block-sharing-reflink

Add option for reserved storage

With 0.7.0 release I saw there is tracking of available space and therefore no scheduling on a node with insufficient capacity happens (RESOURCE_EXHAUSTED_EXIT_CODE). This is great :)

Currently we have the following issue - it may happen that we have e.g. 100GB of storage on a node, single disk but we need to define how much space we can use for rawdisks (e.g. 50%) because there are other components which may need that amount of storage defined as "reserved".

Would it be possible to provide this in future?

Make the data-dir configurable.

The data directory used for creating the sparse file is hardcoded to "/var/csi/rawfile" in the task template.
https://github.com/openebs/rawfile-localpv/blob/master/templates/task.yaml#L14

Similar to image_tag and image_repository, the data-dir also should be configurable via the env variable.

Note: If you want to get started on setting up rawfile and checking how it works, you can follow the helm3 instructions in the repo or use the YAML instructions available here:

https://github.com/kmova/bootstrap/blob/master/openebs-localpv/rawfile/setup.md

Project still active?

Hi,

this project seems exactly what I want to use for my single node kubernetes server.
But the last commit is more than 2.5 years ago, is there still progress in this project? Or is there another project which covers this use case?

Thank you!

No free space

I am new to k8s and deployed rawfile-localpv on k8s 1.28 and all the relevant pods are running. when i make the storage class and use that in my pod i get the following in pod describe:
"1 node(s) did not have enough free storage"
but my node has plenty of free space (around 100G, my request is 10G) and has no DiskPressure in the node describe
can you please guide me

RawFile LocalPV provisioner does not actually delete PV folders

Steps to reproduce

1. Delete a rawfile LocalPV kubectl delete pvc myrawfile-pvc or kubectl delete pv pv-122133123

2. The PV folder will not be deleted on the host file system. It will only be marked deleted with timestamp in the metadata JSON file.

$ cat disk.meta {"schema_version": 2, "volume_id": "pvc-4b772e0e-9c16-4a1c-961d-9a9ba236de2f", "created_at": 1605768126.6610372, "img_file": "/data/pvc-4b772e0e-9c16-4a1c-961d-9a9ba236de2f/disk.img", "size": 10737418240, "fs_type": "ext4", "deleted_at": 1605769609.8736064}

Add fstrim support

To reclaim space in the host one should regularly issue fstrim on the filesystem. Looks like recent kernels pass this through correctly for loopback deivces as well.

Require Linux/ARM64 docker image for openebs/rawfile-localpv

Hi Team,  

I have been working on building openebs/rawfile-localpv images for both amd64 and arm64 platforms. I have modified the .travis.yaml to build and publish the docker images for arm64 platform.

Changes required: - odidev@682c05d

I have not added the e2e-testing for arm64 platform because e2e-testing requires minikube to run on travis for arm64 platform. And travisCI does not support minikube on arm64 platform. There is an open issue for this in kubernetes/minikube.

Do you have any plans for releasing arm64 images?  It will be very helpful if an arm64 image is available. If interested, I will raise a PR.

Option to disable free space checks

With thin provisioning it's quite easy to bump into resource exhausted errors.

I ended up disabling some checks

https://git.k-space.ee/k-space/rawfile-localpv/commit/17f653b15ff1edd43473f8c4f3adba33c3703d1a
https://git.k-space.ee/k-space/rawfile-localpv/commit/9c4cda7c0a33b7e1d65cda67608a09f6fe1fae36

Please make this configurable upstream

KeyError: 'img_file'

I had the problem of rawfile-csi not clearing the block device loop after deleted PVC.
Thank you for creating such a great solution!

Metadata write is not atomic

The write_text function used to write config file is not atomic

https://stackoverflow.com/questions/73883435/is-python-3-path-write-text-from-pathlib-atomic