gottaboy / awesome-go-storage

A curated list of awesome Go storage projects and libraries

Awesome Go Storage

A curated list of awesome Go storage projects and libraries. Inspired by awesome-go.

Contributing

Please take a quick gander at the contribution guidelines first. Thanks to all contributors; you rock!

If you see a package or project here that is no longer maintained or is not a good fit, please submit a pull request to improve this file. Thank you!

Contents

• Awesome Go Storage
  • Storage Server
  • Key-Value Store
  • File System
  • Database
  • Database Drivers

Storage Server

Storage Servers implemented in Go.

• minio - Minio is an open source object storage server compatible with Amazon S3 APIs.
• rclone - "rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfile…
• perkeep - Perkeep is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
• s3git - Git for Cloud Storage. Distributed Version Control for Data.
• storj - Decentralized cloud object storage that is affordable, easy to use, private, and secure.
• rook - Open, Cloud Native, and Universal Distributed Storage.
• longhorn - Longhorn is an open source persistent block storage server delivered via containers.

Key-Value Store

Key-Value Store implemented in Go.

• Bitcask - Bitcask is an embeddable, persistent and fast key-value (KV) database written in pure Go with predictable read/write performance, low latency and high throughput thanks to the bitcask on-disk layout (LSM+WAL).
• Bitraft - Bitraft is a distributed raft-based key/value store with a Redis-compatible protocol. It uses Bitcask for high performance/throughout and low latency.
• BadgerDB - BadgerDB is an embeddable, persistent, simple and fast key-value (KV) database written in pure Go. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.
• biscuit - Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.
• consul - Distributed consistent replicated key-value store for service discovery and configuration.
• diskv - A disk-backed key-value store.
• etcd - Distributed reliable key-value store for the most critical data of a distributed system.
• go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.
• nutsdb - Nutsdb is a simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

File System

File Systems implemented in Go.

• afero - A FileSystem Abstraction System for Go
• fsnotify - Cross-platform file system notifications for Go.
• juicefs - A distributed POSIX file system built on top of Redis and S3.
• gcsfuse - A user-space file system for interacting with Google Cloud Storage.
• git-lfs - Git extension for versioning large files.
• go-systemd - Go bindings to systemd socket activation, journal, D-Bus, and unit files.
• goofys - A high-performance, POSIX-ish Amazon S3 file system written in Go.
• minikeyvalue - A ~1000 line distributed key value store.
• seaweedfs - SeaweedFS is a simple and highly scalable distributed file system for small files.
• svfs - A virtual file system over Openstack Swift built upon fuse.

Database

Databases implemented in Go.

• BigCache - Efficient key/value cache for gigabytes of data.
• bolt - A low-level key/value database for Go. This original version by Ben Johnson has been marked as unmaintained and forked by CoreOS bbolt.
• buntdb - A fast, embeddable, in-memory key/value database for Go with custom indexing and spatial support.
• cache2go - An in-memory key:value cache which supports automatic invalidation based on timeouts.
• cockroach - A Scalable, Geo-Replicated, Transactional Datastore
• couchcache - A RESTful caching micro-service backed by Couchbase server.
• CovenantSQL - A SQL Database with Blockchain features.
• dgraph - Scalable, Distributed, Low Latency, High Throughput Graph Database.
• diskv - A home-grown disk-backed key-value store.
• eliasdb - Dependency-free, transactional graph database with REST API, phrase search and SQL-like query language.
• emitter - Scalable, low-latency, distributed & secure pub/sub database with time-series message storage, suitable for IoT, gaming, apps and real-time web.
• forestdb - Go bindings for ForestDB.
• GCache - Cache library with support for expirable Cache, LFU, LRU and ARC.
• geocache - An in-memory cache that is suitable for geolocation based applications.
• go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.
• goleveldb - An implementation of the LevelDB key/value database in the Go.
• groupcache - Groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.
• influxdb - Scalable datastore for metrics, events, and real-time analytics
• ledisdb - Ledisdb is a high performance NoSQL like Redis based on LevelDB.
• levigo - Levigo is a Go wrapper for LevelDB.
• moss - Moss is a simple LSM key-value storage engine written in 100% Go.
• noms - The versioned, forkable, syncable database.
• nutsdb - Nutsdb is a simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.
• piladb - Lightweight RESTful database engine based on stack data structures.
• pREST - Serve a RESTful API from any PostgreSQL database.
• prometheus - Monitoring system and time series database.
• rqlite - The lightweight, distributed, relational database built on SQLite.
• scribble - A tiny flat file JSON store.
• tidb - TiDB is a distributed SQL database. Inspired by the design of Google F1.
• tiedot - Your NoSQL database powered by Golang.
• Tile38 - A geolocation DB with spatial index and realtime geofencing.

Database schema migration.

• darwin - Database schema evolution library for Go
• goose - Database migration tool. You can manage your database's evolution by creating incremental SQL or Go scripts.
• gormigrate - Database schema migration helper for Gorm ORM.
• migrate - Database migration handling in Golang support MySQL, PostgreSQL, Cassandra, and SQLite.
• pravasan - Simple Migration tool - currently for MySQL but planning to support soon for Postgres, SQLite, MongoDB, etc.,
• soda - Database migration, creation, ORM, etc... for MySQL, PostgreSQL, and SQLite.
• sql-migrate - Database migration tool. Allows embedding migrations into the application using go-bindata.

Database tools.

• go-mysql - A go toolset to handle MySQL protocol and replication.
• go-mysql-elasticsearch - Sync your MySQL data into Elasticsearch automatically.
• kingshard - kingshard is a high performance proxy for MySQL powered by Golang.
• myreplication - MySql binary log replication listener. Support statement and row based replication.
• orchestrator - MySQL replication topology manager & visualizer
• pgweb - A web-based PostgreSQL database browser
• vitess - vitess provides servers and tools which facilitate scaling of MySQL databases for large scale web services.
• usql - universal command-line interface for SQL databases

SQL query builder, libraries for building and using SQL.

• buildsqlx - Go Database query builder library
• dat - Go Postgres Data Access Toolkit
• Dotsql - Go library that helps you keep sql files in one place and use it with ease.
• goqu - An idiomatic SQL builder and query library.
• grimoire - Grimoire is a database access layer and validation for golang. (Support: MySQL, PostgreSQL and SQLite3).
• igor - Abstraction layer for PostgreSQL that supports advanced functionality and uses gorm-like syntax.
• ozzo-dbx - Powerful data retrieval methods as well as DB-agnostic query building capabilities.
• SQLBoiler - a tool to generate a Go ORM tailored to your database schema. It is a "database-first" ORM as opposed to "code-first", and you must first create your database.
• sqrl - SQL query builder, fork of Squirrel with improved performance.
• Squirrel - Go library that helps you build SQL queries.
• xo - Generate idiomatic Go code for databases based on existing schema definitions or custom queries supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server.

Database Drivers

Libraries for connecting and operating databases.

• Relational Databases

  • bgc - Datastore Connectivity for BigQuery for go.
  • firebirdsql - Firebird RDBMS SQL driver for Go
  • go-adodb - Microsoft ActiveX Object DataBase driver for go that using database/sql.
  • go-bqstreamer - BigQuery fast and concurrent stream insert.
  • go-mssqldb - Microsoft MSSQL driver in go language.
  • go-oci8 - Oracle driver for go that using database/sql.
  • go-sql-driver/mysql - MySQL driver for Go.
  • go-sqlite3 - SQLite3 driver for go that using database/sql.
  • gofreetds Microsoft MSSQL driver. Go wrapper over FreeTDS.
  • pgx - PostgreSQL driver supporting features beyond those exposed by database/sql.
  • pq - Pure Go Postgres driver for database/sql.

• NoSQL Databases

  • aerospike-client-go - Aerospike client in Go language.
  • arangolite - Lightweight golang driver for ArangoDB.
  • asc - Datastore Connectivity for Aerospike for go.
  • cayley - A graph database with support for multiple backends.
  • dsc - Datastore connectivity for SQL, NoSQL, structured files.
  • dynago - Dynago is a principle of least surprise client for DynamoDB
  • go-couchbase - Couchbase client in Go
  • go-couchdb - Yet another CouchDB HTTP API wrapper for Go
  • gocb - Official Couchbase Go SDK
  • gocql - A Go language driver for Apache Cassandra.
  • gomemcache - memcache client library for the Go programming language.
  • gorethink - Go language driver for RethinkDB
  • goriak - Go language driver for Riak KV
  • mongo-go-driver - The Go driver for MongoDB
  • neo4j - Neo4j Rest API Bindings for Golang
  • Neo4j-GO - Neo4j REST Client in golang.
  • neoism - Neo4j client for Golang
  • redigo - Redigo is a Go client for the Redis database.
  • redis - Redis client for Golang
  • redis - A simple, powerful Redis client for Go.
  • redeo - Redis-protocol compatible TCP servers/services.

• Search and Analytic Databases

  • bleve - A modern text indexing library for go.
  • elastic - Elasticsearch client for Go.
  • elastigo - A Elasticsearch client library.
  • skizze - A probabilistic data-structures service and storage.

• Multiple Backends

  • gokv - A simple key-value store interface and many implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)