baitcenter / dbfs

Distributed Blockchain-based File Storage 📡

DBFS

Distributed Blockchain-based File Storage in Elixir 📡

DBFS is an experiment to implement a (very naïve) distributed file storage service using a blockchain in Elixir. With growing interest in Blockchains and Cryptocurrencies, I wanted to understand the complex inner workings of Blockchains and their feasibility in the context of business applications. I started by writing a very simple Blockchain to get started, and then wrote DBFS as a slightly more complex version that actually did something.

Screenshots, Talk and Paper

I gave a lightning talk about the project at the ElixirConf 2018 in Europe, which you can check out here.

I also wrote a basic whitepaper detailing the application as part of my semester project, and you can also see some screenshots of the application here:

Architecture

The application is divided into two parts:

• Elixir Backend: The core application, responsible for storing the data, performing consensus among nodes and exposing an API for clients
• React Frontend: A Javascript based web-client that connects to the backend, provides a UI to perform operations and displays statistics about all the connected nodes.

The application is designed as a "Private Blockchain" after weighing the pros and cons of different approaches, and even though it should absolutely not be used in production, it is meant to be run on private infrastructure instead of being made publicly available so any node can connect.

The sys.config file defines 3 nodes. You can change this to add or remove nodes, but you'll have to perform the setup on each node (if you're running it on different machines). You'll also have to make sure that your hosts file points these domains to their correct IPs.

• [email protected]
• [email protected]
• [email protected]

Setup

You need to have these dependencies installed at minimum:

• Erlang/OTP 20.1
• Elixir 1.5.2
• Postgres 9.4
• Node.js 9.6.1 for Assets and Yarn
• Ruby 2.3.3 for Sass

Set up the Backend:

$ git clone to.shyr.io/dbfs ~/dbfs
$ cd ~/dbfs
$ mix deps.get
$ mix compile

Set up the Frontend:

$ git clone to.shyr.io/dbfs-web ~/dbfs-web
$ cd ~/dbfs-web
$ yarn install

For the first node of the backend, we need to create its instance and initialize the blockchain database. For other nodes, we can simply create their instances without creating the blockchain (They will automatically be synchronized when we start them). Assuming, you're starting all nodes on the same machine:

$ cd ~/dbfs
$ NODE=newyork   PRIMARY=1 mix do ecto.create, ecto.migrate, ecto.seed
$ NODE=london    PRIMARY=0 mix do ecto.create, ecto.migrate, ecto.seed
$ NODE=singapore PRIMARY=0 mix do ecto.create, ecto.migrate, ecto.seed

Get it Running

To finally start the backend nodes, you need to pass them a port, node name and boot-up config. You can start one node or multiple, depending on the requirements.

$ NODE=newyork PORT=3000 elixir --name [email protected] --erl "-config sys.config" –S mix phoenix.server

If you decide to run multiple nodes together, you also need to initialize the distributed Mnesia tables which are responsible for performing consensus across the network. First enter the REPL command-line of one of the running nodes, and enter the following:

iex> DBFS.Consensus.Global.setup

Finally, to start the Web-Client:

$ cd ~/dbfs-web
$ PORT=4000 yarn start

Usage

To use the application and upload/download files, they need to be encrypted using your private key. You can generate one using ssh-keygen:

$ ssh-keygen -b 1024 -t rsa -f key -q -N ""

What's Next?

Nothing. This was just an experiment and a learning experience. I'm not planning on doing any more work on this, but you're free to play around and learn from my mistakes (You should absolutely not make the same design decisions I made).

License

All code here is available as open source under the terms of the MIT License.