weilonge / unidisk

Mount a cloud storage as a filesystem.

unidisk

This project can help you to use Cloud Storage in a new way and provide the following features:

• Available storages

  • JSON FS, Dropbox

• Data/Meta Cache to improve response time

• Read-only (or support write-able file system partially)

• Adjustable Data Cache Pool size

• Prefetch data to improve the performance

• Download blocks in multi-threading

• Support Linux and MacOSX

Prerequisite

MacOSX

• OSXFUSE
• brew install pkg-config

Linux

• sudo apt-get install libfuse-dev

Prepare settings.json

• Copy settings.json to your home folder and customize cache_path.

$ mkdir ~/.unidisk && cp dist/settings.json.SAMPLE ~/.unidisk/settings.json

Let's start

unidisk supports three kinds of storage to access: SampleJSON FS and Dropbox. The following instructions will guide you how to use them.

Sample JSON FS

• Please prepare a valid JSON file or get it from dist/samplefs.json
• Give the absolute path to JSONPath of Sample profile in settings.json
• Use the command to mount JSON FS with your sample JSON file:

$ ./udFuse.js -p Sample [mount point]

• The storage is ready at the mount point you gave.

Dropbox

• Please apply a Dropbox development account , and keep your API_KEY and API_SECRET
• Use this command to get the authorization link and browser the link then login your account

$ ./ud.js Dropbox getAuthLink [API_KEY]
== Result ====================
{ authLink: 'https://www.dropbox.com/1/oauth2/authorize?client_id=API_KEY&response_type=code' }
==============================

• Keep device_code shown in the page after login and authrization:

QrlXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

• Use this command to get the acess_token:

$ ./ud.js Dropbox getAccessToken [API_KEY] [API_SECRET] [DEVICE_CODE]
== Result ====================
{ accessToken: 'QrlXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYYYYYYYYYYYYYYYYYYYYYYYYYYYY' }
==============================

• Edit ~/.unidisk/settings.json and add the line in your Dropbox profile (e.g. MyDropbox):

token: "QrlXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYYYYYYYYYYYYYYYYYYYYYYYYYYYY"

• Use this command to mount Dropbox:

$ ./udFuse.js -p MyDropbox [mount point]

• The storage is ready at the mount point you gave.

Reference

• FUSE: Filesystem in Userspace

unidisk's Issues

Refresh remote file meta and file content.

Dropbox provides an API to long-poll the file changes:
https://www.dropbox.com/developers-v1/core/docs#longpoll-delta

Read-Write Support

When 'open' operation is triggered, a request ID should be stored and wait for 'write' and 'close' operations come.

During file writing, some file content can be uploaded with chunked-upload API.
https://www.dropbox.com/developers-v1/core/docs#chunked-upload

Once the file is closed, all content should be uploaded or keep uploading and fire a 'commit' request to finish the file uploading.
https://www.dropbox.com/developers-v1/core/docs#commit-chunked-upload

Refine README for unidisk

Some points have to be written down:

• how to get token, install, launch
• introduction for current support storage
• development guide to add a new storage.
• configuration description

Multiple instances support

Google Spreadsheets Data Plugin

Use this link to get a google spreadsheet data in JSON format:
http://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values?alt=json

Example format:

{
    "version": "1.0",
    "encoding": "UTF-8",
    "feed": {
        "xmlns": "http://www.w3.org/2005/Atom",
        "xmlns$openSearch": "http://a9.com/-/spec/opensearchrss/1.0/",
        "xmlns$gsx": "http://schemas.google.com/spreadsheets/2006/extended",
        "id": {
            "$t": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values"
        },
        "updated": {
            "$t": "2015-10-04T06:20:33.843Z"
        },
        "category": [{
            "scheme": "http://schemas.google.com/spreadsheets/2006",
            "term": "http://schemas.google.com/spreadsheets/2006#list"
        }],
        "title": {
            "type": "text",
            "$t": "工作表1"
        },
        "link": [{
            "rel": "alternate",
            "type": "application/atom+xml",
            "href": "https://docs.google.com/spreadsheets/d/[SPREADSHEETS_KEY]/pubhtml"
        }, {
            "rel": "http://schemas.google.com/g/2005#feed",
            "type": "application/atom+xml",
            "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values"
        }, {
            "rel": "http://schemas.google.com/g/2005#post",
            "type": "application/atom+xml",
            "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values"
        }, {
            "rel": "self",
            "type": "application/atom+xml",
            "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values?alt\u003djson"
        }],
        "author": [{
            "name": {
                "$t": "weilonge"
            },
            "email": {
                "$t": "[email protected]"
            }
        }],
        "openSearch$totalResults": {
            "$t": "4"
        },
        "openSearch$startIndex": {
            "$t": "1"
        },
        "entry": [{
            "id": {
                "$t": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cokwr"
            },
            "updated": {
                "$t": "2015-10-04T06:20:33.843Z"
            },
            "category": [{
                "scheme": "http://schemas.google.com/spreadsheets/2006",
                "term": "http://schemas.google.com/spreadsheets/2006#list"
            }],
            "title": {
                "type": "text",
                "$t": "Sharon"
            },
            "content": {
                "type": "text",
                "$t": "age: 12, gender: Female"
            },
            "link": [{
                "rel": "self",
                "type": "application/atom+xml",
                "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cokwr"
            }],
            "gsx$name": {
                "$t": "Sharon"
            },
            "gsx$age": {
                "$t": "12"
            },
            "gsx$gender": {
                "$t": "Female"
            }
        }, {
            "id": {
                "$t": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cpzh4"
            },
            "updated": {
                "$t": "2015-10-04T06:20:33.843Z"
            },
            "category": [{
                "scheme": "http://schemas.google.com/spreadsheets/2006",
                "term": "http://schemas.google.com/spreadsheets/2006#list"
            }],
            "title": {
                "type": "text",
                "$t": "Tim"
            },
            "content": {
                "type": "text",
                "$t": "age: 8, gender: Male"
            },
            "link": [{
                "rel": "self",
                "type": "application/atom+xml",
                "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cpzh4"
            }],
            "gsx$name": {
                "$t": "Tim"
            },
            "gsx$age": {
                "$t": "8"
            },
            "gsx$gender": {
                "$t": "Male"
            }
        }, {
            "id": {
                "$t": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cre1l"
            },
            "updated": {
                "$t": "2015-10-04T06:20:33.843Z"
            },
            "category": [{
                "scheme": "http://schemas.google.com/spreadsheets/2006",
                "term": "http://schemas.google.com/spreadsheets/2006#list"
            }],
            "title": {
                "type": "text",
                "$t": "Gina"
            },
            "content": {
                "type": "text",
                "$t": "age: 10, gender: Female"
            },
            "link": [{
                "rel": "self",
                "type": "application/atom+xml",
                "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/cre1l"
            }],
            "gsx$name": {
                "$t": "Gina"
            },
            "gsx$age": {
                "$t": "10"
            },
            "gsx$gender": {
                "$t": "Female"
            }
        }, {
            "id": {
                "$t": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/chk2m"
            },
            "updated": {
                "$t": "2015-10-04T06:20:33.843Z"
            },
            "category": [{
                "scheme": "http://schemas.google.com/spreadsheets/2006",
                "term": "http://schemas.google.com/spreadsheets/2006#list"
            }],
            "title": {
                "type": "text",
                "$t": "Frank"
            },
            "content": {
                "type": "text",
                "$t": "age: 6, gender: Male"
            },
            "link": [{
                "rel": "self",
                "type": "application/atom+xml",
                "href": "https://spreadsheets.google.com/feeds/list/[SPREADSHEETS_KEY]/od6/public/values/chk2m"
            }],
            "gsx$name": {
                "$t": "Frank"
            },
            "gsx$age": {
                "$t": "6"
            },
            "gsx$gender": {
                "$t": "Male"
            }
        }]
    }
}

Apply build system

New build system have to support the following requirement:

• Better installation process - get token, generate configuration file.
• launch sample code for web version

debug log

WebDAV

Apply jshint