milindparikh / cellstore

Modern Data Warehouse

CellStore

In traditional data warehousing, it takes months for the IT department to build a single hypercube. With the CellStore, it takes minutes for the business users to build an infinite number of hypercubes.

The CellStore is entirely written in JSONiq and is hosted on 28.io. The deployment can be easily customized to deploy on your own 28.io account.

Resources

The founding paper of cell stores is available here. A CellStore with all SEC filings since 2010 is available at http://secxbrl.info. And the end-user documentation is available at http://28msec.github.io/cellstore/.

Installation

Prerequisites: NPM, Ruby (needed for SASS).

Windows Users: We advice you to run any commands in the Git Bash. Otherwise, the decryption of config.json won't work and you will have to setup the config.json manually. A working Ruby version for windows can be found at rubyinstaller.org.

$ gem install sass
$ npm install gulp -g
$ npm install && bower install

Configuration

To create a CellStore deployment <name> you need to add a config file config/<name>.json with the following example structure.

Encrypt your config file and add the encrypted config/<name>.json.enc to the repo:

$ export TRAVIS_SECRET_KEY=<secret>
$ gulp encrypt --build-id=test --config=<name>
$ git add config/<name>.json.enc

On github create a branch called test. With the matching config file name you have then created a production branch which automatically deploys to your production s3 bucket and 28.io project as defined in your configuration.

Optionally:

• add reports for your deployment to data/<name>/
• add API tests to queries/public/test/<name>/
• add e2e tests to tests/e2e/<name>

Environment Variables

The TRAVIS_SECRET_KEY environnement variable is mandatory in order to decrypt deployment files.

$ export TRAVIS_SECRET_KEY=<secret> # to decrypt / encrypt config files

You might want to set the following environment variables for convenience:

$ export CELLSTORE_BUILD_ID=<mybuild-id> # default build-id if --build-id=xyz is not provided
$ export CELLSTORE_CONFIG=<name> # default config if --config=xyz is not provided

Deployment

Create a CellStore deployment called test using the encrypted configuration in config/secxbrl.json.enc.

$ gulp setup --build-id=test --config=secxbrl

To remove a deployment once you are done:

$ gulp teardown --build-id=test --config=secxbrl

Development

To test the development version using the secxbrl.json configuration (run gulp setup first):

$ gulp server --build-id=mydemo --config=secxbrl

Testing

Run UI tests only:

gulp test --build-id=mydemo --config=secxbrl

Run a single UI test spec:

gulp test --build-id=mydemo --config=secxbrl --specs=tests/e2e/secxbrl/basic-scenario.js

Run unit test:

gulp test:unit --build-id=mydemo --config=secxbrl

Configuration Example

{
    "all": {
        "http-proxy": "http://proxyserver.com:3737",

        "s3-region": "us-east-1",
        "s3-bucketPrefix": "csms.example.com",
        "s3-key": "ADUREIGMKODJAEO43SAJ3",
        "s3-secret": "lkhsohqDFhsasdhaAFjaFajajfdWEhDFHAjhud4e4",
        "s3-reportsBucket": "e2eFailureTestReportBucket",

        "28-projectPrefix": "example-project",
        "28-email": "[email protected]",
        "28-password": "<password>",
        "28-portal-protocol": "http",
        "28-portal-project": "portal",
        "28-portal-domain": "28.io",
        "28-portal-api-prefix": "/api",

        "28-datasources": [
            {
                "category": "MongoDB",
                "name": "xbrl",
                "credentials": {
                    "conn-string": "set-lskjdfaoieurlkajpoidgja/db.example.com:27017,db.example.com:27017",
                    "db": "db",
                    "user": "dbuser",
                    "pass": "<password>"
                }
            },
            {
                "category": "ElasticSearch",
                "name": "xbrl",
                "credentials": {
                    "urls" : [ "es.0.example.com:10210", "es.1.example.com:10210" ],
                    "user" : "ESUser",
                    "pass" : "<password>"
                }
            }
        ],

        "cellstore-protocol": "https",
        "cellstore-profile": "generic",
        "cellstore-filteredAspects": "2",
        "cellstore-allowRegistration": "true",
        "cellstore-adminUser": "[email protected]",
        "cellstore-testUser": "[email protected]",

        "netdna-companyAlias": "mycompanyalias",
        "netdna-consumerKey": "klsjfoaoiqje8ahofdkgjoeqiutjae",
        "netdna-consumerSecret": "lskajforeiwutjgheqoiajejkweasdf",

        "sendmail-host": "smtp.example.com:995/tls/novalidate-cert",
        "sendmail-user": "[email protected]",
        "sendmail-password": "<password>",
        "sendmail-sender-email": "[email protected]",
        "sendmail-sender-name": "CellStore Example Setup"
    },
    "dev": {
        "cellstore-adminPassword": "<password>",
        "cellstore-testPassword": "<password>",
        "cellstore-testToken": "secret",

        "netdna-zone": "none"
    },
    "prod": {
        "cellstore-adminPassword": "<password>",
        "cellstore-testPassword": "<password>",
        "cellstore-testToken": "cdjfalsk-slke-4dkd-73jd-kjsdhfakhjde",

        "netdna-zone": "189474"
    }
}

Git Commit Messages

• Use a single sentence
• Use the present tense ("Add feature" not "Added feature")
• Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
• Limit the first line to 72 characters or less
• Reference issues and pull requests liberally
• Consider starting the commit message with an applicable emoji:
  • 🆕 :new: when writing a new feature
  • 💄 :lipstick: when improving the format/structure of the code
  • 🐎 :racehorse: when improving performance
  • 🚱 :non-potable_water: when plugging memory leaks
  • 📚 :books: when writing docs / documentation
  • 🐛 :bug: when fixing a bug
  • 🔥 :fire: when removing code or files
  • 💚 :green_heart: when fixing the CI build
  • ✅ :white_check_mark: when adding tests
  • 🔒 :lock: when dealing with security
  • ⬆️ :arrow_up: when upgrading dependencies
  • ⬇️ :arrow_down: when downgrading dependencies
  • 🔗 :link: when merging