dreamalligator / marian

just a personal project, not ready for consumption

👸 Marian

Find documentation at nebulousdog.github.io/marian, and github.com/nebulousdog/marian.

Install

Clone

For devs this is the easiest.

git clone --depth=1 [email protected]:nebulousdog/marian.git

PyPi

Marian is distributed via PyPi, and can be installed with pip install marian.

Running Locally

pip install pipenv
pipenv install --dev
pipenv run flask run

You'll now see the dynamically created routes.

This project is primarily a Flask app built on top of the Fast Arrow client. See About This Project for more on that.

You can now visit your locally running app at http://127.0.0.1:5000.

Deploy

simply run pipenv run python deploy.

if this is your first time running this, you will be prompted for your Digital Ocean API token.

> pipenv run python deploy.py
Loading .env environment variables…
Digital Ocean API token not found, retrieve your token from digitalocean.
visit https://cloud.digitalocean.com/account/api/tokens.
enter token:

Take note of the line that says droplet is now active at <ip address>.

Post Deploy

TODO: working on that right now.

Authentication

Via Local Web App

Visit the /login (http://localhost:5000/login) page to authenticate with Robinhood. Your client is subject to the same session durations that exist for the normal Robinhood app.

If you're navigating the app's frontend and you have no authenticated session yet, or need to re-authenticate, you will be automatically redirected to the login page as soon as you try to visit a route that requires authentication.

There are a few reasons that a frontend interface is provided:

• logging into Robinhood with a username/password allows a new user to immediately test Marian.
• this is the easiest UI way to create/regenerate auth tokens.
• all generated routes are supplied on the home page to immediately view what is available. these are clickable, which is nice for a new user.
• pleasant development environment.

That said though; this mode is only available for local development. My intent is to make things incredibly easy for new users or developers. However, after you're set up, auth tokens are meant to be the primary authentication method in production.

Via API

Simply append your security token as a query param. For example, <marian path>?<query params>&token=<token>.

To generate this token, run the app locally and visit http://localhost:5000/token. Here you can see your existing token, or generate a new one.

All old tokens will be invalidated after a token is regenerated.

API

Routes

You can find all supported routes by visiting the base route http://localhost:5000 locally.

Format

By default the output format is JSON. To output in a format that is more readily accepted by Google Sheets, you can supply a CSV param. For example, <marian path>?<query params>&csv. Notice that the CSV param simply must exist. You may also write csv=true if you wish, but this is redundant.

Working with Google Sheets

You'll need two things from the deploy; the secure token generated, and the IP address of the droplet.

You can now import data into your sheet.

Example formula: =IMPORTDATA("<droplet ip>/stock_position/all&token=<token>")

CLI

A CLI wrapper is provided with some handy functionality. This makes things easy for all, because only devs are going to want to use the Pipenv setup. Devs can also utilize the cli though! See the extended notes at nebulousdog.github.io/marian.

See the available Marian cli commands with marian --help.

About This Project

When writing Marian, my goal was just to get my Robinhood data into a Google Sheet. If you've used Google Sheets at all, this is a PITA. Now it isn't.

There are some additional features that I added for my own learning and convenience that I'm proud to share:

• no plaintext saving of Robinhood usernames and passwords
• dynamically built routes. anything that Fast Arrow supports is automatically supported by Marian.
• deploy scripts
• API token generation
• CI and tests
• secure sessions and authentication
• frontend interface for developing
• headless production mode
• JSON and CSV API formats

Upcoming

The full list is located at https://github.com/nebulousdog/marian/issues.

Configuration

Since Marian is a Flask app, all Flask documentation applies for custom configuration. See http://flask.pocoo.org/docs/latest.

Tests

1. pipenv run tests

Linting

1. pipenv run lints

Releasing

1. pipenv run bump_version

License

MIT

marian's Issues

static application for github pages

add a static config option, so can host an example project on github pages. when build frozen site, the routes info builder for the index template will be able to link to all pages.

there is a static flask frozen project that will likely be what I use https://pythonhosted.org/Frozen-Flask/.

move example token file into tests dir

move DIGITALOCEAN_TOKEN_EXAMPLE file into spec path, or arrange the test so that file is created and then read in the tests.

CLI

use Click or such to add CLI. Probably better than doing python deploy.py and such. It will mean better testing of setup.py (#31) as well since required to do.

cookiecutters

Is any of this appropriate for a cookiecutter? Should I use any existing ones for cli/api etc? Also, look at some for best practices or expectations.

https://cookiecutter.readthedocs.io/en/latest/

tests

• write a few pytest unit tests
• add to .travis.yml scripts
• look into tox, https://tox.readthedocs.io/en/latest/example/pytest.html
• look into doctests https://docs.python.org/3/library/doctest.html

codecov patch

drop codecov/patch failure from master commits, https://docs.codecov.io/docs/commit-status

examples / tutorials

need to provide examples, or detailed instructions

• cli
• adding custom routes
• using marian itself as a dependency
• using with google sheets
• authentication
• deploying

Click FlaskGroup vs Marian cli group

atm marian cli is separate from flask command. however, maybe it should be merged. see http://flask.pocoo.org/docs/1.0/cli/#custom-commands.

bash & zsh compatibility

I use zsh locally, and bash on the remote servers. This could cause me to accidentally not know that there is something incompatible if not running everything in both shells. Should do a pass on any shell scripts that they are good.

exception best practices

keep raising requests.exceptions.HTTPError for example, all the way up the stack? or raise a native error, or raise my own defined exception?

• https://docs.python.org/3/library/exceptions.html
• https://docs.python.org/3/tutorial/errors.html#user-defined-exceptions
• https://github.com/westonplatter/fast_arrow/blob/master/fast_arrow/exceptions.py

clean up pylint disables

low priority, but remove all pylint disables.

• global-statement: using for memoization of fast_arrow client instance only. likely will convert that module to utilize a class and save to instance var.
• missing-docstring: I want docstrings for the primary project, but not necessarily tests. issue for splitting the rule apart, would be content with just disabling docstrings for modules. pylint-dev/pylint#1164. would rather extend the base rcfile like how eslint configs can extend another config for different rules in a directory.
• unused-variable: flask route methods arent unused. The decorators hook them up to the corresponding routes. didnt see any flask plugins for this, may need to make own.
• redefined-outer-name: caused by @pytest.fixture decorators.

cli option vs argument best practices

get ocd about whether current cli's options are actually options, or arguments are actually arguments, flags, etc.

example using or creating Environmental, Social, and Governance factors

example to implement https://en.wikipedia.org/wiki/Environmental,_social_and_corporate_governance factors

Environmental, Social, and Governance (ESG) refers to the three central factors in measuring the sustainability and societal impact of an investment in a company or business.

Publish to PyPi

• publish project to PyPi
• CI publishes versions upon passing

create_app triggering twice

the create_app method is triggering twice. I dont think it was before, but maybe I just noticed from extra print logging?

marian-lS42cu4F 💀 > flask run
 * Serving Flask app "marian" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 187-526-671
Maid Marian started.
Maid Marian started.

include setup.py in CI

include setup.py in CI

linting

• python linting
• add linting step to travis-ci

automatic api docs based on route infos

could do one or all of these similar things:

• automated api docs based on iterating on routes
• return a helpful template of missing params when invalid route if the root of it is valid.
• using the second idea as part of the first, and do both.

references:

• http://flask.pocoo.org/docs/1.0/api/#flask.Flask.url_map

autodocs

directly invoke autodoc utils instead of using sphinx cli, or the current docutils publish_file method.

deploy scripts

automated deploy scripts

• local
• production
• add to ci scripts

headless mode

headless mode and/or cli.

generate tokens for api access

need to generate a token to pass as a param when used in google sheets to access the api.

api formatting param?

optionally use a csv, xml, etc flag for easier google-sheets importing.

instructions on extending routes

instructions on extending routes or writing plugins. check if flask extension registry should be part of process.

flask application

basic flask application

watchlist endpoint

• watchlist endpoint
• same formatting as positions endpoint
• maybe make PR for https://github.com/westonplatter/fast_arrow/

continuous integration

• make repo public for Travis-CI
• #2 unit tests
• #3 linting
• #4 deployment test
• tagging on success
• automated deployment to production

authentication without plaintext .ini file

authentication without plaintext .ini file

references:

• http://flask.pocoo.org/docs/1.0/quickstart/#sessions
• https://github.com/pallets/werkzeug/blob/master/src/werkzeug/security.py
• https://github.com/maxcountryman/flask-bcrypt

robinhood authentication

secure robinhood authentication

minimal production footprint

after production env setup is finished, try to minimize any footprint.

reverse proxying

follow up to #4. add reverse proxying.

• http://exploreflask.com/en/latest/deployment.html#making-gunicorn-public

utilize token to persist authentication

• utilize the login token given when signing in to persist authentication
• refreshing
• mfa?

{
  "access_token": "xxxxxx", 
  "backup_code": null, 
  "expires_in": 86400, 
  "mfa_code": null, 
  "refresh_token": "xxxxxx", 
  "scope": "internal", 
  "token_type": "Bearer"
}

docs deprecation warning

$ python ./docs.py
generating docs...
/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/docutils/io.py:245: DeprecationWarning: 'U' mode is deprecated
  self.source = open(source_path, mode, **kwargs)
/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/docutils/io.py:245: DeprecationWarning: 'U' mode is deprecated
  self.source = open(source_path, mode, **kwargs)
done with docs.
The command "python ./docs.py" exited with 0.

full test coverage

full test coverage.

check if these help:

• https://flask-testing.readthedocs.io/en/latest/
• https://flask-webtest.readthedocs.io/en/latest/

RESTful API

basic RESTful API

• https://flask-restful.readthedocs.io

adhere to json api

• https://jsonapi.org/
• https://github.com/miLibris/flask-rest-jsonapi/

caching

caching of requests will greatly improve this applications functionality.