suchak1 / hyperdrive Goto Github PK

For scripts using multiple processes, use a lock or manager to manage a shared variable and figure out if > 5% of data saves / function calls fail.

Installing hyperdrive currently installs to site-packages/src.
Change setup.py to install to site-packages/hyperdrive.

Use real IEXCloud endpoint and API key for intraday tests.

Binance.US decreased their fee to 0%. Dynamically determine the fee via API and use that.

Add ability to trade with decentralized exchange (Bisq) through API using web3.py and 1inch (WBTC - USDC).

Create split worker that updates historical data accordingly (dividends, eod ohlc, intraday ohlc)

Get historical dividend data for all stocks in portfolio using Yahoo Finance (yahooquery package) or Nasdaq. Save table as csv in data/ and preload as part of load_portfolio.

Looking into import pandas_datareader as pdr as well.

Create 2 and 3 dimensional models as part of a script that runs after create_model.py.
(First, use pca, lda, tsne, umap to reduce dimensionality.)
(Will add 1-2 hrs to workflow runtime or use matrix and decide whether to use dimensionality reduction in create_model.py thru env var (DIMENSIONALITY_REDUCTION: ['None', 'PCA', 'LDA', 'TSNE', 'UMAP'])
Create csvs or jsons for 2d and 3d plottable data or plotly html file.

Fix symbols to include only symbols in holdings not all in portfolio history.

• In PR branch, try pipeline w random file change, stefanzweifel/git-auto-commit-action, git pull, then ...?
• Try replacing version bump action in pipeline with this one: https://github.com/mathieudutour/github-tag-action

Best soln:
Try anothrNick/github-tag-action w dry run enabled first, then use version output (steps.bumpVersion.outputs.new_tag) for INPUT_TAGGING_MESSAGE in stefanzweifel/git-auto-commit-action, then create new release (action from release.yml)

Create a function to walk through the portfolio and simulate the actions that led to the current holdings. Might be useful to create the simulation functions for buying and maybe selling here.

buy(symbol, quantity, price)

Use 9:30pm EST price for update_api script

Save data in S3 rather than in repo
Smart load of data (check if data is 1 day or 1 week old?) - local
Mock data for CI tests + step to download relevant data files used in tests from s3 ('AAPL.csv')

Daily update to csv file

Make an Oracle class to deal with common model operations and data prep.

Pass in feature names as env variable and open source as much as possible.

Use Stocktwits api to get social sentiment towards diff symbols.

Namely MARA, RIOT, ARK*

Probably from this commit: 9c338b7

Theory: The time related update didn't work, and the response from Polygon is empty for '1d', so MarketData obj is either returning nothing or most recent row in cached data (csv) instead of previous day's data directly from Polygon.

Make a Config class that loads all environment variables in config.env and import/load config when needed. Move Pathfinder into its own file.

• Add quotes around RH_PASSWORD in util/encrypt.sh and util/decrypt.sh.

Polygon.io intraday update failed for BRK.B.
sleep length must be non-negative

https://github.com/suchak1/hyperdrive/runs/2249034869?check_suite_focus=true#step:6:1160

Fetch last repo release from s3
Run script on schedule

Check easily accessible sources. Look into adding 💱 ⬅️🌊 and 💱➡️🌊 (ex. inflow vs outflow)

1. Set up Travis CI or GitHub Actions CI/CD
2. Pass robinhood.pickle as encrypted file to build agent
3. Set up test framework

Use 9:30pm EST price during prediction and model creation.

• Clean up model to use preprocess fx instead of undersample.
• Add SMOTE
• Use ddof=0
• Don't convert date to int64 for derivative, just assume dx = 1
• Don't use PCA for final

Create a function that calculates n-day moving average for a symbol.

moving_avg(n, symbol, date=today)

Look into use of the ta library for incorporation of even more indicators.

Use Polygon.io as a secondary data provider to add double redundancy to the data collection pipeline.

google "python library docs generator", pick best one, and refactor library to autogenerate docs

get a linter to make sure functions are compliant w doc generation during pr validation?

generate html doc, push to forcepu.sh bucket (better for making docs public acl read - remove delete flag from deploy units workflow) or hyperdrive bucket (better for showing docs from hyperdrive) and serve from forcepush - pull in locally as part of build process s3 download w creds (hyperdrive bucket) or just make request to object url (public access - forcepush bucket) or git checkout hyperdrive and generate docs on the fly

get codecov badge

Try deep learning or reinforcement learning or deep reinforcement learning.

At each point in time, find total dividend / total account value as percentage. Then, divide resulting value at each time t by current time in years. This is the annual dividend yield for all t.

Create a function to plot historicals for a given instrument or symbol.

plot(symbol, start, end, instrument=None)

If Fri or Mon is a holiday, then the OHLC script will produce an error on Mon or Tue bc there is no market data returned during the 3 day buffer.

Check if market holiday is in date range and don't produce error.

Find a way to get optimal position sizing for each time t with size 0 <= s <= 1.0 and s being the proportion of quote asset used to buy base asset or base asset sold for quote asset.

Consider creating poetry.lockfile and use that to manage dependencies

Make model auto update every x months

• Remove mention of "Robinhood"
• Describe as library for algotrading research and execution
• Provide example of how to get data (iex env var), store data (s3 and iex env var), create model, backtest model predictions, execute order based on prediction, visualize research
• Use new logo

Split script reports a failure bc <5% of symbols tracked have had stock splits in the last 3 months. Find a solution to stop false positive failures. Reduce threshold to 1%?

Abstract away common script logic,

Use this: https://github.com/actions/cache/blob/master/examples.md#python---pip

We should make a Scarlett class with an init function that

1. authenticates the current user with Robinhood
2. loads the user's current portfolio holdings
3. loads their portfolio history
4. loads the historical data for each instrument in their portfolio history
5. creates an instrument: symbol lookup table for each instrument above

In execute_order script, should_order script should consider account balance when deciding to order (BTC balance when selling, USD balance when buying) after consecutive same signals.

Perhaps support TSLA, AAPL, QQQ, SPY, or universe portfolio management model.