amicks / speculator

API for predicting the next Bitcoin and Ethereum with machine learning and technical analysis

Speculator is an API for predicting the price trend of cryptocurrencies like Bitcoin and Ethereum.

Normal markets will also be added in future updates.

Getting Started

REST API

Start the Flask app then make requests:

python app.py
curl http://localhost:5000/api/public/predict -X GET

Dependencies

Make sure these packages are installed before running Speculator:

pip3 install delorean requests numpy tensorflow scikit-learn pandas flask flask-caching flask-restful flask-sqlalchemy psycopg2 webargs scipy

Routes Summary

• GET/DELETE: /api/private/market/
  • GET/DELETE: /api/private/market/?<int:id>
    • PUT/POST: /api/private/market/?<int:id>&<float:low>&<float:high>&<float:close>&<float:volume>
• GET: /api/public/predict/?<bool:use_db>&<str:model_type>&<str:symbol>&<str:unit>&<int:count>&<int:period>&<int:partition>&<int:delta>&<int:seed>&<int:trees>&<int:jobs>&<DelimitedList<str>:longs>

---

GET: /api/private/market/?<int:id>

Retrieves market data from optional id. When omitted, all entries will be returned.

Example Output:

[
    {
      "id": 1,
      "low": 2.0,
      "high": 10.0,
      "close": 6.0,
      "volume": 2.0
    },
    {
        "id": 2,
        "low": 23.0,
        "high": 40.0,
        "close": 33.0,
        "volume": 5.0
    }
]

POST: /api/private/market/?<int:id>&<float:low>&<float:high>&<float:close>&<float:volume>

Updates market data from required id and optional keyword arguments of low, high and close prices, and volume.

Example Output:

{
    "id": 1,
    "low": 2.0,
    "high": 10.0,
    "close": 6.0,
    "volume": 2.0
}

PUT: /api/private/market/?<int:id>&<float:low>&<float:high>&<float:close>&<float:volume>

Creates market data from required id and optional keyword arguments of low, high and close prices, and volume. A value of -1 clears the attribute.

Example Output:

{
    "id": 1,
    "low": 2.0,
    "high": 10.0,
    "close": 5.0,
    "volume": null
}

DELETE: /api/private/market/?<int:id>

Deletes market data from optional id. When omitted, all entries will be deleted.

Example Output:

{
    "status": "successful"
}

---

GET: /api/public/predict/?<bool:use_db>&<str:model_type>&<str:symbol>&<str:unit>&<int:count>&<int:period>&<int:partition>&<int:delta>&<int:seed>&<int:trees>&<int:jobs>&<DelimitedList<str>:longs>

Gets prediction of the next trend, including probabilities of various outcomes and test set accuracy.

Example Output:

{
    "trend": "bearish",
    "test_set_accuracy": 0.46153846153846156,
    "probabilities": {
        "bearish": 0.8,
        "neutral": 0.0,
        "bullish": 0.2
    }
}

All arguments are optional:

• use_db: Enables the use of DB market data from the private API
  • note: When True, arguments for automatic gathering of data will be disabled (like unit)
  • default: False
• model_type: Machine learning model to train
  • default: 'random_forest'
  • valid values: 'random_forest', 'rf', 'deep_neural_network', or 'dnn'
• symbol: Currency to predict
  • default: 'USDT_BTC'
  • valid values: Symbols/Conversions are available via Poloniex
• unit: Duration to predict from
  • default: 'month'
  • valid values: second, minute, hour, day, week, month, year
• count: units to predict from
  • default: 6
• period: Seconds for each chart candlestick
  • default: 86400
  • valid values: 300, 900, 1800, 7200, 14400, 86400
• partition: Market dates for each feature
  • note: A K-day RSI would need a partition of K
  • default: 14
• delta: Size of price neutral zone
  • note: Distinguishes between bearish/neutral/bullish trends
  • default: 25
• seed: Produces consistent results for ML models
  • note: When omitted (seed is None), values are inconsistent and not reproducible
  • default: None
• trees: Trees for the Random Forest model
  • note: Higher values (~65) are typically more accurate
  • default: 10
• jobs: CPU threads to use
  • default: 1
• longs: Uses long duration features
  • note: A K-day RSI would also include a 2K-day RSI (long RSI)
  • default: []

---

If you want to access the private API, simply make the DB connection an environment variable:

export SQLALCHEMY_DATABASE_URI='postgresql://username:password@host:port/db'

For a full list of SQLALCHEMY_DATABASE_URI formats, see SQLAlchemy's docs under "Connection URI Format".

Private API access with a DB is enabled by default in api/__init__.py Set ENABLE_DB = False to disable this.

---

I plan to start a server for anyone to access this without starting their own Flask server, but with only public access. Starting this on your own server with authentication for users (private access) will allow you to PUT/POST/DELETE your own market data and analyze that instead of the default. This is currently in development and will be extended in the future.

Python Usage

• program: python main.py
• package: pip3 install speculator

Example Output:

Use the --help flag for a complete list of optional arguments.

Project Structure

LICENSE
README.md

api
  \_ models
           \_ market.py
  \_ resources
              \_ market.py
              \_ trend.py
  \_ app.py
  \_ helpers.py

docs
  \_ CONTRIBUTING.md
  \_ analysis.md
  \_ example.md
  \_ example.py
  \_ utils.md

speculator
    \_ main.py
    \_ market.py
    \_ features
                \_ obv.py
                \_ rsi.py
                \_ sma.py
                \_ so.py
    \_ models
             \_ deep_neural_network.py
             \_ random_forest.py
    \_ tests
            \_ integration
                          \_ test_poloniex.py
            \_ unit
                   \_ test_date.py
                   \_ test_poloniex.py
                   \_ test_obv.py
                   \_ test_rsi.py
                   \_ test_sma.py
                   \_ test_so.py
                   \_ test_stats.py
    \_ utils
            \_ date.py
            \_ poloniex.py
            \_ stats.py

Contact for Feedback, Questions, or Issues

Feel free to send me a message on Reddit at /u/shneap. I am happy to hear any concerns, good or bad, in order to progress the development of this project.

Note: A website for a friendly user experience is in development

Contributing

Please see the contributing doc.

Basis

Part of this project is focused on an implementation of the research paper "Predicting the direction of stock market prices using random forest" by Luckyson Khaidem, Snehanshu Saha, and Sudeepa Roy Dey. I hope to gain insights into the accuracy of market technical analysis combined with modern machine learning methods.

Disclaimer

Speculator is not to be used as financial advice or a guide for any financial investments.

speculator's Issues

Poloniex api error while downloading data using python3

I am trying to run the Speculator library. While trying to run I came across the following error:

PC@PC:~/Speculator/speculator$ python3 main.py 
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
Traceback (most recent call last):
  File "main.py", line 92, in <module>
    raise SystemExit(main())
  File "main.py", line 59, in main
    count=args.count, period=args.period)
  File "/usr/local/lib/python3.5/dist-packages/speculator/market.py", line 48, in __init__
    self.json = self.get_json()
  File "/usr/local/lib/python3.5/dist-packages/speculator/market.py", line 59, in get_json
    self.period, self.symbol)[0]
  File "/usr/local/lib/python3.5/dist-packages/speculator/utils/poloniex.py", line 61, in chart_json
    json = requests.get(url).json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 808, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

The error what I think is coming from this code:

import logging
import requests
from speculator.utils import date

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def json_to_url(json, symbol):
    """ Converts a JSON to a URL by the Poloniex API 

    Args:
        json: JSON data as a list of dict dates, where the keys are
            the raw market statistics.
        symbol: String of currency pair, like a ticker symbol.

    Returns:
        String URL to Poloniex API representing the given JSON.
    """
    start = json[0]['date']
    end = json[-1]['date']
    diff = end - start

    # Get period by a ratio from calculated period to valid periods
    # Ratio closest to 1 is the period
    # Valid values: 300, 900, 1800, 7200, 14400, 86400
    periods = [300, 900, 1800, 7200, 14400, 86400]

    diffs = {}
    for p in periods:
        diffs[p] = abs(1 - (p / (diff / len(json)))) # Get ratio

    period = min(diffs, key=diffs.get) # Find closest period
    
    url = ('https://poloniex.com/public?command'
           '=returnChartData&currencyPair={0}&start={1}'
           '&end={2}&period={3}').format(symbol, start, end, period) 
    return url

def chart_json(start, end, period, symbol):
    """ Requests chart data from Poloniex API
    
    Args:
        start: Int epoch date to START getting market stats from.
            Note that this epoch is FURTHER from the current date.
        end: Int epoch date to STOP getting market stats from.
            Note that this epoch is CLOSER to the current date.
        period: Int defining width of each chart candlestick in seconds.
            Valid values: 300, 900, 1800, 7200, 14400, 86400
        symbol: String of currency pair, like a ticker symbol.

    Returns:
        Tuple of (JSON data, URL to JSON).
        JSON data as a list of dict dates, where the keys are
        the raw market statistics.
        String URL to Poloniex API representing the given JSON.
    """
    url = ('https://poloniex.com/public?command'
           '=returnChartData&currencyPair={0}&start={1}'
           '&end={2}&period={3}').format(symbol, start, end, period) 
    logger.debug(' HTTP Request URL:\n{0}'.format(url))
    json = requests.get(url).json()
    logger.debug(' JSON:\n{0}'.format(json))

    if 'error' in json:
        logger.error(' Invalid parameters in URL for HTTP response')
        raise SystemExit
    elif all(val == 0 for val in json[0]):
        logger.error(' Bad HTTP response.  Time unit too short?')
        raise SystemExit
    elif len(json) < 1: # time to short
        logger.error(' Not enough dates to calculate changes')
        raise SystemExit

    return json, url

def parse_changes(json):
    """ Gets price changes from JSON

    Args:
        json: JSON data as a list of dict dates, where the keys are
            the raw market statistics.

    Returns:
        List of floats of price changes between entries in JSON.
    """
    changes = []
    dates = len(json)
    for date in range(1, dates): 
        last_close = json[date - 1]['close']
        now_close = json[date]['close']
        changes.append(now_close - last_close)
    logger.debug('Market Changes (from JSON):\n{0}'.format(changes))
    return changes

def get_gains_losses(changes):
    """ Categorizes changes into gains and losses

    Args:
        changes: List of floats of price changes between entries in JSON.

    Returns:
        Dict of changes with keys 'gains' and 'losses'.
        All values are positive.
    """
    res = {'gains': [], 'losses': []}
    for change in changes:
        if change > 0:
            res['gains'].append(change)
        else:
            res['losses'].append(change * -1)
    logger.debug('Gains: {0}'.format(res['gains']))
    logger.debug('Losses: {0}'.format(res['losses']))
    return res

def get_attribute(json, attr):
    """ Gets the values of an attribute from JSON

    Args:
        json: JSON data as a list of dict dates, where the keys are
            the raw market statistics.
        attr: String of attribute in JSON file to collect.

    Returns:
        List of values of specified attribute from JSON
    """
    res = [json[entry][attr] for entry, _ in enumerate(json)]
    logger.debug('{0}s (from JSON):\n{1}'.format(attr, res))
    return res

def get_json_shift(year, month, day, unit, count, period, symbol):
    """ Gets JSON from shifted date by the Poloniex API

    Args:
        year: Int between 1 and 9999.
        month: Int between 1 and 12.
        day: Int between 1 and 31.
        unit: String of time period unit for count argument.
            How far back to check historical market data.
            Valid values: 'hour', 'day', 'week', 'month', 'year'
        count: Int of units.
            How far back to check historical market data.
        period: Int defining width of each chart candlestick in seconds.
        symbol: String of currency pair, like a ticker symbol.

    Returns: JSON, list of dates where each entry is a dict of raw market data.
    """
    epochs = date.get_end_start_epochs(year, month, day, 'last', unit, count)
    return chart_json(epochs['shifted'], epochs['initial'],
                      period, symbol)[0]

The issue was with Poloniex api, and I searched many sources for the resolution but could not find anything working. For example, some say that changing IP or using VNP will help but it isn't. Second I tried using the dedicated server, still the issue persists.

Kindly, suggest me what is the best possible technique available for resolving my issue.

Is there a way to get rid of the Google captcha while requesting through requests library? I guess that is the issue.

python3 app.py

Run python3 app.py on my mac.

And i got the following as output.

$ python3 app.py Traceback (most recent call last): File "app.py", line 2, in <module> from api import api, cache, ENABLE_DB, db ModuleNotFoundError: No module named 'api'

Could you shed some light on how to bring up the rest service for btc/etc price predication please?

Error Classification metrics can't handle a mix of continuous-multioutput and multiclass targets

Hi,
I followed the step to use Speculator, but when I run it, I got the error in the image below...
All files are original ones, I did not modify anything.

Test set accuracy

Hi,
Do you manage to get a good test set accuracy with a relatively small delta (-d) ?
If you set a high delta, there's a good chance the next trend will be neutral isn't it ?

day, hour, second - Units that are not "closed" are not working.

Is there any point of having day, hour, second available, when they do not have a "closed" value? Cause they all seem to produce an error. Or should they take the last available timestamp? For example, if you do this, you get an error:

$ python main.py --unit hour
Traceback (most recent call last):
File "main.py", line 106, in
raise SystemExit(main())
File "main.py", line 82, in main
x = x.drop(['close'], axis=1)
File "/home/anaconda3/envs/speculator/lib/python3.6/site-packages/pandas/core/generic.py", line 2530,
in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/home/anaconda3/envs/speculator/lib/python3.6/site-packages/pandas/core/generic.py", line 2562,
in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/home/anaconda3/envs/speculator/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3
741, in drop
labels[mask])
ValueError: labels ['close'] not contained in axis

Select best delta

How to select best delta when predict other coin ex: USDT_ZEC...

new complementary tool

I want to offer a new point of view, and my colaboraty

Why this stock prediction project ?

Things this project offers that I did not find in other free projects, are:

• Testing with +-30 models. Multiple combinations features and multiple selections of models (TensorFlow , XGBoost and Sklearn )
• Threshold and quality models evaluation
• Use 1k technical indicators
• Method of best features selection (technical indicators)
• Categorical target (do buy, do sell and do nothing) simple and dynamic, instead of continuous target variable
• Powerful open-market-real-time evaluation system
• Versatile integration with: Twitter, Telegram and Mail
• Train Machine Learning model with Fresh today stock data

https://github.com/Leci37/stocks-prediction-Machine-learning-RealTime-telegram/tree/develop