tardis-dev / tardis-python

Python client for tardis.dev - historical tick-level cryptocurrency market data replay API.

tardis-client

Python client for tardis.dev - historical tick-level cryptocurrency market data replay API. Provides fast, high level and developer friendly wrapper for more low level HTTP API with local file based caching build in.

Installation

Requires Python 3.7.0+ installed.

pip install tardis-client

Usage

import asyncio
from tardis_client import TardisClient, Channel

async def replay():
    tardis_client = TardisClient()

    # replay method returns Async Generator
    # https://rickyhan.com/jekyll/update/2018/01/27/python36.html
    messages = tardis_client.replay(
        exchange="bitmex",
        from_date="2019-06-01",
        to_date="2019-06-02",
        filters=[Channel(name="trade", symbols=["XBTUSD","ETHUSD"]), Channel("orderBookL2", ["XBTUSD"])],
    )

    # this will print all trades and orderBookL2 messages for XBTUSD
    # and all trades for ETHUSD for bitmex exchange
    # between 2019-06-01T00:00:00.000Z and 2019-06-02T00:00:00.000Z (whole first day of June 2019)
    async for local_timestamp, message in messages:
        # local timestamp is a Python datetime that marks timestamp when given message has been received
        # message is a message object as provided by exchange real-time stream
        print(message)

asyncio.run(replay())

API

tardis-client package provides TardisClient and Channel classes.

from tardis_client import TardisClient, Channel

TardisClient

Optional client constructor parameters.

name	type	default value	description
api_key (optional)	string	""	optional string containing API key for tardis.dev API. If not provided only first day of each month of data is accessible (free access)
cache_dir (optional)	string	<os.tmpdir>/.tardis-cache	optional string with path to local dir that will be used as cache location. If not provided default temp dir for given OS will be used.

Example:

# creates new client instance with access only to sample data (first day of each month)
tardis_client = TardisClient()

# creates new client with access to all data for given API key
tardis_client = TardisClient(api_key="YOUR_API_KEY")

# creates new client with custom cache dir
tardis_client = TardisClient(cache_dir="./cache")

• tardis_client.clear_cache()

  Removes local file cache dir and it's contents.

  Example:

  tardis_client = TardisClient()
  
  tardis_client.clear_cache()

• tardis_client.replay(exchange, from_date, to_date, filters=[])

  Replays historical market data messages for given replay arguments.

  Returns Async Generator with named tuples (namedtuple("Response", ["local_timestamp", "message"])).

  • local_timestamp is a Python datetime object specyfying when message has been received from the exchange real-time data feed.

  • message is Python dict with parsed JSON that has exactly the same format as message provided by particular exchange's real-time data feed.

    replay method parameters:

    name	type	default value	description
    exchange	string	-	requested exchange name - Use /exchanges API call to get allowed exchanges ids
    from_date	string	-	requested UTC start date of data feed - valid ISO date string, eg: 2019-04-05 or 2019-05-05T00:00:00
    to_date	string	-	requested UTC end date of data feed - valid ISO date string, eg: 2019-04-05 or 2019-05-05T00:00:00
    filters (optional)	List[Channel]	[]	optional filters of requested data feed. Use /exchanges/:exchange API call to get allowed channel names and symbols for requested exchange

    Channel class

    Channel class constructor parameters.

    name	type	description
    name	string	Use /exchanges/:exchange API call to get allowed channel names and symbols for requested exchange
    symbols	List[string]	Use /exchanges/:exchange API call to get allowed channel names and symbols for requested exchange

    Channel(name="trade", symbols=["XBTUSD","ETHUSD"])
    Channel("orderBookL2", ["XBTUSD"])

FAQ

How to debug it if something went wrong?

tardis-client uses Python logging on DEBUG level for that purpose. In doubt please create issue in this repository with steps how to reproduce the issue.

Where can I find more details about tardis.dev API?

Check out API docs.

License

MPL-2.0

tardis-python's Issues

writes millions of temp files, never cleans up

When sync'ing large amounts of data, for example all PERP symbol data from FTX for a few months, tardis-client never cleans up its temp files in /tmp/.tardis-cache/. Left running for a day, there is currently over 3M files on my server, and left running for multiple days without manually cleaning up, it can easily run a disk out of inodes.

Rename to tardis

Given new roadmap (open source real-time data handling, normalization etc) I'd like to rename this lib to tardis but name is already taken on https://pypi.org/project/tardis/ . @cedk assuming you don't actively use this name on pypi would you consider releasing it? Thanks!

[Urgent] Please fix compatibility with aiofiles. See log below

Because no versions of tardis-client match >1.3.1,<2.0.0 tardis-client (1.3.1) depends on aiofiles (>=0.4.0,<0.5.0), tardis-client (>=1.3.1,<2.0.0) requires aiofiles (>=0.4.0,<0.5.0).

And because sanic (22.3.2) depends on aiofiles (>=0.6.0)
and no versions of sanic match >22.3.2,<23.0.0, tardis-client (>=1.3.1,<2.0.0) is incompatible with sanic (>=22.3.2,<23.0.0).
So, because poseidon depends on both sanic (^22.3.2) and tardis-client (^1.3.1), version solving failed.

Frequent ConnectionResetErrors and EOFErrors

Hi. I'm trying to replay Deribit "trades" and "book" channel for all symbols active on '2021-05-31'. I am hit with ConnectionResetError or EOFError everytime I'm trying to do replay and write into a file. Moreover, these only come into picture when I'm replaying for all symbols. When doing it for only ETH-PERPETUAL and BTC_PERPETUAL, haven't faced this issue.

Here is the stack trace for EOFError:

Here is the stack trace for ConnectionResetError:

1. 
2. 

python==3.8.3
tardis-client==1.2.12

I'm assuming the EOFError comes when the caching is not done properly. Not sure about the ConnectionResetError.
Can you suggest ways in which these can be avoided?

Google Colab support

urllib - HTTP Error 414

Hi. I'm trying to replay Deribit "trades" and "book" channel for 2019-03-30 for all active symbols on that day (filtered using Instruments Metadata API). Hence I have around 312 symbols for that day for which I'm trying to replay the above 2 channels.

Stack Trace:

python==3.8.3
tardis-client==1.2.12

The only change in the replay params is the list of symbols for both the channels under filters.

Is there a way I can replay data for all active symbols for a particular day for both these channels using some other approach or other clients/tardis-machine?

Asyncio TimeoutError

I get an asyncio.exceptions.TimeoutError if downloading data takes too long. Is there any way to prevent this or increase the timeout? Generally good for a few minutes until I get the error

Stack trace:

Just running the basic script from the documentation site

python==3.8.8
tardis-client==1.2.12

Support HTTP proxy

Can you please support specifying a HTTP proxy (or using the system-configured default proxy) with this library? Corporate networks often require all traffic to go via a proxy.

aiohttp does have this capability: https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support