jamesmawm / high-frequency-trading-model-with-ib

A high-frequency trading model using Interactive Brokers API with pairs and mean-reversion in Python

Purpose

A basic trading model on Interactive Brokers' API dealing with high-frequency data studies.

Requirements

• Python 3.7
• IB Trader Workstation Build 973.2
• IB paper or live trading account
• (Optional) Docker and docker-compose

What's new

19 Jun 2019

• Version 3.0 released
• ibpy library is dropped in favour of the newer ib_insync library.
• The same code logic is ported over to use the features of ib_insync, compatible with Python 3.7. Includes various code cleanup.
• Dropped matplotlib charting in favour of headless running inside Docker.

14 Jun 2019

• Version 2.0 released

• Merged pull request from: https://github.com/chicago-joe/IB_PairsTrading_Algo

  Thanks to chicago-joe for updating to work with Python 3.

  As this is only a compatibility update, there are many outdated components and the trading model is quite unlikely to be working as intended.

8 Jun 2015

• Version 1.0 released
• Refactor and conform to PEP8 standards
• New chart display with 4 subplots

Setting up

You can choose to run this model in your console OR in Docker.

Running on a local Python console

Steps to run the trading model on your command line:

• Within a Python 3.7 environment, install the requirements:

    pip install -r requirements.txt
  

• In IB Trader Workstation (TWS), go to Configuration > Api > Settings and:

  • enable ActiveX and Socket Clients
  • check the port number you will be using
  • If using Docker, uncheck Allow connections from localhost only and enter the machine IP running this model to Trusted IPs.

• Update main.py with the required parameters and run the model with the command:

    python main.py
  

Running from a Docker container

This step is optional. You can choose to deploy one or several instances of these algos on a remote machine for execution using Docker.

A Docker container helps to automatically build your running environment and isolate changes, all in just a few simple commands!

To run this trading model in headless mode:

• In TWS, ensure that remote API connections are accepted and the Docker machine's IP is added to Trusted IPs.

• Ensure your machine has docker and docker-compose installed. Build the image with this command:

    docker-compose build
  

• Update the parameters in docker-compose.yml. I've set the TWS_HOST value in my environment variables. This is the IP address of the remote machine running TWS. Or, you can just manually enter the IP address value directly. Then, run the image as a container instance:

    docker-compose up
  

  To run in headless mode, simply add the detached command -d, like this:

    docker-compose up -d
  

  In headless mode, you would have to start and stop the containers manually.

Key concepts

At the present moment, this model utilizes statistical arbitrage incorporating these methodologies:

• Bootstrapping the model with historical data to derive usable strategy parameters
• Resampling inhomogeneous time series to homogeneous time series
• Selection of highly-correlated tradable pair
• The ability to short one instrument and long the other.
• Using volatility ratio to detect up or down trend.
• Fair valuation of security using beta, or the mean over some past interval.
• One pandas DataFrame to store historical prices

Other functions:

• Generate trade signals and place buy/sell market orders based on every incoming tick data.
• Re-evaluating beta every some interval in seconds.

And greatly inspired by these papers:

• MIT - Developing high-frequency equities trading model @ http://dspace.mit.edu/handle/1721.1/59122
• SMU - Profiting from mean-reverting yield-curve trading strategies @ http://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=3488&context=lkcsb_research

And book:

• Introduction to High-Frequency Finance @ http://www.amazon.com/Introduction-High-Frequency-Finance-Ramazan-Gen%C3%A7ay/dp/0122796713

Step-by-step guide to more trading models

I published a book titled 'Mastering Python for Finance - Second Edition', discussing additional algorithmic trading ideas, statistical analysis, machine learning and deep learning, which you might find it useful. It is available on major sales channels including Amazon, Safari Online and Barnes & Noble, in paperback, Kindle and ebook. Get it from:

• https://www.amazon.com/dp/1789346460

Source codes and table of contents on GitHub:

• https://github.com/jamesmawm/mastering-python-for-finance-second-edition

Topics covered with source codes:

• Applying kernel PCA. Forecasting and predicting a time series.
• Replicating the VIX index
• Building a mean-reverting and trend-following trading model
• Implementing a backtesting system
• Predicting returns with a cross-asset momentum machine learning model
• Credit card payment default prediction with Keras. Get started in deep learning with TensorFlow for predicting prices.

If you would like a FREE review copy, drop me an email at [email protected].

Suggested enhancements

Some ideas that you can extend this model for better results:

• Extending to more than 2 securities and trade on optimum prices
• Generate trade signals based on correlation and co-integration
• Using PCA for next-period evaluation. In my book I've described the use of PCA to reconstruct the DOW index. Source codes here.
• Include vector auto-regressions
• Account for regime shifts (trending or mean-reverting states)
• Account for structural breaks
• Using EMA kernels instead of a rectangular one
• Add in alphas(P/E, B/P ratios) and Kalman filter prediction

Disclaimer

• Any securities listed is not a solicitation to trade.
• This model has not been proven to be profitable in a live account.
• I am not liable for any outcome of your trades.

Is this HFT?

Sure, I had some questions "how is this high-frequency" or "not for UHFT" or "this is not front-running". Let's take a closer look at these definitions:

• High-frequency finance: the studying of incoming tick data arriving at high frequencies, say hundreds of ticks per second. High frequency finance aims to derive stylized facts from high frequency signals.
• High-frequency trading: the turnover of positions at high frequencies; positions are typically held at most in seconds, which amounts to hundreds of trades per second.

This models aims to incorporate the above two functions and present a simplistic view to traders who wish to automate their trades, get started in Python trading or use a free trading platform.

Other software of interest

I write software in my free time. One of them for trading futures was simply called 'The Gateway'. It is a C# application that exposes a socket and public API method calls for interfacing Python with futures markets including CME, CBOT, NYSE, Eurex and ICE. Targets the T4 API.

More information on GitHub: https://github.com/hftstrat/The-Gateway-code-samples or view on the website.

Final notes

• I haven't come across any complete high-frequency trading model lying around, so here's one to get started off the ground and running.
• This model has never been used with a real account. All testing was done in demo account only.
• The included strategy parameters are theoretical ideal conditions, which have not been adjusted for back-tested results.
• This project is still a work in progress. A good model could take months or even years!

high-frequency-trading-model-with-ib's Issues

Need help configuring forked repo for future pull requests

Hi @jamesmawm ,

I'm trying to properly set up my forked repo so that I can pull your commits and branch trees into my fork on a regular basis. How should I do this?

I think I need to set your repo as a remote branch, but I'm not sure how to configure tracking, set-upstream, and other settings necessary to do this correctly.

Also, if I merge one of your branches with mine, will the commit log retain both of our histories?

Thanks for the help!

IndexError in is_overbought_or_oversold()

I get an index error while trying to run the main.py related to the last_prices_a & last_prices_b:

High-Frequency-Trading-Model-with-IB/test/models/hft_model_1.py", line 208, in is_overbought_or_oversold last_price_b = self.df_hist[symbol_b].dropna().values[-1] IndexError: index -1 is out of bounds for axis 0 with size 0

related to line 208:

def is_overbought_or_oversold(self):
		[symbol_a, symbol_b] = self.symbols

		last_price_a = self.df_hist[symbol_a].dropna().values[-1]
		last_price_b = self.df_hist[symbol_b].dropna().values[-1]

		expected_last_price_a = last_price_b * self.beta

		is_overbought = last_price_a < expected_last_price_a  # Cheaper than expected
		is_oversold = last_price_a > expected_last_price_a  # Higher than expected

Any thoughts or help would be appreciated. Thanks.

Tried to run Main.py & ended with error

Please help on this.
Tried to run Main.py & ended with error

from models.hft_model import HFTModel

File "C:\Users\Raji\Downloads\High-Frequency-Trading-Model-with-IB-master\models\hft_model.py", line 378
except Exception, e:
^
SyntaxError: invalid syntax

after runing for about 5 mins it starts getting NANs on beta and volatility values

i was running a test and modified the following line in hft_model_1

self.moving_window_period = dt.timedelta(hours=1)

self.moving_window_period = dt.timedelta(minutes=5)

here is part of the log:

[2021-04-29 02:42:45.632009][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=-1.59|beta:0.01 volatility:0.95
[2021-04-29 02:42:47.011282][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
[2021-04-29 02:42:48.036130][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
[2021-04-29 02:42:50.217625][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
[2021-04-29 02:42:52.519213][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=-1.59|beta:0.01 volatility:0.95
[2021-04-29 02:42:54.824365][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
[2021-04-29 02:42:56.327518][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
[2021-04-29 02:42:57.553074][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=-1.59|beta:0.01 volatility:0.95
[2021-04-29 02:43:00.026884][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=0.26|beta:0.01 volatility:0.95
.....
.....
.....
......
.....
....
[2021-04-29 02:46:44.470763][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=17.76|beta:nan volatility:nan
[2021-04-29 02:46:46.292348][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=16.76|beta:nan volatility:nan
[2021-04-29 02:46:47.674391][account]EURUSD pos=-100000.0 avgPrice=1.2139057214|USDJPY pos=100000.0|rpnl=0.00 upnl=15.76|beta:nan volatility:nan
[2021-04-29 02:46:49.276006][account]EURUSD pos=-100000.0 avgPrice=1.2139057|USDJPY pos=100000.0|rpnl=0.00 upnl=16.75|beta:nan volatility:nan
[2021-04-29 02:46:53.797603][account]EURUSD pos=-100000.0 avgPrice=1.2139057|USDJPY pos=100000.0|rpnl=0.00 upnl=18.68|beta:nan volatility:nan
[2021-04-29 02:46:57.196735][account]EURUSD pos=-100000.0 avgPrice=1.2139057|USDJPY pos=100000.0|rpnl=0.00 upnl=17.67|beta:nan volatility:nan
[2021-04-29 02:46:58.990993][account]EURUSD pos=-100000.0 avgPrice=1.2139057|USDJPY pos=100000.0|rpnl=0.00 upnl=17.67|beta:nan volatility:nan

Issue with Matplotlib/PyQt5 Chart Updates

I think there was an update to Matplotlib because the charting function stopped working completely out of the blue this week.
The error was "Exception: Show() got an unexpected keyword argument 'block'.

I spent all morning debugging and was finally able to produce the chart again, but there are still some issues with the chart scaling.

I'm trying to change the matplotlib to a different backend, but I could use a hand if anyone has experience with this?

Error when running

Hi, when I try running the main.py after including all the libraries this happens:

RESTART: C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\main.py 
Connecting on host: 127.0.0.1 port: 4002
[2020-02-26 19:19:43.100615]started
Error 162, reqId 4: Historical Market Data Service error message:No market data permissions for AMEX STK, contract: Stock(symbol='SPY', exchange='SMART', currency='USD')
Error 162, reqId 5: Historical Market Data Service error message:No market data permissions for ISLAND STK, contract: Stock(symbol='QQQ', exchange='SMART', currency='USD')
Error 354, reqId 6: Requested market data is not subscribed.Delayed market data is available.Error&BEST/STK/Top&BEST/STK/Top, contract: Stock(symbol='SPY', exchange='SMART', currency='USD')
Error 354, reqId 7: Requested market data is not subscribed.Delayed market data is available.Error&BEST/STK/Top&BEST/STK/Top, contract: Stock(symbol='QQQ', exchange='SMART', currency='USD')
Traceback (most recent call last):
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\main.py", line 29, in <module>
    model.run(to_trade=to_trade, trade_qty=100)
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\models\hft_model_1.py", line 67, in run
    self.recalculate_strategy_params()
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\models\hft_model_1.py", line 182, in recalculate_strategy_params
    resampled = self.df_hist.resample('30s').ffill().dropna()
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\generic.py", line 8155, in resample
    base=base, key=on, level=level)
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\resample.py", line 1250, in resample
    return tg._get_resampler(obj, kind=kind)
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\resample.py", line 1380, in _get_resampler
    "but got an instance of %r" % type(ax).__name__)
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'

I would really appreciate help, thank you.

how to add an account

I'm new to IB and cannot find where can I put my login information

AttributeError: 'NoneType' object has no attribute 'realizedPnL' immediately after " python main.py" is executed

Traceback (most recent call last):
File "/home/miniconda3/lib/python3.8/site-packages/eventkit/event.py", line 184, in emit
func(obj, *args)
File "/developer/Downloads/High-Frequency-Trading-Model-with-IB/models/hft_model_1.py", line 78, in on_tick
self.perform_trade_logic()
File "/developer/Downloads/High-Frequency-Trading-Model-with-IB/models/hft_model_1.py", line 101, in perform_trade_logic
self.print_strategy_params()
File "/developer/Downloads/High-Frequency-Trading-Model-with-IB/models/hft_model_1.py", line 127, in print_strategy_params
rpnl=self.pnl.realizedPnL,
AttributeError: 'NoneType' object has no attribute 'realizedPnL'

Note sure what is missing in ib_sync or interactive Brokers response.

Volatility becomes NAN after running for a day or several hours

The Volatility value starts printing NANs after several hours or a day. I am not sure if this is due to anything . i am running just the code as is . Nothing changed for testing . I did stop it and then restarted it again and its ok for now . have you experienced this?

Futures - Modify to accept Specific Future Symbol like MESU0 or MESU20

Can this be done? I hope so this is awesome work.

Thanks Steve H.
Stillwater, OK

Running multiple pairs using the same single model

Under main.py is there a way to run multiple pairs at once using the same hftmodel1

below will only run ( EURUSD , USDJPY ). if for example let's say i want to run 3 pairs

(EURUSD, USDJPY ) , (AUDUSD, USDCAD) , (GBPUSD , USDCHF )

how would i implement that in the main .py

model = HftModel1(
		host=TWS_HOST,
		port=TWS_PORT,
		client_id=2,
	)

	to_trade = [
		('EURUSD', Forex('EURUSD')),
		('USDJPY', Forex('USDJPY'))
	]

	model.run(to_trade=to_trade, trade_qty=100000)

How to get pre-market-price from tws by using IB.

I would like to get the pre-market-price -> the price between 8 -9.30.
By the way, I don't know how to get the price.
Would you please inform me about it?
I will wait for your favorable reply.
Thanks.

index -1 is out of bounds for axis 0 with size 0

I am getting this error:

Exception has occurred: IndexError
index -1 is out of bounds for axis 0 with size 0

in this function:

def is_overbought_or_oversold(self):
	[symbol_a, symbol_b] = self.symbols
	last_price_a = self.df_hist[symbol_a].dropna().values[-1]

Does anybody know of a way to solve it?

Thanks

getting error: RuntimeError: main thread is not in main loop

29-Jan-17 18:39:08 ERROR Exception in message dispatch. Handler '__on_tick_event' for 'tickPrice'
Traceback (most recent call last):
File "/hft/High-Frequency-Trading-Model-with-IB/envz/lib/python2.7/site-packages/ib/opt/dispatcher.py", line 44, in call
results.append(listener(message))
File "hft/High-Frequency-Trading-Model-with-IB/models/hft_model.py", line 323, in __on_tick_event
self.__update_charts()
File "/hft/High-Frequency-Trading-Model-with-IB/models/hft_model.py", line 334, in __update_charts
self.strategy_params.indicators)
File "/hft/High-Frequency-Trading-Model-with-IB/classes/chart.py", line 28, in display_chart
plt.pause(0.01)
File "/hft/High-Frequency-Trading-Model-with-IB/envz/lib/python2.7/site-packages/matplotlib/pyplot.py", line 298, in pause
canvas.draw()
File "/hft/High-Frequency-Trading-Model-with-IB/envz/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py", line 352, in draw
tkagg.blit(self._tkphoto, self.renderer._renderer, colormode=2)
File "/hft/High-Frequency-Trading-Model-with-IB/envz/lib/python2.7/site-packages/matplotlib/backends/tkagg.py", line 22, in blit
id(data), colormode, id(bbox_array))
RuntimeError: main thread is not in main loop

Please update Model to use native IB Python package instead of ibsync

Please update Model to use native IB Python package instead of ibsync. Reduce complexity by removing unnecessary packages

Plot not responding and no new orders

First of all thank you for this amazing example of Python + IB + HFT!

May I ask what version of python and the other packages you are using? I seem to be getting a deprecation warning in matplotlib.

When I start the program the first two orders come through, but when the plot pops up it seems to stop responding(deprecation warning below) and despite the console still going and showing BUY or SELL signals, nothing else happens in IB.

Warning (from warnings module):
File "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line 2407
warnings.warn(str, mplDeprecation)
MatplotlibDeprecationWarning: Using default event loop until function specific to this GUI is implemented

Is this program supposed to be profitable?

I am running the program with stocks on paper trade. The code is losing money. Does the order of the 2 stocks I use matter? And how much time does it usually takes until I see even a plus in the profit?

Thank you

Error when running

Hi, when I try running the main.py after including all the libraries this happens:

RESTART: C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\main.py 
Connecting on host: 127.0.0.1 port: 4002
[2020-02-26 19:19:43.100615]started
Error 162, reqId 4: Historical Market Data Service error message:No market data permissions for AMEX STK, contract: Stock(symbol='SPY', exchange='SMART', currency='USD')
Error 162, reqId 5: Historical Market Data Service error message:No market data permissions for ISLAND STK, contract: Stock(symbol='QQQ', exchange='SMART', currency='USD')
Error 354, reqId 6: Requested market data is not subscribed.Delayed market data is available.Error&BEST/STK/Top&BEST/STK/Top, contract: Stock(symbol='SPY', exchange='SMART', currency='USD')
Error 354, reqId 7: Requested market data is not subscribed.Delayed market data is available.Error&BEST/STK/Top&BEST/STK/Top, contract: Stock(symbol='QQQ', exchange='SMART', currency='USD')
Traceback (most recent call last):
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\main.py", line 29, in <module>
    model.run(to_trade=to_trade, trade_qty=100)
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\models\hft_model_1.py", line 67, in run
    self.recalculate_strategy_params()
  File "C:\Users\Devx86\Desktop\InteractiveBrokers-PairsTrading-Algo-master\models\hft_model_1.py", line 182, in recalculate_strategy_params
    resampled = self.df_hist.resample('30s').ffill().dropna()
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\generic.py", line 8155, in resample
    base=base, key=on, level=level)
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\resample.py", line 1250, in resample
    return tg._get_resampler(obj, kind=kind)
  File "C:\Users\Devx86\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\resample.py", line 1380, in _get_resampler
    "but got an instance of %r" % type(ax).__name__)
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'

I would really appreciate help, thank you.

WARNING: Please only use this Algorithm as a test strategy not as a Live strategy .

I tested this simple HFT strategy on a small account with IB both on Live Account and Paper Account not Demo Account.

Pairs tested: EURUSD & USDJPY on 24 hours period

1. On Live Account : 98% Losses, 2% wins on all trades ( 62 trades)
2. On Paper account : 96% Wins , 4% Losses on all trades ( 60 trades)
3. No DEMO account testing ( Demo account contains replayed back market data so its not useful for any type of testing )

Both Live and Paper account from IB use the same realtime Market Datastream and only differ in execution . Live account executes a Live trade while paper account simulates the execution but does not open a live trade. So what's the difference between this two accounts at IB . I could not find any difference on configuration of both accounts.

May be you can share some insights ?