pmorissette / bt

bt - flexible backtesting for Python

bt - Flexible Backtesting for Python

bt is currently in alpha stage - if you find a bug, please submit an issue.

Read the docs here: http://pmorissette.github.io/bt.

What is bt?

bt is a flexible backtesting framework for Python used to test quantitative trading strategies. Backtesting is the process of testing a strategy over a given data set. This framework allows you to easily create strategies that mix and match different Algos. It aims to foster the creation of easily testable, re-usable and flexible blocks of strategy logic to facilitate the rapid development of complex trading strategies.

The goal: to save quants from re-inventing the wheel and let them focus on the important part of the job - strategy development.

bt is coded in Python and joins a vibrant and rich ecosystem for data analysis. Numerous libraries exist for machine learning, signal processing and statistics and can be leveraged to avoid re-inventing the wheel - something that happens all too often when using other languages that don't have the same wealth of high-quality, open-source projects.

bt is built atop ffn - a financial function library for Python. Check it out!

Features

• Tree Structure The tree structure facilitates the construction and composition of complex algorithmic trading strategies that are modular and re-usable. Furthermore, each tree Node has its own price index that can be used by Algos to determine a Node's allocation.

• Algorithm Stacks Algos and AlgoStacks are another core feature that facilitate the creation of modular and re-usable strategy logic. Due to their modularity, these logic blocks are also easier to test - an important step in building robust financial solutions.

• Charting and Reporting bt also provides many useful charting functions that help visualize backtest results. We also plan to add more charts, tables and report formats in the future, such as automatically generated PDF reports.

• Detailed Statistics Furthermore, bt calculates a bunch of stats relating to a backtest and offers a quick way to compare these various statistics across many different backtests via Results display methods.

Roadmap

Future development efforts will focus on:

• Speed Due to the flexible nature of bt, a trade-off had to be made between usability and performance. Usability will always be the priority, but we do wish to enhance the performance as much as possible.

• Algos We will also be developing more algorithms as time goes on. We also encourage anyone to contribute their own algos as well.

• Charting and Reporting This is another area we wish to constantly improve on as reporting is an important aspect of the job. Charting and reporting also facilitate finding bugs in strategy logic.

Installing bt

The easiest way to install bt is from the Python Package Index using pip:

pip install bt

Since bt has many dependencies, we strongly recommend installing the Anaconda Scientific Python Distribution, especially on Windows. This distribution comes with many of the required packages pre-installed, including pip. Once Anaconda is installed, the above command should complete the installation.

Recommended Setup

We believe the best environment to develop with bt is the IPython Notebook. From their homepage, the IPython Notebook is:

"[...] a web-based interactive computational environment
where you can combine code execution, text, mathematics, plots and rich
media into a single document [...]"

This environment allows you to plot your charts in-line and also allows you to easily add surrounding text with Markdown. You can easily create Notebooks that you can share with colleagues and you can also save them as PDFs. If you are not yet convinced, head over to their website.

bt's Issues

The example `SMA Crossover Strategy` failed

The example SMA Crossover Strategy in the document failed, and the error log is below:

AttributeError: 'Series' object has no attribute 'items'

I Think this is because the WeighTarget does not assign an dict to target.temp['weights'], so I just repacle target.temp['weights'] = w.dropna() with {w.index[i]:w.values[i] for i in range(len(w.index))} maybe there will be a better way.

plot not showing

Hi Philippe,

I'm new to bt but enjoyed playing with it, thanks to the well thought architecture of the project.
After a few adjustments for libraries, I managed to make bt run properly.
The last issue I face is not being able to show plots. It seems to me that this may be a library issue again.
I run Python version 2.7.9 (default, Apr 2 2015, 15:33:21) [GCC 4.9.2] + Pandas version 0.17.0.

matplotlib looks fine as I can show plots easily directly with basic scripts using matplotlib.
Pandas 0.17.0 recently released on Oct 9. Do you think this can be the issue? (looks like you generate the standard plots from Pandas, pls correct me if I'm wrong).

Thanks a lot for your help.
I'm looking forward to building on bt !!!

ImportError: No module named 'core'

Hi, I'm a newbie to python, but I'm really interested in using your bt project for backtesting. I tried to follow the instructions --> install Anaconda for Mac (I used the 64bit version, Python 3.4). Everything installs fine and I can use Anaconda's python dist via terminal.

Then, I try to import bt but I get this error:

import bt
Traceback (most recent call last):
File "", line 1, in
File "//anaconda/lib/python3.4/site-packages/bt/init.py", line 1, in
import core
ImportError: No module named 'core'

What is the core module? I can't find it.

Thanks!

AttributeError: 'numpy.int64' object has no attribute 'month'

Hello Philippe,

i just discover your backtest framework and I would like to say bravo for all the work you've done.
However, when I try to launch it at home, I got this Stack Error:

Traceback (most recent call last):
File "C:/Program Files (x86)/Projects/ForexBTEngine/BTmorissetteTest.py", line 13, in
res = bt.run(test)
File "C:\Program Files (x86)\Miniconda3\lib\site-packages\bt\backtest.py", line 26, in run
bkt.run()
File "C:\Program Files (x86)\Miniconda3\lib\site-packages\bt\backtest.py", line 165, in run
self.strategy.run()
File "bt/core.py", line 1161, in bt.core.Strategy.run (bt/core.c:15301)
File "bt/core.py", line 1106, in bt.core.AlgoStack.call (bt/core.c:14703)
else:
File "C:\Program Files (x86)\Miniconda3\lib\site-packages\bt\algos.py", line 218, in call
if now.month != self.last_date.month:
AttributeError: 'numpy.int64' object has no attribute 'month'

I'm on python 3.4 but I thought that I made all the necessary changes to make it work (in ffn and in bt)... Did I miss something? Any idea?

Thanks for everything philippe.

Mathieu

algos selectthese and weightspecified for various specified dates?

Hi Phil, I'm not very familiar with python, but bt seems simple enough for my purposes. I want to specify Date,ticker,weight for different dates. I'll also provide the prices for those tickers....it should simply use the specified tickers and weights for the different dates and just do the back-testing (in other words, i only need the back-test engine, don't need its selection and weighting routines, for starts). I searched the documentation, but couldn't find something that would meet my purposes. can you point me to the right modules?

Thank you very much.

backtest.py duplicated columns

Bonjour Philippe,
Thanks for the good work on this package!

When I run your quick example from the webpage I get the following error:
in ..backtest.py line 123 'if data.columns.duplicated().any():'
AttributeError: 'Index' object has no attribute 'duplicated'

I am using python 2.7.6. I haven't installed anaconda as I already had a working setup going. I did run a pandas update, I have version 0.13.1

I was able to get around this by creating a local copy of backtest.py and commenting out that section and the backtest ran OK after that.

Do you have an idea what is causing this issue?
regards
Martin

'Index' object has no attribute 'duplicated'

I have recently installed bt. I'm super excited about its potential -- however I've run into a problem when I try to call bt.backtest(strategy, data). I'm following along the examples page, and when I call t = bt.backtest(s, data) I get an error "'Index' object has no attribute 'duplicated'"

I've tried to remove bt and install it again, no avail. I'm working on a vagrant box, so I destroyed the box and recreated the environment, still nothing. I'm fairly new to python still, otherwise I would try to fix this myself.

The full error:

AttributeError Traceback (most recent call last)
in ()
5
6 # now we create the Backtest
----> 7 t = bt.Backtest(s, data)
8
9 res = bt.run(t)

/home/vagrant/anaconda/lib/python2.7/site-packages/bt/backtest.pyc in init(self, strategy, data, name, initial_capital, commissions, integer_positions)
121 integer_positions=True):
122
--> 123 if data.columns.duplicated().any():
124 cols = data.columns[data.columns.duplicated().tolist()].tolist()
125 raise Exception(

AttributeError: 'Index' object has no attribute 'duplicated'

Feature Request

Hi,

I have been using bt for the past year. It's a really useful backtesting tool. I have a feature update request for the security weight plot. I backtest using multiple securities. It's difficult for me to see individual trades using the plot_security_weights method. Could you either update or grate a new method that displays individual security trades each within it's own separate plot. That would be really useful.

Thanks,

Moses

Tree structure and _paper_trade

Hi Phil,

could you please explain how does (and should) tree structure actually work? Do I understand correctly that the actual money is allocated (fees are paid and _capital is changed) only at the Rebalance() of the parent? If so - why do we change weights of securities and prices of the parent strategy?

For example here's an example of quite unexpected behavior. Below is the same strategy which is backtested stand-alone and as a part of another strategy:

data = bt.get('spy,eem', start='2014-01-01')

com = lambda q, p: 10 * abs(q)

mom_s = bt.Strategy('mom_s', [bt.algos.RunMonthly(),
                              bt.algos.SelectAll(),
                              bt.algos.SelectMomentum(1),
                              bt.algos.WeighEqually(),
                              bt.algos.Rebalance()],
                    ['spy', 'eem'])

from copy import deepcopy
mom_s1 = deepcopy(mom_s)

master = bt.Strategy('master', [bt.algos.RunMonthly(),
                                bt.algos.SelectAll(),
                                bt.algos.WeighEqually(),
                                bt.algos.Rebalance()],
                     [mom_s])

t0 = bt.Backtest(mom_s1, data, commissions=com)
r0 = bt.run(t0)

t = bt.Backtest(master, data, commissions=com)
r = bt.run(t)

Note I chose quite a substantial commission to make an effect vivid. Also I have here deepcopy explicitly, so you can avoid #19 and test it independently from merging #20.

The performance of master and mom_s differ quite dramatically:

ax = r.prices.plot()
r0.prices.plot(color='r', ax=ax)

And the reason is commissions that are paid in mom_s and ignored in master.

So far I don't understand the whole "paper trading" in the bt.core, so I don't see how to fix this issue easily. Could you please advise?

General questions

I have a few general questions regarding bt that I would like to ask. I hope it is acceptable to ask questions like this here.

1. For what type of investment style was this project created? It seems to me that bt is focussed on longer-term portfolios with concentration on rebalancing. Would bt also work well for short-term investments? I am not referring to HFT, but daily trading or perhaps hourly trading.

2. Is it at all possible to use bt for live trading through extending the code? I assume it is not since it seems that bt is not event-driven. Could you provide some clarity?

3. The roadmap does not provide any specific features that could be implemented, but is a little vague. What are the current shortcomings of this library? What would be valuable features to add other than providing more algos?

I have scanned through the code and would just like to complement you on your neat and thorough coding style. Your use of comments is better than most of the code I see these days...

New features : Intraday

Hi,

Just wondering if we can 'twist' the existing class to put intraday data ?
Think this should be the same, just more timestamp.

Thanks
PS: do you have an email where we discuss on this ?

Specifics of implementation

Hello Philippe,

first of all, thank you very much for releasing your library and congratulations on it - I like your approach to backtesting very much.

Let me ask you here a couple of questions about specifics of implementation:

1. Perhaps I'm not understanding it correctly, but it looks like you have one day look-ahead bias for momentum strategies: bt.algos.SelectMomentum() is based on bt.algos.StatTotalReturn(), which calculates the return from target.now - self.lookback to target.now. Then the rebalancing is performed by the same price of target.now. Typically one would work on daily closing prices, and it's impossible to enter the market on the same day - one should use next day's close instead. Am I right?
2. Another issue related to look-ahead bias. Suppose we're closing position on the day where we don't have the data any more (e.g. the ticker is already dead / delisted / company went bankrupt). We still need to be able to do that, assuming that we've lost everything (current price is zero). I suggest that this part of bt.core.SecurityBase.allocate() that raises an exception could be removed and NaN should be better treated as zero. Does it make sense?
3. What was the reason for introducing price-independent commissions (i.e. that only depend on quantity)? Usually one adds a slippage as a fraction of target price to the total transaction costs.
   If I was to introduce that kind of "implementation fees" in bps, would it be enough to change bt.core.SecurityBase.outlay() or are there any other places to modify?
4. I'm also interested in recording each of transactions (which are so far implicit in the engine). Would you rather suggest to modify the backtesting engine itself and build-in the logging system directly to the core, or is it better to implement an "Algo" that is to be called after before bt.algos.Rebalance() and would record all implied transactions?

Best regards
Vladimir

Exception: Cannot allocate capital to XNAS/BLUD because price is nan as of 2011-08-19 00:00:00

hey guys-

I'm getting an error due to stocks that became de-listed at some time during the simulation. bt is throwing an error when it tries to run Strategy.close(child) at the line 716: c.allocate(-c.value)

Here's some info:

• my data was pulled from the Exchange Data International via the Quandl API directly into pandas using pd.read_csv()
• the relevant area of the data:

            XNAS/KMB  XNAS/SHOP  XNAS/BLUD  XNAS/WFT_1_UADJ
Date                                                        `
2011-08-12  61.817510        NaN      26.79           17.295
2011-08-15  62.719193        NaN      26.83           17.685
2011-08-16  62.728684        NaN      26.93           17.200
2011-08-17  62.605296        NaN      26.94           17.380
2011-08-18  62.254114        NaN      26.81           16.010
2011-08-19  62.064286        NaN        NaN           15.010
2011-08-22  62.415468        NaN        NaN           14.690
2011-08-23  63.820196        NaN        NaN           15.510
2011-08-24  64.484594        NaN        NaN           15.600

I think the solution (if this is even a real bug) might be that Strategy.close(child) (bt.core.py line 704) needs to test whether c.price is not nan like:

if isnan(c.value):
    c.allocate(-c._last_value)
else:
    c.allocate(-c.value)

I'd love to know if there is an Algo or something for this I skipped over, or if this is an actual bug! Thanks!

Exception: Cannot allocate capital to close-002142 because price is 0 as of 2008-12-03 00:00:00

[issue]
When executing the code "bt.run(strategy2)" , It raise the "Exception: Cannot allocate capital to close-002142 because price is 0 as of 2008-12-03 00:00:00" , but the price of 'close-002142' is not zero as of 2008-12-03. Then I debug the code, but it is blocked when it call "target.rebalance(item[1], child=item[0], base=base)" (shown as picture).
[question]
Could you tell me which function`s the 'target.rebalance' calling ?

[The Coding]
class Rebalance(Algo):

"""
Rebalances capital based on temp['weights']

Rebalances capital based on temp['weights']. Also closes
positions if open but not in target_weights. This is typically
the last Algo called once the target weights have been set.

I got the error :

Exception: Cannot allocate capital to close-002142 because price is 0 as of 2008-12-03 00:00:00

Requires:
    * weights

"""

def __init__(self):
    super(Rebalance, self).__init__()

def __call__(self, target):
    if 'weights' not in target.temp:
        return True

    targets = target.temp['weights']

    # de-allocate children that are not in targets
    not_in = [x for x in target.children if x not in targets]
    for c in not_in:
        target.close(c)

    # save value because it will change after each call to allocate
    # use it as base in rebalance calls
    base = target.value
    for item in iteritems(targets):
         ~~target.rebalance(item[1], child=item[0], base=base)~~
    return True

[picture]

position error

Hi Philippe,

Thank you so much for sharing your excellent work. I quite like this library and did some experiments on that. I find some weird issues concerning position property.

Here is the code:

import bt
data = bt.get('spy, ashr, agg, eem, ewz', start='2014-01-01', end='2014-02-02')

class PrintAlgo(bt.Algo):
def init(self):
super(PrintAlgo, self).init()

def __call__(self, target):
    print '\n' + '*' * 20 + str(target.now) + '*' * 20
    print target.positions
    print pd.DataFrame({x.name: x.positions for x in target.members
                             if isinstance(x, bt.core.SecurityBase)})
    return True

s = bt.Strategy('s1', [
bt.algos.RunWeekly(),
bt.algos.SelectAll(),
bt.algos.WeighEqually(),
bt.algos.Rebalance(),
bt.algos.run_always(PrintAlgo()),
])

test = bt.Backtest(s, data, initial_capital=80000)
res = bt.run(test)

Results:
print target.positions will always give empty dataframe (if print target._positions, it is None), however, if you extract vals (which is the second print) from positions function, it does give the position dataframe.
Any idea about this? It should give positions for all the children like second print when you call the property.

Result print (part)

_2014-01-02 00:00:00_
Empty DataFrame
Columns: []
Index: []
Empty DataFrame
Columns: []
Index: []

_2014-01-03 00:00:00_
Empty DataFrame
Columns: []
Index: []
Empty DataFrame
Columns: []
Index: []

_2014-01-06 00:00:00_
Empty DataFrame
Columns: []
Index: []
agg ashr eem ewz spy
Date
2014-01-02 0 0 0 0 0
2014-01-03 0 0 0 0 0
2014-01-06 155 683 414 392 90

how to use commission function?

here is my code:
data=pd.DataFrame(zz500['close']).truncate(before='2011/1/1')
s=bt.Strategy('Strategy',[bt.algos.SelectWhere(data.pct_change()>0),bt.algos.WeighEqually(),bt.algos.Rebalance()])
bench=bt.Strategy('000001',[bt.algos.RunOnce(),bt.algos.SelectAll(),bt.algos.WeighEqually(),bt.algos.Rebalance()])
test = bt.Backtest(s, data,commissions=lambda p,q: 0.05_p_q)
b = bt.Backtest(bench, data,commissions=lambda p,q: 0.05_p_q)
res = bt.run(test,b)

the final Nav become larger when I add commissions to my code? how can this happen?

Thank you for any help.

More performance improvements

OK, suppose that #24 is closed and #25 is merged.

Now we can see some other performance bottlenecks. Checking the simple strategy:

x = np.random.randn(10000, 100) * 0.01
idx = pd.date_range('1975-01-01', freq='B', periods=x.shape[0])
df = np.exp(pd.DataFrame(x, index=idx).cumsum())

s0 = bt.Strategy('s0', [bt.algos.RunMonthly(),
                        bt.algos.SelectAll(),
                        bt.algos.SelectRandomly(len(df.columns)/2),
                        bt.algos.WeighRandomly(),
                        bt.algos.Rebalance()])

Here's the call-graph for it (before #25 the importance of other nodes was not seen due to an overhead in .update()):

What do we see:

1. StrategyBase.update() calls SecurityBase.value, which in turn sometimes calls StrategyBase.update().
2. Strategy.close() calls SecurityBase.value, which sometimes calls StrategyBase.update().
3. SecurityBase.weight calls StrategyBase.update()
4. and some other calls of either of .update() from other members.

Up to my tests:

• calls of SecurityBase.value in StrategyBase.update() are redundant - the parameter _value was never changed here. So I think that the call could be changed to a plain access of protected field and save us ~10% more.
• In Strategy.close() - for the simple strategy again, call of SecurityBase.value does not provide any value in comparison with protected _value.

I'm still looking into call dependencies and trying to understand how can we optimize it. But, Phil, I'd appreciate your feedback on two bullets above and your thoughts on this issue. It seems that .stale can be ignored for some cases (like above) and the call to a property could be replaced by a direct access to a protected field - which could save us quite some time.

Design question for RFQ/non-continuous markets

Congrats on you work.

Just pondering how this would be used in R.F.Q. style markets where we only have sporadic buying and selling opportunities and where we compete with other dealers to win a customer order. Supposing we have a trade record of the winning response price/size, I think I would do something like the following:

• Write class MaybeBuy(Algo): to return True only when there is a buying opportunity. Similarly for sell opportunities. Write another "Algo" to return suggested bid/offer that fires when MaybeBuy is True. Write another Algo WinRFQ that returns True if our price is the best.

[Perhaps combine the above in one algo?]

• Write a Strategy that has member class commision_fn to mock the actual economics of buying/selling at the price where we won the auction. This would also "allocate" an amount equal to the trade size.

Can you think of anything I'm missing?

Rounding of positions

Regarding this part of the code that calculates how much do we need to buy/sell for rebalancing a position:

if amount > 0:
    q = math.floor(amount / (self._price * self.multiplier))
else:
    q = math.ceil(amount / (self._price * self.multiplier))

I suggest that we have for both of buys (amount>0) and sells (amound<0)

q = math.floor(amount / (self._price * self.multiplier))

Suppose we're holding a portfolio and some securities drop in price, so we need to sell other securities to keep the balance. I think it is better to sell more (math.floor for negative amount), so we have positive remainder in the _capital and all weights sum to a value less than 1, in contrast to selling less (math.ceil for negative amount) and been overdrafted (negative _capital) which would result in a sum of all weights greater than 1.

I understand, that in the present logic fees will overdraft us anyway, but I think that this way of rounding should be still more consistent. In particular if we're working with expensive futures or even equities (this is indeed an extreme, but consider Lindt & Sprüngli: ~60'000 USD per share).

Phil, what do you think?

AttributeError: 'SelectWhere' object has no attribute 'signal'

Having a problem I can't seem to figure out the solution to this error:

import bt
import pandas as pd
%pylab inline

data = bt.get('HYG', start = '2000-01-01')

sma = pd.rolling_mean(data, 200)
plot = bt.merge(data,sma).plot(figsize=(15,5))

signal = data > sma

s = bt.Strategy('above200sma', [SelectWhere(data>sma),bt.algos.WeighEqually(), bt.algos.Rebalance()])
t = bt.Backtest(s, data)
res = bt.run(t)
res.display()

Here is the error:

AttributeError Traceback (most recent call last)
in ()
3 s = bt.Strategy('above200sma', [SelectWhere(data>sma),bt.algos.WeighEqually(), bt.algos.Rebalance()])
4 t = bt.Backtest(s, data)
----> 5 res = bt.run(t)
6 res.display()

C:\Users\nnayyar\Anaconda\lib\site-packages\bt\backtest.pyc in run(*backtests)
23 """
24 # run each backtest
---> 25 for bkt in backtests:
26 bkt.run()
27

C:\Users\nnayyar\Anaconda\lib\site-packages\bt\backtest.pyc in run(self)
162 for dt in self.dates:
163 self.strategy.update(dt)
--> 164 if not self.strategy.bankrupt:
165 self.strategy.run()
166 # need update after to save weights, values and such

C:\Users\nnayyar\Anaconda\lib\site-packages\bt\core.pyd in bt.core.Strategy.run (bt/core.c:14847)()

C:\Users\nnayyar\Anaconda\lib\site-packages\bt\core.pyd in bt.core.AlgoStack.call (bt/core.c:14249)()

in call(self, target)
5
6 def call(self,target):
----> 7 if target.now in self.signal.index:
8 sig = self.signal.ix[target.now]
9

AttributeError: 'SelectWhere' object has no attribute 'signal'

Backtesting on intraday data

Hi,

I've been using bt for a while now and I love the framework. However, I have been struggling in using it for intraday data since you'd have two relevant price series when backtesting ( bid + ask and not only the close) . How can I fix this issue when running the backtest ?

Thanks.

plot error based on length of data

on 12/20 start dates before 11-26 work but on 11-26 you get this error

In [95]:

prices = ffn.get(bmr, start='2014-11-26')
In [96]:

returns = prices.to_returns().dropna().cumsum()
In [98]:

returns.plot()

AttributeError Traceback (most recent call last)
in ()
----> 1 returns.plot()

//anaconda/lib/python2.7/site-packages/pandas/tools/plotting.pyc in plot_frame(frame, x, y, subplots, sharex, sharey, use_index, figsize, grid, legend, rot, ax, style, title, xlim, ylim, logx, logy, xticks, yticks, kind, sort_columns, fontsize, secondary_y, *_kwds)
2163 secondary_y=secondary_y, *_kwds)
2164
-> 2165 plot_obj.generate()
2166 plot_obj.draw()
2167 if subplots:

//anaconda/lib/python2.7/site-packages/pandas/tools/plotting.pyc in generate(self)
900 self._compute_plot_data()
901 self._setup_subplots()
--> 902 self._make_plot()
903 self._add_table()
904 self._make_legend()

//anaconda/lib/python2.7/site-packages/pandas/tools/plotting.pyc in _make_plot(self)
1539 if self._is_ts_plot():
1540 data = self._maybe_convert_index(self.data)
-> 1541 self._make_ts_plot(data)
1542 else:
1543 x = self._get_xticks(convert_period=True)

//anaconda/lib/python2.7/site-packages/pandas/tools/plotting.pyc in _make_ts_plot(self, data, *_kwargs)
1632 y_values = self._get_stacked_values(y, label)
1633
-> 1634 newlines = plotf(y_values, ax, label, style, *_kwds)
1635 self._add_legend_handle(newlines[0], label, index=i)
1636

//anaconda/lib/python2.7/site-packages/pandas/tools/plotting.pyc in _plot(data, ax, label, style, *_kwds)
1609 else:
1610 lines = tsplot(data, plotf, ax=ax, label=label,
-> 1611 style=style, *_kwds)
1612 return lines
1613 return _plot

//anaconda/lib/python2.7/site-packages/pandas/tseries/plotting.pyc in tsplot(series, plotf, **kwargs)
76
77 # set date formatter, locators and rescale limits
---> 78 format_dateaxis(ax, ax.freq)
79 left, right = _get_xlim(_get_all_lines(ax))
80 ax.set_xlim(left, right)

//anaconda/lib/python2.7/site-packages/pandas/tseries/plotting.pyc in format_dateaxis(subplot, freq)
249 subplot.xaxis.set_major_formatter(majformatter)
250 subplot.xaxis.set_minor_formatter(minformatter)
--> 251 pylab.draw_if_interactive()

//anaconda/lib/python2.7/site-packages/IPython/utils/decorators.pyc in wrapper(_args, *_kw)
41 def wrapper(_args,__kw):
42 wrapper.called = False
---> 43 out = func(_args,**kw)
44 wrapper.called = True
45 return out

//anaconda/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.pyc in draw_if_interactive()
235 figManager = Gcf.get_active()
236 if figManager is not None:
--> 237 figManager.canvas.invalidate()
238
239

AttributeError: 'FigureCanvasAgg' object has no attribute 'invalidate'

Unexpected position quantity change

Hi everyone,

I have just tested the new positions property that was added by Vladimir. I printed the entire pandas DataFrame (Backtest.positions) and noticed something strange. The sample strategy that I used consists of SelectAll, SelectMomentum(1), WeighEqually, Rebalance, RunWeekly. I did not use the new delay functionality that was added for SelectMomentum. Three random securities were chosen for the backtest. Attached is a snippet of the DataFrame:

The phenomenon occurs throughout the DataFrame content. The positions always decrease a little the day after changing. For example, on 2010-09-08 position 1 was 127802. The day thereafter it is suddenly 127765. The same happens on 2010-09-13 and 2010-09-14.

I might be missing something, but what would cause this change in position quantities?

Error no module named future

The new version of ffn which bt uses requires the future module for compatibility between Python 2 & 3. If you used the Anaconda installation for Python and then install bt using "pip install bt" when you run bt you get an error no module named future. "future" is not in the standard Anaconda packages installed and not loaded when installing bt. Hope this isn't too trivial an issue but it took me a couple of hours to track down.
Using "conda install future" fixes the problem.

StatTotalReturn() Date Indexing Bug

Hi,

if found a pretty significant bug(s) when i was testing the momentum functionality.
In the momentum bt.algos.SelectMomentum() algo, the class StatTotalReturn(Algo) is called through call. In this function there's an issue with getting the dates right:

Where:
algo.py > class StatTotalReturn(Algo) > def call(self, target)

Problem:
Getting wrong momentum weights/signals on every monday and day after each holiday.

Description:

1. In the call function the lookback period is blindly subtracted from current date.

"prc = target.universe[selected].ix[t0 - self.lookback:t0]"

The resulting date of this subtraction is however not always a business/trading day, hence
indexing this date in my price dataframe will return a NaN dataframe (using the .ix function). Leading to the momentum signal constantly selecting one specific Asset on the mondays
and day after the holidays.

2. Also, the function does not take into account that on the first day (or days, depending on the lookback period), a return cannot be computed as we don't have previous data. Hence in this case we cannot establish a momentum signal and are to remain fully positioned in cash during this period. The reason the program was working was due to issue 1), where at t_0 we index with a date that doesn't exist in the dataframe, and a false signal is returned.

Solution:

I rewrote the function for myself. Instead of using '.ix' for indexing, which doesn't throw an Error when indexing wrongly, i use '.loc'. Then i use a loop with error catching inside. This catches an
KeyError, and iteratively decreases the lookback date "t0 - self.lookback" by a single day, untill a valid date is found. This takes care of 1). To solve 2), i just compare current date, "target.now", with the very first date + lookback period, if the former is smaller than the latter, the target measure, "target.temp['stat']", is set to a row of NaNs.

As far as i have tested this works for me.
Hope it was helpfull. Please let me know if i overlooked something.

Dion

# EDITTED BY DION JOREN 8/28/2015
def __call__(self, target):
    selected = target.temp['selected']
    tgt = target.universe[selected]

    # initial no-data period, set all values to NA within this window
    if target.now < tgt.index[0] + self.lookback:
        target.temp['stat'] = tgt.iloc[0, :] * np.nan
    else:
        t_now = target.now - self.lag
        t_prev = t_now - self.lookback

        # select lookback period: current date - lookback might be weekend or holiday!
        # hence we need to look for a business day close to this in our dataframe index
        # this is implemented here by stepping backward untill a valid date is found
        while True:
            try:
                tgt.loc[t_prev]  # test if date in index, throws KeyError if not
                prc = target.universe[selected].ix[t_prev : t_now]  # lookback period
                break
            except KeyError:
                t_prev -= pd.DateOffset(days=1)
                pass

        target.temp['stat'] = prc.calc_total_return()
    return True

use BT.get() or ffn.get() to retrieve data for SMI

Hello @pmorissette , I am using BT.get() or ffn.get() to retrieve data to calculate SMI(stochastic momentum index). I like this function as it can get historical intraday high and low for the calculation.
what I have done is like these

data = bt. get("aapl:close, aapl:high, aapl:low")
closing =data. aaplclose
-- more codes---
bt.backtest(strategy, closing)

But I encounter the problem when running the backtesting function, bt.backtest(strategy, closing). The error message says something like the data "closing" does not have attribute of columns.

Could you please check?

Thanks,

Jun

Target weights

Hey,

Could you provide an example of how to use the target weight DF?

Thanks,
Mike

Example for implementation of CapitalFlow algo

Hi I just started to use the bt backtesting framework. I understand how to use most of the framework, but don't quite understand how to apply the CapitalFlow algo. My understanding is that the CapitalFlow algo inserts an amount specified by the user within every iteration run of the Algo whether daily, monthly or annually. I'm not quite sure how to visualize the results with capital injections. I have only found the plot visualization. I just would like an example how the CapitalFlow algo works.

Thanks,

moses

A few questions

Hi Philippe,

First of all : Bonne année 2016 (I guess from your name that you may be speaking French).

I had a lot of fun playing with your project and it actually fits my needs to rapidly prototype some strategies. It also fits my way of thinking about a trading system implementation. I'm pretty new to python, designed my own trading system in C++, and it's a great tool for what I'm intending to achieve: I'll probably build on it for my personal ETF trading.

The way I use Bt today is the following:

• I load the data
• do my computing sauce to generate weights for each asset traded
• give the weights to the backtester
• enjoy the simulation and results (not always though :-) )

In this context, I have a few question:

• how to pass a commission function for a fixed $ commission + % of the $ traded? (with a bit more experience of python, I could pbly figure that out, but a quick hint would save me a couple of days)
• can you please confirm (in my context of giving weights for assets) that you adjust the position and execute on the same day and price for which the new weight is given? (no lag in trading, such as trading the next day, ... ; btw, I'm perfectly fine with trading instantly)
• is that possible to build a cumulated performance instead of a compounded performance?

Thank you very much for your time and help.

Best,
Joris

Could not find suitable distribution for Requirement.parse('numpy>=1.7.0')

Hi I ran the command
"pip install bt"

on unbuntu 14.4, but I received this error. I'm not sure how to fix it.
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy>=1.7.0')

I tried to install numpy and pandas using ubuntu app-get command, but that didn't fix the problem.

Moses

On rerunning a back test

Is the following use case supported?

I would like to run a strategy from t1 to t2. Then I would like to run that same strategy from t2 to t3 but have the performance stats print for the period t1 to t3. The use case for this is to prevent having to re-run a back test for full history each day.

Resultant weights of backtest not summing to 1

If I call this after running the backtest, for example:
mybacktest = bt.Backtest(long_equity_bt, data=dfp['RI'],initial_capital = 100000000,commissions=lambda q,p:100.0 + 0.003*p*q),

mybacktest.weights.ix[some_date,1:] is returning 0.9923 and not 1. I understand that this difference is the held back amount for rounding of stocks and fees if any.

This subsequently means that I trade more stocks than I expect. Is this expected behaviour?

On further investigation, if I rebase my desired target weights to the sum of target.children[c].weight for all nodes I get my desired number of trades, but it seems that the return of the strategy in question is impacted by the differential.

Am I using the bt framework correctly?

Going to account for the unallocated capital in strategy and see how that goes.

Problem with bt.Backtest

Code:
import bt

data = bt.get('spy, agg', start='2010-01-01')

s = bt.Strategy('s1', [bt.algos.RunMonthly(),
bt.algos.SelectAll(),
bt.algos.WeighEqually(),
bt.algos.Rebalance()])

test = bt.Backtest(s, data)
res = bt.run(test)

AttributeError Traceback (most recent call last)
in ()
8 bt.algos.Rebalance()])
9
---> 10 test = bt.Backtest(s, data)
11 res = bt.run(test)

C:\Users****\Anaconda\lib\site-packages\bt\backtest.pyc in init(self, strategy, data, name, initial_capital, commissions, integer_positions)
120 integer_positions=True):
121
--> 122 if data.columns.duplicated().any():
123 cols = data.columns[data.columns.duplicated().tolist()].tolist()
124 raise Exception(

AttributeError: 'Index' object has no attribute 'duplicated'

order type simulation

How to simulate market-on-open (MOC), limit order or stop limit order?

Sentiment Trading

Hi I am having an issue running this code, can anyone figure out what mistake I am making?
I am basically downloading the SPY and a sentiment dataset from Quandl and doing exactly what Phil did in the "SMA" tutorial. When I run the code I get lots of errors.

import bt
import Quandl
import pandas as pd

#Download the Sentiment Data
sent= Quandl.get("ISE/EQU_SI", transform="rdiff", start_date="2010-01-01")
sent=sent['2010':'2016']

#Download SPY data
data1=Quandl.get("GOOG/NYSE_SPY", start_date="2010-01-01")
data1=data1['2010':'2016']
data1=data1.dropna()
data1=pd.DataFrame({"price":data1['Close']},)

#Generate Weights
tw1=sent.copy(deep=True)
signal=tw1['Equities']
tw1[signal >= 0]=1.0
tw1[signal < 0]=-1.0
tw1=pd.DataFrame({"sent":tw1['Equities']},)


#Convert to monthly Data
data2=data1.copy(deep=True)
data2=data2.to_monthly()

tw2=tw1.copy(deep=True)
tw2=tw2.to_monthly()

#Run the Strategy
strad=bt.Strategy('strad',[bt.algos.WeighSpecified(tw2),bt.algos.Rebalance()])
t1=bt.Backtest(strad,data2)
res1=bt.run(t1)
res1.plot()

When I run this code, I get the following errors:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/bt/backtest.py", line 26, in run
    bkt.run()
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/bt/backtest.py", line 169, in run
    self.stats = self.strategy.prices.calc_perf_stats()
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/ffn/core.py", line 1005, in calc_perf_stats
    return PerformanceStats(obj)
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/ffn/core.py", line 85, in __init__
    self._update(self.prices)
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/ffn/core.py", line 107, in _update
    self._calculate(obj)
  File "/Applications/Anaconda_Python2.7/anaconda/lib/python2.7/site-packages/ffn/core.py", line 315, in _calculate
    self.yearly_vol)
ZeroDivisionError: float division by zero
>>> res1.plot()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'res1' is not defined

Error when backtesting momentum on CAC 40

I tried backtesting a simple momentum strategy on the CAC 40. The strategy code is as follows:

s = bt.Strategy('Strategy1', 
    [bt.algos.RunWeekly(),
     bt.algos.SelectAll(),
     bt.algos.SelectMomentum(5, lookback=pd.DateOffset(months=1), lag=pd.DateOffset(days=1)),
     bt.algos.WeighEqually(),
     bt.algos.Rebalance()])

I generated the CAC 40 securities from a text file. The funny thing is that when I backtest the first 20 securities, everything is fine. The same for the last 20. When the entire 40 securities are used for the backtest, I get the following error:

Traceback (most recent call last):
  File "SampleBacktest.py", line 262, in <module>
    res = bt.run(t)
  File "/home/fractally/anaconda/lib/python2.7/site-packages/bt/backtest.py", line 25, in run
    bkt.run()
  File "/home/fractally/anaconda/lib/python2.7/site-packages/bt/backtest.py", line 155, in run
    self.strategy.run()
  File "bt/core.py", line 1132, in bt.core.Strategy.run (bt/core.c:14486)
  File "bt/core.py", line 1077, in bt.core.AlgoStack.__call__ (bt/core.c:13888)
  File "/home/fractally/anaconda/lib/python2.7/site-packages/bt/algos.py", line 1131, in __call__
    target.rebalance(item[1], child=item[0], base=base)
  File "bt/core.py", line 673, in bt.core.StrategyBase.rebalance (bt/core.c:8907)
  File "bt/core.py", line 900, in bt.core.SecurityBase.update (bt/core.c:11717)
ValueError: setting an array element with a sequence.

I have yet to find the issue - I'm not sure which array is set with a sequence internally. I'll see if I can find anything later. If someone could perhaps run the backtest on the tickers (that I will list below) using any simple momentum strategy (like the one I provided), to check if the same issue arises I would appreciate it. In order to use Yahoo Finance as the data source, just add .PA to each ticker symbol e.g. AC.PA. Any ideas on the error would also be great.

AC
AI
ALU
AIR
ALO
MT
CS
BNP
EN
CAP
CA
ACA
EDF
EI
GTO
GSZ
BN
KER
OR
LG
LR
MC
ML
ORA
RI
KER
PUB
RNO
SAF
SGO
SAN
SU
GLE
STM
TEC
FP
UL
FR
VK
VIE
DG
VIV

Thanks!

res.plot() don't work

i am trying to follow the example code ,but this method didn't work

class bt.algos.RunOnDate(*dates)

s = bt.Strategy('s1', [bt.algos.RunOnDate(my_dates), bt.algos.SelectAll(), bt.algos.WeighEqually(), bt.algos.Rebalance()])

test = bt.Backtest(s, data) res = bt.run(test)

Here are the dates I am trying to pass:
['2016-01-04', '2016-02-01', '2016-02-29', '2016-03-28', '2016-04-25', '2016-05-23', '2016-06-20', '2016-07-18', '2016-08-15', '2016-09-12', '2016-10-10', '2016-11-07', '2016-12-05']

And here is the error:

`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in ()
1 test = bt.Backtest(s, data)
----> 2 res = bt.run(test)

C:\Anaconda\envs\BT\lib\site-packages\bt\backtest.py in run(_backtests)
24 # run each backtest
25 for bkt in backtests:
---> 26 bkt.run()
27
28 return Result(_backtests)

C:\Anaconda\envs\BT\lib\site-packages\bt\backtest.py in run(self)
163 self.strategy.update(dt)
164 if not self.strategy.bankrupt:
--> 165 self.strategy.run()
166 # need update after to save weights, values and such
167 self.strategy.update(dt)

C:\Anaconda\envs\BT\lib\site-packages\bt\core.cp35-win_amd64.pyd in bt.core.Strategy.run (bt/core.c:14718)()

C:\Anaconda\envs\BT\lib\site-packages\bt\core.cp35-win_amd64.pyd in bt.core.AlgoStack.call (bt/core.c:14120)()

C:\Anaconda\envs\BT\lib\site-packages\bt\algos.py in call(self, target)
333
334 def call(self, target):
--> 335 return target.now in self.dates
336
337

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
`

more information about drawdown details

Hi,
In my writing my backtest, I was interested in the lowest day of any drawdown, so, please consider add lowest day column in drawdown dataframe.

Downloading data from Yahoo

Hey guys,

I've modified this post to correspond with what I've noticed when using the ticker FR.PA. The data does exist, but during backtesting (with a simple momentum strategy) the following error occurs when executing the following code during the backtest: target.temp['stat'] = prc.calc_total_return(). The error is:

IndexError: index -1 is out of bounds for axis 0 with size 0

At first I thought that the data was unavailable, but I think it is a matter of the .csv being corrupt after downloading. I'll delete the issue later once I've confirmed by suspicion.

Type of license

Hi Philippe,

I think it would be good if you include the type of license in you repo. Especially it is important since this library in most of the cases imply commercial and proprietary use, and you definitely don't want to have any liabilities to those who use your code ;-) By default with no-license your code is protected by the copyright, so: "This means that you retain all rights to your source code and that nobody else may reproduce, distribute, or create derivative works from your work."

Unfortunately I could not advise you much here - I'm not a legal specialist. But you can have a look at a rather good summary here: https://tldrlegal.com/ or here: http://choosealicense.com/. Also there's a FAQ here: https://help.github.com/articles/open-source-licensing/.

Personally, I prefer non-restrictive open-source licenses, and license my code under MIT license. One of the drawback of restrictive GPL-type of licenses is a requirement to keep the license type and pass source code together with binaries when distributing, which often might be a "no-go" for financial applications.

Best regards
Vladimir

Plots not apperaring

I'm trying bt and seems everything working, except that plots not showing up. Using Python 2.7, Anaconda, IPython Notebook, Windows 8 64 bit. No errors whatsoever.

BUG: reusing of child strategy of a tree strategy

There's a problem of reusing child strategies of a tree strategy.
Direct approach does not work, raising a strange exception (AttributeError: 'int' object has no attribute 'month'):

df = bt.get('aapl,msft,gs', start='2014-01-01')

mom_s = bt.Strategy('mom_s', [bt.algos.RunMonthly(),
                              bt.algos.SelectAll(),
                              bt.algos.SelectMomentum(1),
                              bt.algos.WeighEqually(),
                              bt.algos.Rebalance()],
                    ['aapl', 'msft'])

master = bt.Strategy('master', [bt.algos.RunMonthly(),
                                bt.algos.SelectAll(),
                                bt.algos.WeighEqually(),
                                bt.algos.Rebalance()],
                     [mom_s, 'gs'])


t2 = bt.Backtest(mom_s, df)
r2 = bt.run(t2)

t = bt.Backtest(master, df)
r = bt.run(t)

test_strategybase_tree_rebalance_level2 failed

When I run this test, This function test_strategybase_tree_rebalance_level2 failed because of the nan value of c1.weight.

BTW, I port bt and ffn to python3 and run the test with nose, so maybe the port is not so perfect which lead to this bug.

'Series' object has no attribute 'items'

hello,
I try to run the example code on your website, the "SMA Crossover Strategy".
Things happen in

ma_cross = bt.Strategy('ma_cross', [bt.algos.WeighTarget(tw),
bt.algos.Rebalance()])

t = bt.Backtest(ma_cross, data)
res = bt.run(t)

res.plot()

which the error is

AttributeError Traceback (most recent call last)
in ()
25
26 t = bt.Backtest(ma_cross, data)
---> 27 res = bt.run(t)
28
29 res.plot()

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/bt/backtest.pyc in run(_backtests)
24 # run each backtest
25 for bkt in backtests:
---> 26 bkt.run()
27
28 return Result(_backtests)

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/bt/backtest.pyc in run(self)
163 self.strategy.update(dt)
164 if not self.strategy.bankrupt:
--> 165 self.strategy.run()
166 # need update after to save weights, values and such
167 self.strategy.update(dt)

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/bt/core.so in bt.core.Strategy.run (bt/core.c:15301)()

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/bt/core.so in bt.core.AlgoStack.call (bt/core.c:14703)()

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/bt/algos.pyc in call(self, target)
1222 # use it as base in rebalance calls
1223 base = target.value
-> 1224 for item in targets.items():
1225 target.rebalance(item[1], child=item[0], base=base)
1226

/opt/anaconda/envs/np18py27-1.9/lib/python2.7/site-packages/pandas/core/generic.pyc in getattr(self, name)
2143 or name in self._metadata
2144 or name in self._accessors):
-> 2145 return object.getattribute(self, name)
2146 else:
2147 if name in self._info_axis:

AttributeError: 'Series' object has no attribute 'items'

Help with WeighTarget algo

I am trying to rebalance annually. If my data starts on 1/1/2000 then the target weights do not kick in until 12/31/2000 so the strat return flatlines from 1/1/2000 until the first rebalance at 12/31/2000. i am using resample to get the annual tw df. I do not have this issue with df.to_monthly... it seems the target weights are provided at the beginning of the strat period. any suggestions on how to handle this?

How to calculate return?

Hello from Korea, I'm eunbi, cho.

I have analyzed 'bt' for using, but I found some error at result what I didn't expect on 'bt'.

here is my testing code.
When I see the result of this backtest, the value of return is different from what I expected.

I think 's1' is strategy of Rebalancing when month is changed, and at the rebalance change weight based on 'tw'

I tested with asset 'a' and 'b' and I found that bt.Backtest(s,data) execute run() function in Backtest class and in the function there is self.strategy.run() code. I think this strategy.run() function might calculate portfolio values at each day. However, I cannot find implementation of this strategy.run() function. I think self.strategy is class Strategy, and the Strategy is child class of StrategyBase. So, I think that self.strategy.run()'s implementaion is run() in StrategyBase, but when I debug, that code is not executable and I can't debug about run() function...

s = bt.Strategy('s1', [bt.algos.RunMonthly(),
bt.algos.SelectAll(),
bt.algos.WeighTarget(tw),
bt.algos.Rebalance()])

t = bt.Backtest(s, data)
res = bt.run(t)

res.display()

I want to the each asset's number of stocks is maintain in period of rebalancing.
for example, this is number of stocks, rebalance frequency is monthly.
a
2011/1/1 20
2011/1/2 20
2011/1/3 20
...
...
...
2011/2/1 30
2011/2/2 30

Because, exsisting logic is based on daily trading, then the number of stocks is also daily changed.
What I want is I want to fix the number of stocks before the rebalancing.

thank you for reading, and I wait your response about this problem.:)

Unable to find vcvarsall.bat

Would you be able to provide a pre compiled version of BT available for pip or standalone? I am using multiple versions of Python and have never been able to get the C Compiler for Windows right. I want to spend my time developing systems not worrying about Windows inadequacies.

Using CSV files

Is there a way to load data from CSV files instead of Google or Yahoo finance - ie through Pandas?
An example would be great, thanks.

Bottleneck of update() - pandas operations

For simple strategies bt.core.StrategyBase.update() and bt.core.SecurityBase.update() are the bottlenecks in terms of performance. E.g. this is a profiling for a simple (SelectAll - WeightEqually - Rebalance) strategy for ~100 assets and 45 years of history:

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   138                                               @profile
   139                                               def run(self):
   140                                                   """
   141                                                   Runs the Backtest.
   142                                                   """
   143                                                   # set run flag
   144         1            4      4.0      0.0          self.has_run = True
   145                                           
   146                                                   # setup strategy
   147         1         1823   1823.0      0.0          self.strategy.setup(self.data)
   148                                           
   149                                                   # adjust strategy with initial capital
   150         1           72     72.0      0.0          self.strategy.adjust(self.initial_capital)
   151                                           
   152                                                   # loop through dates
   153     15382       243227     15.8      0.6          for dt in self.dates:
   154     15381     30341164   1972.6     72.0              self.strategy.update(dt)
   155     15381        16845      1.1      0.0              if not self.strategy.bankrupt:
   156     15381      3982239    258.9      9.5                  self.strategy.run()
   157                                                           # need update after to save weights, values and such
   158     15381      7133247    463.8     16.9                  self.strategy.update(dt)
   159                                           
   160         1       420509 420509.0      1.0          self.stats = self.strategy.prices.calc_perf_stats()
   161         1          193    193.0      0.0          self._original_prices = self.strategy.prices

If we go deeper - then the bottlenecks of .update() are pandas operations of setting a value:

self._values[date] = val

and call of c.update(date, data) where the same operations waste most of the time.