mhallsmoore / qsforex Goto Github PK

First run of trading/trading.py:

"Import error: No module named qsforex.execution.execution"

The warning is as follows:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.or...
InsecureRequestWarning)

Hello,

it will be a great idea to provide a pip package.

Kind regards

Have useful input instead of direct pasting of server/response messages to the screen.

When I'm reading the code, I noticed that qsforex can't have position with trailing stop.
https://github.com/mhallsmoore/qsforex/blob/master/execution/execution.py#L62

Is this my misunderstand? If somebody knows how to make order with trailing stop, please teach me.

Looks like all new accounts go to the V20 interface. I don't see an option to use the old legacy v1 interface.

Is there any plans on supporting v20?

Would be fun to toy with LSTMs using tensorflow or something lol. I've already started switching some of the code. Pretty much just stream and execution to use v20 using the abstraction layer https://github.com/hootnot/oanda-api-v20. I'll make a pull request when I am done if anyone want to check that out.

Re: http://www.quantstart.com/articles/Forex-Trading-Diary-5-Trading-Multiple-Currency-Pairs#comment-2029154151

Hello. I am running Ubuntu 14.04 and have tried using this program. When I try to run trading.py I obtain the following error message:

Traceback (most recent call last):
File "trading/trading.py", line 68, in
prices, events, equity=equity, backtest=False
File "/home/justin/forexEnv/local/lib/python2.7/site-packages/qsforex/portfolio/portfolio.py", line 31, in init
self.backtest_file = self.create_equity_file()
File "/home/justin/forexEnv/local/lib/python2.7/site-packages/qsforex/portfolio/portfolio.py", line 74, in create_equity_file
out_file = open(os.path.join(OUTPUT_RESULTS_DIR, filename), "w")
File "/home/justin/forexEnv/lib/python2.7/posixpath.py", line 77, in join
elif path == '' or path.endswith('/'):

I would really appreciate your help, and thanks for making such a helpful blog.

Note: I doubt this would make much of a difference, but I was having a lot of difficulty installing scipy in a virtualenv so I used the --system-site-packages options when creating the virtualenv.

Metatrader 5 provides 2 kind of position accounting
https://www.mql5.com/en/articles/2299

• Netting system
• Hedging system

It will be a nice feature to have in qsforex / qstrader

I would be interested in using qsforex for trading Bitcoins on Kraken and would be willing to implement PriceHandler and ExecutionHandler.
My problem is that I am not sure how these modifications will fit into the overall architecture without making a total mess. If someone could help me or point me to solutions, I would be willing to code the suggested changes if I have time.

Or should I look into qstrader instead?

Currently the code is only working for Python 2.7.x, but since Python 3.4.x is now rapidly becoming the standard, the code needs to be updated.

Hi,

You might use requests instead of urllib
https://github.com/mhallsmoore/qsforex/blob/d191aad64102ebfa113357987f59f61f0667a190/execution/execution.py

http://docs.python-requests.org/en/latest/

So you won't have to maintain boilerplate code to support both Python 2 and Python 3

Kind regards

Exception in thread Thread-1:

Traceback (most recent call last):

File "C:\Python27\lib\threading.py", line 810, in

__bootstrap_inner

self.run()

File "C:\Python27\lib\threading.py", line 763, in run

self.__target(_self.__args, *_self.__kwargs)

File "trading.py", line 40, in trade

execution.execute_order(event)

File "c:\documents\qsforex\execution\execution.py", line 71, in

execute_order

response = self.conn.getresponse().read()

File "C:\Python27\lib\httplib.py", line 1132, in getresponse

response.begin()

File "C:\Python27\lib\httplib.py", line 453, in begin

version, status, reason = self._read_status()

File "C:\Python27\lib\httplib.py", line 417, in _read_status

raise BadStatusLine(line)

BadStatusLine: ''

Some people have experienced this bug when holding positions for a long time (20 mins+). Investigate!

"I was reading through your code, specially the 'Position' class and something seemed a bit confusing. When you define 'remove_units' method, it doesn't seem that you are using the 'remove_price', shouldn't that be used to update the 'self.cur_price' instead?"

$ python qsforex/examples/mac.py
/Users/femto/cache/data/random/GBPUSD_20140101.csv
Traceback (most recent call last):
  File "qsforex/examples/mac.py", line 27, in <module>
    equity=settings.EQUITY
  File "/Users/femto/github/femto/qsforex/backtest/backtest.py", line 29, in __init__
    self.ticker = data_handler(self.pairs, self.events, self.csv_dir)
  File "/Users/femto/github/femto/qsforex/data/price.py", line 107, in __init__
    self.file_dates[self.cur_date_idx]
  File "/Users/femto/github/femto/qsforex/data/price.py", line 144, in _open_convert_csv_files_for_day
    names=("Time", "Ask", "Bid", "AskVolume", "BidVolume")
  File "//anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "//anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 315, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "//anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 641, in __init__
    self.options, self.engine = self._clean_options(options, engine)
  File "//anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 755, in _clean_options
    _validate_header_arg(options['header'])
  File "//anaconda/lib/python3.5/site-packages/pandas/io/common.py", line 265, in _validate_header_arg
    raise TypeError("Passing a bool to header is invalid. "
TypeError: Passing a bool to header is invalid. Use header=None for no header or header=int or list-like of ints to specify the row(s) making up the column names

According docstring

`````` ?pd.read_csv```

header : int or list of ints, default 'infer'
    Row number(s) to use as the column names, and the start of the data.
    Default behavior is as if set to 0 if no ``names`` passed, otherwise
    ``None``. Explicitly pass ``header=0`` to be able to replace existing
    names. The header can be a list of integers that specify row locations for
    a multi-index on the columns e.g. [0,1,3]. Intervening rows that are not
    specified will be skipped (e.g. 2 in this example is skipped). Note that
    this parameter ignores commented lines and empty lines if
    ``skip_blank_lines=True``, so header=0 denotes the first line of data
    rather than the first line of the file

so in https://github.com/mhallsmoore/qsforex/blob/master/data/price.py#L140

        self.pair_frames[p] = pd.io.parsers.read_csv(
            pair_path, header=True, index_col=0, 
            parse_dates=True, dayfirst=True,
            names=("Time", "Ask", "Bid", "AskVolume", "BidVolume")
        )

should be replaced by

        self.pair_frames[p] = pd.io.parsers.read_csv(
            pair_path, header=0, index_col=0, 
            parse_dates=True, dayfirst=True,
            names=("Time", "Ask", "Bid", "AskVolume", "BidVolume")
        )

Hello,

it will be a good idea to add some unit tests and continuous integration (Travis might help)

Kind regards

Dear Michael,

when you are inverting prices in the PriceHandler
inv_bid shall be 1/ask and inv_ask shall be 1/bid
since bid is always smaller than ask

Thanks,
Maxim

Hello,

I see in your code https://github.com/mhallsmoore/qsforex/blob/master/backtest/backtest.py#L60
usage of string

Why not using Enum ?

For Python 3
https://docs.python.org/3/library/enum.html

For Python 2.7 there is enum34 https://pypi.python.org/pypi/enum34

Kind regards

PS: "buy", "sell", "long", "short" should also be Enum

Here is an example usage

in event.py

from enum import Enum
EventType = Enum("EventType", "TICK BAR SIGNAL ORDER FILL")

in backtest.py

if event.type == EventType.TICK:
    self.strategy.calculate_signals(event)
    self.portfolio.update_portfolio(event)
    self.ticks += 1
elif event.type == EventType.SIGNAL:
    self.portfolio.execute_signal(event)
elif event.type == EventType.ORDER:
    self.execution.execute_order(event)

Other Enums to define:

in price_handler/__init__.py

PriceHandlerType = Enum("PriceHandlerType", "TICK BAR")

in portfolio/position.py

PositionType = Enum("PositionType", "LONG", "SHORT")

For qstrader

in position/position.py

PositionActionType = Enum("PositionActionType", "BOT", "SLD")

HI, I'm looking at the code trying to figure out how hard to make it works to another instrument, like commodity. I'm getting into commodity trading strategy, focusing on intraday pair trading arb.

thanks

Hi,
I use Windows 10. When I've ran "traiding.py" in Spyder appear the following error: ImportError: No module named 'qsforex'

I setup an account to obtain the API authentication credentials, which you will need to carry out live trading.
I made the changes in "settings.py" according to "Forex Trading Diary 1 - Automated Forex Trading with the AONDA API".
I've downloaded the zip file of the current master branch at https://github.com/mhallsmoore/qsforex/archive/master.zip.
I put all the files in the same directory.
I created two directories in it: "QSFOREX_CSV_DATA_DIR" and "QSFOREX_OUTPUT_RESULTS_DIR".
I don't know set environment variables.

Can anyone help to solve the problem?

Best Regards.
Edilson

Hi,

Would be useful to add a little GUI (Graphic User Interface) to the module.

=> A few entry forms for key parameters
=> A few entry forms for config settings
=> A button to start and stop trading.

I will be doing some work upon it also for my instance of qsforex.

Firstly, thank you very much for the great work on qsforex.

I believe one could develop and backtest strategies based on OHLCV data faster (considering calculations necessary, time jumps faster etc..)... Since I thought considering spread, trading would be profitable at least in the scale of half an hour... As such feature is already established in qstrader, do you think it could also be established on qsforex?

mhallsmoore/qstrader#12

Thank you :)

I would also try to do this on my side as it seems to be very important to me :)

the format of pair string is too hard coded. Thus the whole package can only deal with pure currency pairs. But for some new supported product, it doesn't work at all, ex:AU200_AUD, wheat_usd, etc. The format should be changed such that instead of
instrument = "%s_%s" % (event.instrument[:3], event.instrument[3:])
we can simply input the pair string. and we just need pair.split('_') to get the list of the two instrument.

moreover, why not use some oanda api wrapper to do the execution instead of writing our own?
There are well developed python wrappers for v1 and v20 account, we just need to do an conditional import

The portfolio class works with the variable 'side' with possible values: 'buy' and 'sell' where the position class except it to be LONG or SHORT (line 21 position.py)

Add a logging system with timestamps to account for new positions being added/removed, changes to account balance etc.

Dear Michael,

If one wants to trade GBPEUR pair using USD nominated account,
he will need GBPEUR, GBPUSD and EURUSD prices to calculate pnl.

So, I think that home/quote and home/base pairs as well as their reciprocals shall be in included in the price_dict.

Thanks,
Maxim

Hi Mike,

I previously implemented your code that you provided in Trading Diary #1 and #2, and it seems good. However, after I updated the code according to the Github resource, when I run the trading.py there's no feedback from the screen, and no error information neither. Any suggestions?

Best,

Weibo

Hi Michael,
Can I ask a question about the heartbeat?
Does the thread stop as the heartbeat interval if there is no new ticks coming in? And the thread resumes back to work if there's new ticks coming in?
How can I make my trading system working on a 5 or 15 minutes time frame interval instead of tick by tick?

Thank you.

Regards,
Wei

2016-08-22 20:01:33,803 - qsforex.trading.trading - INFO - Received new order event: Type: ORDER, Instrument: GBPUSD, Units: 2000, Order Type: market, Side: buy
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "trading/roc.py", line 45, in trade
execution.execute_order(event)
File "/home/weiwu/.virtualenvs/qsforex/local/lib/python2.7/site-packages/qsforex/execution/execution.py", line 73, in execute_order
response = self.conn.getresponse().read().decode("utf-8").replace("\n","").replace("\t","")
File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 415, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''

2016-08-22 21:15:34,320 - qsforex.trading.trading - INFO - Received new order event: Type: ORDER, Instrument: GBPUSD, Units: 2000, Order Type: market, Side: sell
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "trading/roc.py", line 45, in trade
execution.execute_order(event)
File "/home/weiwu/.virtualenvs/qsforex/local/lib/python2.7/site-packages/qsforex/execution/execution.py", line 71, in execute_order
params, headers
File "/usr/lib/python2.7/httplib.py", line 979, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
self.putrequest(method, url, **skips)
File "/usr/lib/python2.7/httplib.py", line 877, in putrequest
raise CannotSendRequest()
CannotSendRequest

After this error the trade thread is dead, the program will only show price streaming debug log into the console.
how to solve this? thank you.

When running the backtest for mac.py I get an error as per below.
(qsforex) jbowler@GEN-U-DAE-01:~/Projects/qsforex$ python examples/mac.py

Traceback (most recent call last):
File "examples/mac.py", line 29, in
backtest.simulate_trading()
File "/home/jbowler/venv/qsforex/lib/python2.7/site-packages/qsforex/backtest/backtest.py", line 81, in simulate_trading
self._run_backtest()
File "/home/jbowler/venv/qsforex/lib/python2.7/site-packages/qsforex/backtest/backtest.py", line 57, in _run_backtest
self.ticker.stream_next_tick()
File "/home/jbowler/venv/qsforex/lib/python2.7/site-packages/qsforex/data/price.py", line 182, in stream_next_tick
bid = Decimal(str(row["Bid"])).quantize(
File "/home/jbowler/ProgramData/anaconda2/lib/python2.7/decimal.py", line 547, in new
"Invalid literal for Decimal: %r" % value)
File "/home/jbowler/ProgramData/anaconda2/lib/python2.7/decimal.py", line 3873, in _raise_error
raise error(explanation)
decimal.InvalidOperation: Invalid literal for Decimal: 'Bid'

How is this fixed?

Thank-you

The drawdown curve produced subsequent to the backtest takes a long time to calculate. Profile it and improve the speed (possibly by removing a direct for-loop).

This is the output I get after running python trading.py
Traceback (most recent call last): File "trading/trading.py", line 12, in <module> from qsforex.execution.execution import OANDAExecutionHandler File "/root/venv/qsforex/local/lib/python2.7/site-packages/qsforex/execution/execution.py", line 13, in <module> import urllib3 ImportError: No module named urllib3

This could just be my setup, but I've tried pip install urllib3 --upgrade and the issue persists

Hi there!

is this project still alive? I see many pull requests still opened.

I see that development ceased about two years ago as per the datetime in the code listing.

1. Has the code moved elsewhere or has development in fact stopped?
2. Is qstrader a replacement for qsforex? (I realize that qstrader is explicitly mentioned to be for equities but I figured that I would check)

I ran trading/trading.py but the output to the screen did not occur
(qsforex) gs-macbook-air:qsforex ghbcode$ python trading/trading.py
2017-02-09 10:24:10,978 - qsforex.trading.trading - INFO - Starting trading thread
2017-02-09 10:24:10,978 - qsforex.trading.trading - INFO - Starting price streaming thread

I found a discussion about this but it didn't help. I changed the heartbeat from 0.0 to 0.5 and no luck. So I traced the program and found that 'resp' from trading.py returns
<Response [400]>

In the two years that development ceased the request syntax may have changed.

3. If I wanted to pick up where would I find out the correct syntax for the request and in fact the syntax for Oanda's forex api currently?

Please excuse me if this is covered elsewhere. I looked around and via google without much luck.

Thanks!

"Also, when we are calculating the PnL we always hedge back to home currency. So why aren't we using 'Bid' always? For example, say we are short on GBP_USD and the account is denominated in HKD. If we want to close the position then we buy back GBP_USD however we should still use 'bid' price for 'USD_HKD'."

Hi Mike, Thanks for sharing the code. I noticed the time interval between two ticks from Oanda is not constant. Sometimes it is short, around 0.1s. Sometimes, it can be more than 5 seconds. I am wondering is it my internet problem or code itself?
Jack

Hi, Michael, I am getting this error under the virtual environment when executing this command:
python trading/trading.py
can you help? thank you.

Traceback (most recent call last): File "trading/trading.py", line 49, in <module> logging.config.fileConfig('../logging.conf') File "/usr/lib/python2.7/logging/config.py", line 77, in fileConfig formatters = _create_formatters(cp) File "/usr/lib/python2.7/logging/config.py", line 113, in _create_formatters flist = cp.get("formatters", "keys") File "/usr/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'formatters'