penberg / helix Goto Github PK

Which version of libuv does Helix require? I have libuv 1.6.1 and its API is incompatible with the API Helix uses.

a lot of protocols has decimal conversion factor other than ITCH's 1e-4, for example MDP3 has a factor 1e-7. some protocols has variable (by instrument) factor.

while processing large ITCH files would be great to print progress to stderr

order book / order should be able to hold negative price. NASDAQ TV does not have this but a lot of derivative exchanges do

we need to add the following to the order:

• timestamp
• priority_id (uint64_t) - sometimes it's = timestamp, some venues assign it differently. for ITCH it's a copy of timestamp
• flags - for example if we know that it's ours order we can set FLAG_OWN or FLAG_IMPLIED
• filled_quantity - for partially filled orders it may be relevant to know
• user_data - opaque pointer

helix-trace assumes that prices have a scaling factor of 10000 and two decimal digits and that sizes have a scaling factor of 1 and no decimal digits. This works for Nasdaq and Parity until 0.6.0, but Parity 0.7.0 introduces support for configurable number of decimal digits for both prices and sizes.

I'm working on a patch to update the Parity PMD support to Parity 0.7.0. What's your preference for displaying prices and sizes in helix-trace? I can think of two options:

• display the values as-they-are without scaling, or

• add command line options to specify the number of decimal digits for prices and sizes.

The NASDAQ MoldUDP and MoldUDP64 protocol handlers don't currently check the session ID at all.

do you have all the sample NASDAQ data you want? i may have access to some files

would be great for tools and other classes to support addition of new / user parsers. for example helix-trace could discover available parsers

The MoldUDP handler does not support gap-fill like MoldUDP64 does since commit 78f976c

i've added some statistics printing to helix-trace. seems like the trade size is too high, could you please have a look?

volume (mio): 10433240092.4563, notional (mio): 2085880461126.7920

i think unordered_map is rather slow thing. can we consider something else? like own fixed size hash map? also you want to NOT resize at all cost

as a test of correctness would be great to do assert(best bid < best ask)

one of my most important applications to decent ITCH parser would be resampling data files into BBO (only best bid price, total size, # of orders, best ask, total size, # of orders) format. plus trades.

the output format would be simple fixed offsets with timestamp, current bid & ask, order book id in case of new quote. and timestamp, price, aggressor, size of a trade.

csv is great but for performance would be even better to have binary format option

would be great to output multiple symbols (or all symbols)

The unit of timestamps is not standardized across the API. For example, the feed handler API returns milliseconds for Nasdaq Nordic ITCH but nanoseconds for the US ITCH variant.

when you start a trading system in the middle of a day (it's often needed) history of orders is not available. therefore there is 100% chance of getting a lot of "invalid_argument(string("invalid order id: ") + to_string(order_id))" exceptions (hundreds).

as you probably know exceptions takes a lot of time to process in C++ unfortunately. i believe we should get rid of all the "throw" clauses.

typedef multi_index_container<
order,
indexed_by<
hashed_unique<
member<order, uint64_t, &order::id>>>
> orders_type;

see https://github.com/divaykin/quick-n-dirty-bench/blob/master/hashmap.cpp#L98

it could be seriously faster for our case.

also you can replace your map<> with ordered containers. or even try to place bids and asks into the same container just multiple indices

Great tool, thanks so much for sharing!
I am running helix-trace using the following command line:

./helix-trace -i /media/sf_data/ITCH5.0/08302017.NASDAQ_ITCH50 -s AAPL -P nasdaq-binaryfile-itch50 -f csv -o AAPL.csv

08302017.NASDAQ_ITCH50 is downloaded from ftp://emi.nasdaq.com/ITCH/ and is roughly 5.4gb

When I run the tool, it steadily uses more and more memory, until it crashes (I only have 4gb of memory). I am running using Fedora 26. Any ideas what is going on? Does helix not delete orders after they are executed or cancelled?

thanks
Nick

most of orders are of no interest to the user. i'd be interested in something what changes TOB (best bid / offer) only.

could you please create the following filter levels:

• all events
• only TOB (new price or total size change)
• TOB on new price or size changed comparing to previous by +/- N%
• only TOB with new price

maybe it'd be easier to create two kind of filters - by price and by size or something along these lines

according to perf now a lot of time is spent in here https://github.com/penberg/helix/blob/master/src/nasdaq/itch50_handler.cc#L132

is stock locate constant for a symbol always? what we can do is - in a build file user should define list of symbols + stock locate he need. it's transformed to C/C++ array r switch statement which would contain order book pointer. should be faster than map.find().

The MoldUDP handlers invoke callbacks even if the stream doing gap-fill. This is dangerous because the callers have no idea they're seeing stale data.

current performance test assumes constant price. therefore multiple price_levels are not used

Currently, we have a separate callback for order book events and trades. This is a problematic API because a trade event obviously has an impact on the order book but it's impossible to tell whether a price level size changed because of an execution or an cancellation. Trade event callbacks, on the other hand, are invoked after order book has been updated which makes it difficult to see what the order book state was when the trade occurred.

The issue can be fixed by unifying the order book event and trade event callbacks into one callback that updates the order book (new order or cancellation), reports a trade (non-displayable order was executed or trade happened on another venue), or both updates the order book and reports a trade (order execution).