bluematrix007 / hft_price_level_forecasting

high frequency forecasting of stock prices using LSTM, SVR, Random forests, KNN regressor and Kalman filter

IE 598 Project Assignment 1

Project Proposal

Abstract:

The scope of this project is to model and forecast prices levels generic to instruments such as equity, futures, options etc. Since, financial data tends to be noisy with different events impacting the price in a non deterministic way, it would be difficult to forecast the price long into the future (much like weather forecasting). Precisely for this reason, we shall restrict our scope to forecast short term prices that can be modeled with the help of either statistical tools or machine learning tools. In this project, we will try out different tools and also ensemble of these models to forecast the price levels. The subsequent sections focus on few of the important aspects of the study namely input data and sourcing, testing and comparing performance of different algorithms. Other logistics such as project effort distribution and timelines are included at the end

Input Data and Sourcing:

There are two kinds of inputs we are considering for this project:

• Market by Order(MBO) time series data: This data provides not only provides the price level information but also the full/partial depth of book. Therefore, the model can use this information to also judge if the instrument is oversold or high in demand and try to forcast the price level behaviour.
• Historical price level information:

Algorithms considered

A few algorithms that are being considered for this project are:

• Extended Kalman Filter: Kalman filter is a state estimation algorithm that is effectively used in control systems to predict the state (ex:trajectory) based on previous output of the system. The same algorithm can also be used in a discrete sense to predict price level curve based on previous price levels. Extended Kalman filter is an extension of this algorithm that can generalise data where the underlying dynamics is non linear.

• Fast Fourier Transform (FFT): FFT is another data modelling technique that is extensively used in engineering applications such as signal processing. This transformation technique models the price level curve as an combination of signals with different amplitudes and frequencies. Given historical data of any price level, it can ouput a model that can predict the future price level fairly accurately provided the model is not overfit to the noise in the data.

• Machine learning models: We shall be trying out several machine learning based models such as SVM regressor, KNN regressor, Decision Tree regressor and its ensembled counterparts such as Adaboost and RandomForest Regressors. These models can make use of market depth unlike the above models and hence MBO data would be the input here. Prior to fitting these models feature extraction (PCA) and engineering (adding percent changes) would be done to fit these models well with regard to time series data

• Import a HTML file and watch it magically convert to Markdown

• Drag and drop images (requires your Dropbox account be linked)

• Import and save files from GitHub, Dropbox, Google Drive and One Drive

• Drag and drop markdown and HTML files into Dillinger

• Export documents as Markdown, HTML and PDF

Markdown is a lightweight markup language based on the formatting conventions that people naturally use in email. As John Gruber writes on the Markdown site

The overriding design goal for Markdown's formatting syntax is to make it as readable as possible. The idea is that a Markdown-formatted document should be publishable as-is, as plain text, without looking like it's been marked up with tags or formatting instructions.

This text you see here is *actually- written in Markdown! To get a feel for Markdown's syntax, type some text into the left window and watch the results in the right.

Tech

Dillinger uses a number of open source projects to work properly:

• AngularJS - HTML enhanced for web apps!
• Ace Editor - awesome web-based text editor
• markdown-it - Markdown parser done right. Fast and easy to extend.
• Twitter Bootstrap - great UI boilerplate for modern web apps
• node.js - evented I/O for the backend
• Express - fast node.js network app framework @tjholowaychuk
• Gulp - the streaming build system
• Breakdance - HTML to Markdown converter
• jQuery - duh

And of course Dillinger itself is open source with a public repository on GitHub.

Installation

Dillinger requires Node.js v10+ to run.

Install the dependencies and devDependencies and start the server.

cd dillinger
npm i
node app

For production environments...

npm install --production
NODE_ENV=production node app

Plugins

Dillinger is currently extended with the following plugins. Instructions on how to use them in your own application are linked below.

Plugin	README
Dropbox	plugins/dropbox/README.md
GitHub	plugins/github/README.md
Google Drive	plugins/googledrive/README.md
OneDrive	plugins/onedrive/README.md
Medium	plugins/medium/README.md
Google Analytics	plugins/googleanalytics/README.md

Development

Want to contribute? Great!

Dillinger uses Gulp + Webpack for fast developing. Make a change in your file and instantaneously see your updates!

Open your favorite Terminal and run these commands.

First Tab:

node app

Second Tab:

gulp watch

(optional) Third:

karma test

Building for source

For production release:

gulp build --prod

Generating pre-built zip archives for distribution:

gulp build dist --prod

Docker

Dillinger is very easy to install and deploy in a Docker container.

By default, the Docker will expose port 8080, so change this within the Dockerfile if necessary. When ready, simply use the Dockerfile to build the image.

cd dillinger
docker build -t <youruser>/dillinger:${package.json.version} .

This will create the dillinger image and pull in the necessary dependencies. Be sure to swap out ${package.json.version} with the actual version of Dillinger.

Once done, run the Docker image and map the port to whatever you wish on your host. In this example, we simply map port 8000 of the host to port 8080 of the Docker (or whatever port was exposed in the Dockerfile):

docker run -d -p 8000:8080 --restart=always --cap-add=SYS_ADMIN --name=dillinger <youruser>/dillinger:${package.json.version}

Note: --capt-add=SYS-ADMIN is required for PDF rendering.

Verify the deployment by navigating to your server address in your preferred browser.

127.0.0.1:8000

License

MIT

Free Software, Hell Yeah!