quantconnect / research

Open sourced research notebooks by the QuantConnect team.

This repository is a collection of research notebooks and tutorials using the QuantConnect LEAN platform. Research covers a range of topics from tutorial focused demonstrations to topical analysis of modern movements in the financial markets.

Topical Events

• May 7, 2020 - S&P500 Hope vs Fear CV2019
• June 4, 2020 - Airline Bailout & Buybacks Research

Idea Streams PodCast

• May 28, 2020 Episode 5 - Tail Risk Hedging
• May 22, 2020 Episode 4 - Nowcasting News Announcements of Vaccine Trials
• May 10, 2020 Episode 3 - Correlation Analysis Major CV19 Economies
• April 28, 2020 Episode 2 - Unemployment Claims with SKLean
• April 20th, 2020 Episode 1 - Traunch Rebalancing

Research 2 Production Notebook Series

• Mean Reversion
• Random Forest Regression
• Uncorrelated Assets
• Kalman Filters and Pairs Trading
• Stationary Processes and Z-Scores
• Principal Component Analysis
• Hidden Markov Models
• Long Short-Term Memory

Analysis Examples

• Fudamental Factor Analysis: This research applies MorningStar fundamental data to demonstrate how to select the effective factors for long/short strategies.

• Kalman Filter Based Pairs Trading: This research demonstrates the basic principle of pairs trading and introduces the concepts of cointegration and Kalman Filter for pairs trading.

• Mean-Variance Portfolio Optimization: This research demonstrates the mean-variance approach to asset allocation in modern portfolio theory and shows how to find the efficient frontier.

• EMA Cross Strategy Based on VXX: This research demonstrates how to build a simple EMA cross strategy with Python and how to get the performance statistics of the strategy.

• Pairs Trading Strategy Based on Cointegration: This research goes through the development process step-by-step of a pairs trading strategy and shows how to backtest the strategy.

research's Issues

Enable Weekly Options Access

Both of the code snippets below should return weekly options data, but instead, they return a NullReferenceException. This issue stems for this forum thread: https://www.quantconnect.com/forum/discussion/4552/weekly-options-in-notebook/p1

class MyQuantBook(QuantBook):
def UniverseFunc(self, universe):
return universe.IncludeWeeklys().Strikes(-2, 2).Expiration(timedelta(0), timedelta(365*2))
qb = MyQuantBook()
spy = qb.AddOption("GOOG")
spy.SetFilter(qb.UniverseFunc)

qb = QuantBook()

def UniverseFunc(self, universe):
        return universe.IncludeWeeklys().Strikes(-8, 8).Expiration(timedelta(25), timedelta(35))
qb.AddEquity("SPY", Resolution.Daily)
spy = qb.AddOption("SPY", Resolution.Daily)
spy.SetFilter(UniverseFunc)

GetOptionHistory fails to work with option created by Symbol.CreateOption()

Currently, the functionality of GetOptionHistory() works only with universes, not single Options which can be created by Symbol.CreateOption(). The code below, run in the QC research environment, can recreate the issue

#amzn = qb.AddEquity("AMZN")
amzn = qb.AddOption('AMZN')
amzn_specific = Symbol.CreateOption('AMZN',Market.USA,OptionStyle.American,OptionRight.Call,1665,datetime(2018,12,21))
amzn.SetFilter(-2, 2, timedelta(0), timedelta(180))
print(amzn_specific)
history = qb.GetOptionHistory(amzn_specific, datetime(2018,11,29))

Custom Indicator support

Enable support for custom indicators, similar to how one would be created for a QC Algorithm

class CustomSimpleMovingAverage:
    def __init__(self, name, period):
        self.Name = name
        self.Time = datetime.min
        self.Value = 0
        self.IsReady = False
        self.queue = deque(maxlen=period)

    def __repr__(self):
        return "{0} -> IsReady: {1}. Time: {2}. Value: {3}".format(self.Name, self.IsReady, self.Time, self.Value)

    # Update method is mandatory
    def Update(self, input):
        self.queue.appendleft(input.Close)
        count = len(self.queue)
        self.Time = input.EndTime
        self.Value = sum(self.queue) / count
        self.IsReady = count == self.queue.maxlen

custom = CustomSimpleMovingAverage('SPY', 21)
x = qb.Indicator(CustomSimpleMovingAverage, 'SPY', 360, Resolution.Daily)

Add Stock Picking With Fundamental Data Notebook

Add Visualizing Data With Seaborn Notebook

Foundation update for matplotlib.finance

matplotlib.finance module isn't found when imported to the Research Environment. An update is needed so that the current module is available in the RE.

QuantBook GetOptionHistory() fails to return proper data defined by date range

Expected Behavior

Users should be able to access a section of options data in the research environment defined by using the startDate and endDate arguments in GetOptionHistory().

Actual Behavior

GetOptionHistory() returns a minimum of 24 hours of data even if the range specified by startDate and endDate is less than 24 hours, and fails to return the full data in a specified date range.

Potential Solution

Uncertain.

Reproducing the Problem

In research environment, the issue of 24-hour minimum of data can be reproduced by runnings the lines

option = qb.AddOption("SPY", Resolution.Minute)
option.SetFilter(-2, 2, timedelta(0), timedelta(20))
option_history = qb.GetOptionHistory(option.Symbol, datetime(2017, 1, 4, 10 ,0, 0), datetime(2017, 1, 4, 10 ,5, 0),sort = True)
df = option_history.GetAllData()
print(df.tail())

The second issue can be reproduced by running the lines

option = qb.AddOption("SPY", Resolution.Minute)
option.SetFilter(-2, 2, timedelta(0), timedelta(20))
option_history = qb.GetOptionHistory(option.Symbol, datetime(2017, 1, 4, 10 ,0, 0), datetime(2017, 1, 7, 12 ,5, 0),sort = True)
df = option_history.GetAllData()
print(df.tail())

Checklist

• I have completely filled out this template
• I have confirmed that this issue exists on the current master branch
• I have confirmed that this is not a duplicate issue by searching issues
• I have provided detailed steps to reproduce the issue

Upgrade Plotly to 4.0.0 version

Currently, using the plotly.graph_objs package throws an error AttributeError: 'Figure' object has no attribute 'show' when trying to plot candlestick chart using the following code block.

import plotly.graph_objs as go
fig = go.Figure(data=[go.Candlestick(x=h1['open'].index,
                                    open = h1['open'],
                                    high = h1['high'],
                                    low = h1['low'],
                                    close = h1['close'])])
fig.show()

A possible solution is to upgrade plotly package to 4.0.0 version in RE as suggested on stackoverflow

Futures History GetAllData() Crashes Kernel with R1-4 node

With a R1-4 node executed the following code in a research.ipynb file and the kernel crashed.

import calendar

qb = QuantBook()
es = qb.AddFuture(Futures.Indices.SP500EMini)
es.SetFilter(timedelta(-360), timedelta(360))

Year to Date

start_time = datetime(date.today().year, 1, 1, 0, 0)
end_time = datetime(date.today().year, 6, 30, 23, 59)
future_history = qb.GetFutureHistory(es.Symbol, start_time, end_time)

Get all futures data as a dataframe

history = future_history.GetAllData()

Is there a better method to access and query a year of futures contracts data that does not crash the kernel?

Are you using docker or not? How to run this stuff in local?

Found a docker build. Will create a Makefile and PR it if it is helpful.

History calls crash kernel

after subscribing, I call this code.

qb.History(["EURUSD"], datetime(2015,1,1), datetime.now(), Resolution.Daily)
It runs no problem. But if I run it 5 times, it crashes the kernel on the 5th time.

The error seems to be : Fatal Python error: deallocating None

Fatal Python error: deallocating None

Thread 0x000064dc (most recent call first):
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\parentpoller.py", line 97 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 916 in _bootstrap_inner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 884 in _bootstrap

Thread 0x00005184 (most recent call first):
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 295 in wait
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 551 in wait
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\history.py", line 829 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\history.py", line 58 in needs_sqlite
File "", line 2 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 916 in _bootstrap_inner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 884 in _bootstrap

Thread 0x00007274 (most recent call first):
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\heartbeat.py", line 61 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 916 in _bootstrap_inner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 884 in _bootstrap

Thread 0x00005acc (most recent call first):
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\selectors.py", line 314 in _select
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\selectors.py", line 323 in select
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\asyncio\base_events.py", line 1398 in _run_once
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\asyncio\base_events.py", line 422 in run_forever
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\platform\asyncio.py", line 132 in start
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\iostream.py", line 78 in _thread_main
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 864 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 916 in _bootstrap_inner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\threading.py", line 884 in _bootstrap

Current thread 0x00005c80 (most recent call first):
File "", line 1 in
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\interactiveshell.py", line 3267 in run_code
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\interactiveshell.py", line 3191 in run_ast_nodes
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\interactiveshell.py", line 3020 in run_cell_async
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\async_helpers.py", line 67 in _pseudo_sync_runner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\interactiveshell.py", line 2845 in _run_cell
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\IPython\core\interactiveshell.py", line 2819 in run_cell
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\zmqshell.py", line 536 in run_cell
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\ipkernel.py", line 294 in do_execute
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\gen.py", line 326 in wrapper
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\kernelbase.py", line 534 in execute_request
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\gen.py", line 326 in wrapper
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\kernelbase.py", line 267 in dispatch_shell
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\gen.py", line 326 in wrapper
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\kernelbase.py", line 357 in process_one
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\gen.py", line 1147 in run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\gen.py", line 1233 in inner
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\stack_context.py", line 300 in null_wrapper
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\ioloop.py", line 758 in _run_callback
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\asyncio\events.py", line 145 in _run
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\asyncio\base_events.py", line 1434 in _run_once
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\asyncio\base_events.py", line 422 in run_forever
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\tornado\platform\asyncio.py", line 132 in start
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel\kernelapp.py", line 505 in start
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\traitlets\config\application.py", line 658 in launch_instance
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\site-packages\ipykernel_launcher.py", line 16 in
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\runpy.py", line 85 in _run_code
File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\runpy.py", line 193 in _run_module_as_main
[I 12:00:05.236 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports

Code proving history is causing the crash:

%matplotlib inline
# Imports
from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from math import ceil, floor
# Create an instance
qb = QuantBook()
plt.style.use('seaborn-whitegrid')
qb.AddForex("EURUSD")
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)

Separate issue which may be the same cause: This case will also cause a crash

qb = QuantBook()
qb.AddForex("EURUSD")
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)
qb = QuantBook()
qb.AddForex("EURUSD")
qb.History(["EURUSD"], datetime(2010,1,1), datetime.now(), Resolution.Daily)

Add tslearn and dtw Python modules

Add the tslearn v0.3.0 Python module. It supports time-series analysis and machine learning methods. Same with dtw-python v1.0.4

Port Quantopian Lectures Series

Port the Quantopian Lecture Series to the QC/ API/

Add TensorFlow Price Prediction Notebook

How to retreive historical Option Data in C#

Hello!

I am trying to understand how I get retreive historical option chain data and how found this code in one example.

But this C# code does not compile. I am trying to use this code in the terminal here:
https://www.quantconnect.com/terminal/#

option = qb.AddOption("SPY", Resolution.Minute)
option.SetFilter(-2, 2, timedelta(0), timedelta(20))
option_history = qb.GetOptionHistory(option.Symbol, datetime(2017, 1, 4, 10 ,0, 0), datetime(2017, 1, 4, 10 ,5, 0),sort = True)
df = option_history.GetAllData()
print(df.tail())

Is it possible to know how to write a code that is complete including the correct classes etc that prints out the requested optionHistory Data. That would be so greatful as I can't find such example?

Kind Regards

Create Predicting Future Prices With Sklearn Notebook

Create predicting future prices with sklearn notebook following the pattern from Keras and Tensorflow.

Notebook disconnects

When using https://www.quantconnect.com/research after a period of time (20 minutes?) the notebook disconnects and I have to refresh the page to reconnect.

This message is shown: "A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration"

So maybe this is related: jupyter/docker-stacks#199

Cannot Access the Link in README

All of them got 404 page not found.

Some longer-range ValuationRatios class members returning 0

Expected Behavior

A call such as

qb = QuantBook()
begin = datetime(2009,2,27)
end = datetime.now()
stocks = ["AAPL","MSFT","ADBE","IBM","BRK.B"]
data = qb.GetFundamental(stocks, 'ValuationRatios.EVToEBIT3YrAvg' ,begin,end).fillna(method='ffill')
df = data.apply(pd.to_numeric)

should return a data frame of values. This problem exists for a number of ValuationRatios class members, the full extent of which is unknown.

Actual Behavior

The above call returns the dataframe

            AAPL R735QTJ8XC9X  MSFT R735QTJ8XC9X  ADBE R735QTJ8XC9X  \
2009-02-27                0.0                0.0                0.0   
2009-02-28                0.0                0.0                0.0   
2009-03-02                0.0                0.0                0.0   
2009-03-31                0.0                0.0                0.0   
2009-04-03                0.0                0.0                0.0   

            IBM R735QTJ8XC9X  BRKB R735QTJ8XC9X  
2009-02-27               0.0                0.0  
2009-02-28               0.0                0.0  
2009-03-02               0.0                0.0  
2009-03-31               0.0                0.0  
2009-04-03               0.0                0.0

where all entries are 0

Potential Solution

Reproducing the Problem

Code snippet above can reproduce the problem, or see this notebook: https://www.quantconnect.com/research/8d474cf62e3f2fd72bf8891abafa7f0a

Checklist

• I have completely filled out this template
• I have confirmed that this issue exists on the current master branch
• I have confirmed that this is not a duplicate issue by searching issues
• I have provided detailed steps to reproduce the issue

How to extract Implied volatility from Quantconnect

I am trying to create a trading strategy that uses Implied Volatility from options to buy or sell stocks. How can I get the implied volatility data in the research env?

QuantBook returns incorrect timestamps

QuantBook returns timestamps for equities data that don't align with proper market dates. Running the code below in the QC research environment will reproduce the problem.

%matplotlib inline
# Imports
from clr import AddReference
AddReference("System")
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Custom import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd

qb = QuantBook()

spy = qb.AddEquity("SPY")
h1 = qb.History(qb.Securities.Keys, 10000, Resolution.Minute)
df_spy = pd.DataFrame(h1.loc["SPY"]["close"])
print(df_spy[:60])

The returned data is

                      close
time                       
2018-12-18 15:56:00  255.96
2018-12-18 15:57:00  255.54
2018-12-18 15:58:00  255.45
2018-12-18 15:59:00  255.34
2018-12-18 16:00:00  255.14
2018-12-19 09:31:00  255.21
2018-12-19 09:32:00  255.37
2018-12-19 09:33:00  255.63
2018-12-19 09:34:00  255.68
2018-12-19 09:35:00  255.78

12/18 is a Friday but 12/19 is a Saturday. This issue does not occur when printing 12/28 and 12/31, which are a Friday and Monday, respectively.

print(df_spy[2585:2595])

                       close
time                        
2018-12-28 15:56:00  247.610
2018-12-28 15:57:00  247.460
2018-12-28 15:58:00  247.375
2018-12-28 15:59:00  247.690
2018-12-28 16:00:00  247.810
2018-12-31 09:31:00  249.500
2018-12-31 09:32:00  249.510
2018-12-31 09:33:00  249.560
2018-12-31 09:34:00  249.810
2018-12-31 09:35:00  250.030