microsoft / qlib

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.

📰 What's NEW!   💖

Recent released features

Feature	Status
KRNN and Sandwich models	📈 Released on May 26, 2023
Release Qlib v0.9.0	Released on Dec 9, 2022
RL Learning Framework	🔨 📈 Released on Nov 10, 2022. #1332, #1322, #1316,#1299,#1263, #1244, #1169, #1125, #1076
HIST and IGMTF models	📈 Released on Apr 10, 2022
Qlib notebook tutorial	📖 Released on Apr 7, 2022
Ibovespa index data	🍚 Released on Apr 6, 2022
Point-in-Time database	🔨 Released on Mar 10, 2022
Arctic Provider Backend & Orderbook data example	🔨 Released on Jan 17, 2022
Meta-Learning-based framework & DDG-DA	📈 🔨 Released on Jan 10, 2022
Planning-based portfolio optimization	🔨 Released on Dec 28, 2021
Release Qlib v0.8.0	Released on Dec 8, 2021
ADD model	📈 Released on Nov 22, 2021
ADARNN model	📈 Released on Nov 14, 2021
TCN model	📈 Released on Nov 4, 2021
Nested Decision Framework	🔨 Released on Oct 1, 2021. Example and Doc
Temporal Routing Adaptor (TRA)	📈 Released on July 30, 2021
Transformer & Localformer	📈 Released on July 22, 2021
Release Qlib v0.7.0	Released on July 12, 2021
TCTS Model	📈 Released on July 1, 2021
Online serving and automatic model rolling	🔨 Released on May 17, 2021
DoubleEnsemble Model	📈 Released on Mar 2, 2021
High-frequency data processing example	🔨 Released on Feb 5, 2021
High-frequency trading example	📈 Part of code released on Jan 28, 2021
High-frequency data(1min)	🍚 Released on Jan 27, 2021
Tabnet Model	📈 Released on Jan 22, 2021

Features released before 2021 are not listed here.

Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning.

An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies.

It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution. For more details, please refer to our paper "Qlib: An AI-oriented Quantitative Investment Platform".

Frameworks, Tutorial, Data & DevOps	Main Challenges & Solutions in Quant Research
Plans Framework of Qlib Quick Start Installation Data Preparation Auto Quant Research Workflow Building Customized Quant Research Workflow by Code Quant Dataset Zoo Learning Framework More About Qlib Offline Mode and Online Mode Performance of Qlib Data Server Related Reports Contact Us Contributing	Main Challenges & Solutions in Quant Research Forecasting: Finding Valuable Signals/Patterns Quant Model (Paper) Zoo Run a Single Model Run Multiple Models Adapting to Market Dynamics Reinforcement Learning: modeling continuous decisions

Plans

New features under development(order by estimated release time). Your feedbacks about the features are very important.

Framework of Qlib

The high-level framework of Qlib can be found above(users can find the detailed framework of Qlib's design when getting into nitty gritty). The components are designed as loose-coupled modules, and each component could be used stand-alone.

Qlib provides a strong infrastructure to support Quant research. Data is always an important part. A strong learning framework is designed to support diverse learning paradigms (e.g. reinforcement learning, supervised learning) and patterns at different levels(e.g. market dynamic modeling). By modeling the market, trading strategies will generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can be nested to be optimized and run together. At last, a comprehensive analysis will be provided and the model can be served online in a low cost.

Quick Start

This quick start guide tries to demonstrate

1. It's very easy to build a complete Quant research workflow and try your ideas with Qlib.
2. Though with public data and simple models, machine learning technologies work very well in practical Quant investment.

Here is a quick demo shows how to install Qlib, and run LightGBM with qrun. But, please make sure you have already prepared the data following the instruction.

Installation

This table demonstrates the supported Python version of Qlib:

	install with pip	install from source	plot
Python 3.7	✔️	✔️	✔️
Python 3.8	✔️	✔️	✔️
Python 3.9	❌	✔️	❌

Note:

1. Conda is suggested for managing your Python environment. In some cases, using Python outside of a conda environment may result in missing header files, causing the installation failure of certain packages.
2. Please pay attention that installing cython in Python 3.6 will raise some error when installing Qlib from source. If users use Python 3.6 on their machines, it is recommended to upgrade Python to version 3.7 or use conda's Python to install Qlib from source.
3. For Python 3.9, Qlib supports running workflows such as training models, doing backtest and plot most of the related figures (those included in notebook). However, plotting for the model performance is not supported for now and we will fix this when the dependent packages are upgraded in the future.
4. QlibRequires tables package, hdf5 in tables does not support python3.9.

Install with pip

Users can easily install Qlib by pip according to the following command.

  pip install pyqlib

Note: pip will install the latest stable qlib. However, the main branch of qlib is in active development. If you want to test the latest scripts or functions in the main branch. Please install qlib with the methods below.

Install from source

Also, users can install the latest dev version Qlib by the source code according to the following steps:

• Before installing Qlib from source, users need to install some dependencies:

  pip install numpy
  pip install --upgrade  cython

• Clone the repository and install Qlib as follows.

  git clone https://github.com/microsoft/qlib.git && cd qlib
  pip install .

  Note: You can install Qlib with python setup.py install as well. But it is not the recommended approach. It will skip pip and cause obscure problems. For example, only the command pip install . can overwrite the stable version installed by pip install pyqlib, while the command python setup.py install can't.

Tips: If you fail to install Qlib or run the examples in your environment, comparing your steps and the CI workflow may help you find the problem.

Tips for Mac: If you are using Mac with M1, you might encounter issues in building the wheel for LightGBM, which is due to missing dependencies from OpenMP. To solve the problem, install openmp first with brew install libomp and then run pip install . to build it successfully.

Data Preparation

Load and prepare data by running the following code:

Get with module

# get 1d data
python -m qlib.run.get_data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

# get 1min data
python -m qlib.run.get_data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

Get from source

# get 1d data
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

# get 1min data
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

This dataset is created by public data collected by crawler scripts, which have been released in the same repository. Users could create the same dataset with it. Description of dataset

Please pay ATTENTION that the data is collected from Yahoo Finance, and the data might not be perfect. We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to the related document.

Automatic update of daily frequency data (from yahoo finance)

This step is Optional if users only want to try their models and strategies on history data.

It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically.

NOTE: Users can't incrementally update data based on the offline data provided by Qlib(some fields are removed to reduce the data size). Users should use yahoo collector to download Yahoo data from scratch and then incrementally update it.

For more information, please refer to: yahoo collector

• Automatic update of data to the "qlib" directory each trading day(Linux)

  • use crontab: crontab -e

  • set up timed tasks:

    * * * * 1-5 python <script path> update_data_to_bin --qlib_data_1d_dir <user data dir>
    

    • script path: scripts/data_collector/yahoo/collector.py

• Manual update of data

  python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date>
  

  • trading_date: start of trading day
  • end_date: end of trading day(not included)

Auto Quant Research Workflow

Qlib provides a tool named qrun to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). You can start an auto quant research workflow and have a graphical reports analysis according to the following steps:

1. Quant Research Workflow: Run qrun with lightgbm workflow config (workflow_config_lightgbm_Alpha158.yaml as following.

     cd examples  # Avoid running program under the directory contains `qlib`
     qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

   If users want to use qrun under debug mode, please use the following command:

   python -m pdb qlib/workflow/cli.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml

   The result of qrun is as follows, please refer to Intraday Trading for more details about the result.

   'The following are analysis results of the excess return without cost.'
                          risk
   mean               0.000708
   std                0.005626
   annualized_return  0.178316
   information_ratio  1.996555
   max_drawdown      -0.081806
   'The following are analysis results of the excess return with cost.'
                          risk
   mean               0.000512
   std                0.005626
   annualized_return  0.128982
   information_ratio  1.444287
   max_drawdown      -0.091078

   Here are detailed documents for qrun and workflow.

2. Graphical Reports Analysis: Run examples/workflow_by_code.ipynb with jupyter notebook to get graphical reports

   • Forecasting signal (model prediction) analysis

     • Cumulative Return of groups
     • Return distribution
     • Information Coefficient (IC)
     • Auto Correlation of forecasting signal (model prediction)

   • Portfolio analysis

     • Backtest return

   • Explanation of above results

Building Customized Quant Research Workflow by Code

The automatic workflow may not suit the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. Here is a demo for customized Quant research workflow by code.

Main Challenges & Solutions in Quant Research

Quant investment is a very unique scenario with lots of key challenges to be solved. Currently, Qlib provides some solutions for several of them.

Forecasting: Finding Valuable Signals/Patterns

Accurate forecasting of the stock price trend is a very important part to construct profitable portfolios. However, huge amount of data with various formats in the financial market which make it challenging to build forecasting models.

An increasing number of SOTA Quant research works/papers, which focus on building forecasting models to mine valuable signals/patterns in complex financial data, are released in Qlib

Quant Model (Paper) Zoo

Here is a list of models built on Qlib.

• GBDT based on XGBoost (Tianqi Chen, et al. KDD 2016)
• GBDT based on LightGBM (Guolin Ke, et al. NIPS 2017)
• GBDT based on Catboost (Liudmila Prokhorenkova, et al. NIPS 2018)
• MLP based on pytorch
• LSTM based on pytorch (Sepp Hochreiter, et al. Neural computation 1997)
• GRU based on pytorch (Kyunghyun Cho, et al. 2014)
• ALSTM based on pytorch (Yao Qin, et al. IJCAI 2017)
• GATs based on pytorch (Petar Velickovic, et al. 2017)
• SFM based on pytorch (Liheng Zhang, et al. KDD 2017)
• TFT based on tensorflow (Bryan Lim, et al. International Journal of Forecasting 2019)
• TabNet based on pytorch (Sercan O. Arik, et al. AAAI 2019)
• DoubleEnsemble based on LightGBM (Chuheng Zhang, et al. ICDM 2020)
• TCTS based on pytorch (Xueqing Wu, et al. ICML 2021)
• Transformer based on pytorch (Ashish Vaswani, et al. NeurIPS 2017)
• Localformer based on pytorch (Juyong Jiang, et al.)
• TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)
• TCN based on pytorch (Shaojie Bai, et al. 2018)
• ADARNN based on pytorch (YunTao Du, et al. 2021)
• ADD based on pytorch (Hongshun Tang, et al.2020)
• IGMTF based on pytorch (Wentao Xu, et al.2021)
• HIST based on pytorch (Wentao Xu, et al.2021)
• KRNN based on pytorch
• Sandwich based on pytorch

Your PR of new Quant models is highly welcomed.

The performance of each model on the Alpha158 and Alpha360 datasets can be found here.

Run a single model

All the models listed above are runnable with Qlib. Users can find the config files we provide and some details about the model through the benchmarks folder. More information can be retrieved at the model files listed above.

Qlib provides three different ways to run a single model, users can pick the one that fits their cases best:

• Users can use the tool qrun mentioned above to run a model's workflow based from a config file.

• Users can create a workflow_by_code python script based on the one listed in the examples folder.

• Users can use the script run_all_model.py listed in the examples folder to run a model. Here is an example of the specific shell command to be used: python run_all_model.py run --models=lightgbm, where the --models arguments can take any number of models listed above(the available models can be found in benchmarks). For more use cases, please refer to the file's docstrings.

  • NOTE: Each baseline has different environment dependencies, please make sure that your python version aligns with the requirements(e.g. TFT only supports Python 3.6~3.7 due to the limitation of tensorflow==1.15.0)

Run multiple models

Qlib also provides a script run_all_model.py which can run multiple models for several iterations. (Note: the script only support Linux for now. Other OS will be supported in the future. Besides, it doesn't support parallel running the same model for multiple times as well, and this will be fixed in the future development too.)

The script will create a unique virtual environment for each model, and delete the environments after training. Thus, only experiment results such as IC and backtest results will be generated and stored.

Here is an example of running all the models for 10 iterations:

python run_all_model.py run 10

It also provides the API to run specific models at once. For more use cases, please refer to the file's docstrings.

Adapting to Market Dynamics

Due to the non-stationary nature of the environment of the financial market, the data distribution may change in different periods, which makes the performance of models build on training data decays in the future test data. So adapting the forecasting models/strategies to market dynamics is very important to the model/strategies' performance.

Here is a list of solutions built on Qlib.

• Rolling Retraining
• DDG-DA on pytorch (Wendi, et al. AAAI 2022)

Reinforcement Learning: modeling continuous decisions

Qlib now supports reinforcement learning, a feature designed to model continuous investment decisions. This functionality assists investors in optimizing their trading strategies by learning from interactions with the environment to maximize some notion of cumulative reward.

Here is a list of solutions built on Qlib categorized by scenarios.

RL for order execution

Here is the introduction of this scenario. All the methods below are compared here.

• TWAP
• PPO: "An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization", IJCAL 2020
• OPDS: "Universal Trading for Order Execution with Oracle Policy Distillation", AAAI 2021

Quant Dataset Zoo

Dataset plays a very important role in Quant. Here is a list of the datasets built on Qlib:

Dataset	US Market	China Market
Alpha360	√	√
Alpha158	√	√

Here is a tutorial to build dataset with Qlib. Your PR to build new Quant dataset is highly welcomed.

Learning Framework

Qlib is high customizable and a lot of its components are learnable. The learnable components are instances of Forecast Model and Trading Agent. They are learned based on the Learning Framework layer and then applied to multiple scenarios in Workflow layer. The learning framework leverages the Workflow layer as well(e.g. sharing Information Extractor, creating environments based on Execution Env).

Based on learning paradigms, they can be categorized into reinforcement learning and supervised learning.

• For supervised learning, the detailed docs can be found here.
• For reinforcement learning, the detailed docs can be found here. Qlib's RL learning framework leverages Execution Env in Workflow layer to create environments. It's worth noting that NestedExecutor is supported as well. This empowers users to optimize different level of strategies/models/agents together (e.g. optimizing an order execution strategy for a specific portfolio management strategy).

More About Qlib

If you want to have a quick glance at the most frequently used components of qlib, you can try notebooks here.

The detailed documents are organized in docs. Sphinx and the readthedocs theme is required to build the documentation in html formats.

cd docs/
conda install sphinx sphinx_rtd_theme -y
# Otherwise, you can install them with pip
# pip install sphinx sphinx_rtd_theme
make html

You can also view the latest document online directly.

Qlib is in active and continuing development. Our plan is in the roadmap, which is managed as a github project.

Offline Mode and Online Mode

The data server of Qlib can either deployed as Offline mode or Online mode. The default mode is offline mode.

Under Offline mode, the data will be deployed locally.

Under Online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found in Qlib-Server. The online mode can be deployed automatically with Azure CLI based scripts. The source code of online data server can be found in Qlib-Server repository.

Performance of Qlib Data Server

The performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib data server, we compare it with several other data storage solutions.

We evaluate the performance of several storage solutions by finishing the same task, which creates a dataset (14 features/factors) from the basic OHLCV daily data of a stock market (800 stocks each day from 2007 to 2020). The task involves data queries and processing.

	HDF5	MySQL	MongoDB	InfluxDB	Qlib -E -D	Qlib +E -D	Qlib +E +D
Total (1CPU) (seconds)	184.4±3.7	365.3±7.5	253.6±6.7	368.2±3.6	147.0±8.8	47.6±1.0	7.4±0.3
Total (64CPU) (seconds)					8.8±0.6	4.2±0.2

• +(-)E indicates with (out) ExpressionCache
• +(-)D indicates with (out) DatasetCache

Most general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions. Such overheads greatly slow down the data loading process. Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.

Related Reports

• Guide To Qlib: Microsoft’s AI Investment Platform
• 微软也搞AI量化平台？还是开源的！
• 微矿Qlib：业内首个AI量化投资开源平台

Contact Us

• If you have any issues, please create issue here or send messages in gitter.
• If you want to make contributions to Qlib, please create pull requests.
• For other reasons, you are welcome to contact us by email([email protected]).
  • We are recruiting new members(both FTEs and interns), your resumes are welcome!

Join IM discussion groups:

Gitter

Contributing

We appreciate all contributions and thank all the contributors!

Before we released Qlib as an open-source project on Github in Sep 2020, Qlib is an internal project in our group. Unfortunately, the internal commit history is not kept. A lot of members in our group have also contributed a lot to Qlib, which includes Ruihua Wang, Yinda Zhang, Haisu Yu, Shuyu Wang, Bochen Pang, and Dong Zhou. Especially thanks to Dong Zhou due to his initial version of Qlib.

Guidance

This project welcomes contributions and suggestions.
Here are some code standards and development guidance for submiting a pull request.

Making contributions is not a hard thing. Solving an issue(maybe just answering a question raised in issues list or gitter), fixing/issuing a bug, improving the documents and even fixing a typo are important contributions to Qlib.

For example, if you want to contribute to Qlib's document/code, you can follow the steps in the figure below.

If you don't know how to start to contribute, you can refer to the following examples.

Type	Examples
Solving issues	Answer a question; issuing or fixing a bug
Docs	Improve docs quality ; Fix a typo
Feature	Implement a requested feature like this; Refactor interfaces
Dataset	Add a dataset
Models	Implement a new model, some instructions to contribute models

Good first issues are labelled to indicate that they are easy to start your contributions.

You can find some impefect implementation in Qlib by rg 'TODO|FIXME' qlib

If you would like to become one of Qlib's maintainers to contribute more (e.g. help merge PR, triage issues), please contact us by email([email protected]). We are glad to help to upgrade your permission.

Licence

Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the right to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

qlib's Issues

Design analysis framework

Currently, we provide some functions to help analyse the model predictions and trading strategies.

In the future, we will design a dedicated framework for analysing and reorganize these functions.

ValueError: instruments not exists for market csi500

instruments = D.instruments(market='csi500')
D.list_instruments(instruments=instruments, start_time='2015-01-01', end_time='2016-02-15', as_list=True)

xxx/qlib/data/data.py in _load_instruments(self, market)
    574         fname = self._uri_inst.format(market)
    575         if not os.path.exists(fname):
--> 576             raise ValueError("instruments not exists for market " + market)
    577         _instruments = dict()
    578         with open(fname) as f:

ValueError: instruments not exists for market csi500

---

xxx/scripts/data_collector/csi/collector.py
No CSI500 and other market

How to manually add qrun to PATH

❓ Questions and Help

After installing Qlib, the qrun command is still not found. How can we manually add the qrun path to PATH? Thanks.

video tutorials？

Qlib is so powerful.
Will you provide video tutorials or more detailed Chinese courses?I think it may help you focus more on the key issues.

My visuals are on the fritz my name is Shawn Daniel McLean

Improve the performance of Loading Exchange

To simulating order execution for backtesting.
Qlib have to load quote data and create the Exchange.
Currently such an object is slow because all the quote data are load

Tick by tick marketdata

Could be possible a intraday tick by tick marketdata?

How to run "estimator -c configuration.yaml" on Ubuntu ?

It works om WinOS.
But I do not know where is estimator, and how to run "estimator -c configuration.yaml" on Ubuntu.

Thanks

On Jupyter,

! cd ./examples
! estimator -c estimator/estimator_config.yaml

---

/bin/sh: 1: estimator: not found

ValueError: instruments not exists for market <ticker> using my converted csv data into Qlib format

❓ Questions and Help

Hi,

I've converted my data for a particular US stock into Qlib format using command from doc python scripts/dump_bin.py dump_all --csv_path ~/.qlib/csv_data/my_data --qlib_dir ~/.qlib/qlib_data/my_data --include_fields open,close,high,low,volume,factor. Should a .txt file be created in ~/.qlib/qlib_data/my_data/instruments? I only see all.txt but the context inside the all.txt changes each time I add another ticker.

I think this is causing an issue for me to run the "train mode" example you provided with my data because I get a ValueError: instruments not exists for market AAPL for example.

Thanks.

disk cache generate failure when load data from disk

776 
777         # Now, it is time to initialize the File extension

--> 778 self._g_new(filename, mode, **params)
779
780 # Check filters and set PyTables format version for new files.

tables/hdf5extension.pyx in tables.hdf5extension.File._g_new()

HDF5ExtError: HDF5 error back trace

File "H5F.c", line 444, in H5Fcreate
unable to create file
File "H5Fint.c", line 1531, in H5F_open
unable to truncate a file which is already open

End of HDF5 error back trace

Unable to open/create file xxxxx qlib_data/cn_data/dataset_cache/pytables-k9tea7t3.tmp

How to use estimator_config_dnn.yaml without GPU ?

If there is not GPU,

MODEL_CONFIG = {
                "loss": "mse",
                'input_dim': 158,
                'output_dim': 1,
                'lr': 0.002,
                'lr_decay': 0.96,
                'lr_decay_steps': 100,
                'optimizer': 'adam',
                'max_steps': 8000,
                'batch_size': 4096,
                # 'GPU': '0',
                }

     73 def _check_driver():
     74     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
---> 75         raise AssertionError("Torch not compiled with CUDA enabled")
     76     if not torch._C._cuda_isDriverSufficient():
     77         if torch._C._cuda_getDriverVersion() == 0:

AssertionError: Torch not compiled with CUDA enabled

---

MODEL_CONFIG = {
                "loss": "mse",
                'input_dim': 158,
                'output_dim': 1,
                'lr': 0.002,
                'lr_decay': 0.96,
                'lr_decay_steps': 100,
                'optimizer': 'adam',
                'max_steps': 8000,
                'batch_size': 4096,
                'GPU': '0',
                }

     73 def _check_driver():
     74     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
---> 75         raise AssertionError("Torch not compiled with CUDA enabled")
     76     if not torch._C._cuda_isDriverSufficient():
     77         if torch._C._cuda_getDriverVersion() == 0:

AssertionError: Torch not compiled with CUDA enabled

数据频率最低就是日吗？自己的分钟数据，可以用qlib吗？

❓ Questions and Help

我看帮助文档freq周期至少是日，对于分钟，五分钟这些数据，能使用吗？谢谢
We sincerely suggest you to carefully read the documentation of our library as well as the official paper. After that, if you still feel puzzled, please describe the question clearly under this issue.

Supporting python 3.6 & 3.9 in CI

🐛 Bug Description

To Reproduce

Install Qlib under 3.6 & 3.9 in CI

Expected Behavior

The CI fails.

Screenshot

Environment

Note: One could run python scripts/collect_info.py under the qlib directory to get the following information.

• Qlib version:
• Python version:
• OS (Windows, Linux, MacOS):
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

AttributeError: 'Redis' object has no attribute 'client'

[9268:MainThread](2020-12-12 09:48:29,096) INFO - qlib.Initialization - [init.py:42] - default_conf: client.
Traceback (most recent call last):
File "D:\python\lib\site-packages\pyqlib-0.6.0.dev0-py3.7-win-amd64.egg\qlib\utils_init_.py", line 604, in can_use_cache
r.client()
AttributeError: 'Redis' object has no attribute 'client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/code/QuantInvest/invest-core/src/main/python/invest/qlib_apply/qlib_test.py", line 85, in
qlib.init()
File "D:\python\lib\site-packages\pyqlib-0.6.0.dev0-py3.7-win-amd64.egg\qlib_init_.py", line 56, in init
if not can_use_cache():
File "D:\python\lib\site-packages\pyqlib-0.6.0.dev0-py3.7-win-amd64.egg\qlib\utils_init_.py", line 608, in can_use_cache
r.close()
AttributeError: 'Redis' object has no attribute 'close'

No trading operations on custom dataset

❓ Questions and Help

Hi,
I was trying the qlib backtest experiment on my custom dataset, which was formerly built from SPY. And the data generation part seemed ok.

When it came to backtesting, I fed the input dataset and the benchmark data as:


Then the model training process was perfectly finished, and the parameters are following:

But for the prediction step, there was no trading operations reported, compared to the default example scripts.

Is it a problem from my dataset? For the factor column in the .csv inputs, Im somewhat confused about its meaning. In your paper, 'factor' is regarded as features. Should I comprehend the 'factor' as the alphas in your scripts such as Alpha158?

Merge the automatic paramater tuning feature.

Could you provide the dataset used to benchmark various storage solutions in the paper?

❓ Questions and Help

Could you provide the dataset used to benchmark various storage solutions in the paper?
We would like to further benchmark other potential solutions such as partitioned parquet, tiledb, duckdb, dolphindb etc.

did you open source for all code?

i found estimator is just a binary executable file, does this open sourced?

Why the dataset is normalized by the closed price of the first day?

❓ Questions and Help

By using the data download by the get_data.py and using the code from data retrieval tutorial, the price in the dataframe is normalized by the closed price of the first day. How can I get the real price of the stock?

We sincerely suggest you to carefully read the documentation of our library as well as the official paper. After that, if you still feel puzzled, please describe the question clearly under this issue.

尝试执行示例代码出错了ValueError: operands could not be broadcast together with shapes (0,) (3011,)

❓ Questions and Help

前面安装配置一切OK

官方说明是使用qrun benchmarks/LightGBM/workflow_config_lightgbm.yaml

但是我目录下没有这个文件，只有类似的以yaml结尾的两个文件，于是不管执行

qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha360.yaml 还是

qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml 都提示

ValueError: operands could not be broadcast together with shapes (0,) (3011,)

似乎是数据上格式不匹配，没有对数据及文件进行任何修改，执行XGBOOST目录下其他yaml都是同样的问题！

scripts folder not available after pip install pyqlib

Hi,

I am trying to convert my csv data file into Qlib format using your direction in your doc. However, after installing pyqlib I don't see scripts available. Will you please help? Thanks.

platform: MacOS
python v: 3.8

Make Internal Data Index <datetime, instruments> instead of <instruments, datetime>

🌟 Feature Description

Make Internal Data Index <datetime, instruments> instead of <instruments, datetime>

Motivation

In time series data, the datetime dimension is more frequently used than instruments.

Make the underlying data use the time dimension as the first index level will improve the data efficiency in more scenario

Add CI for `pip install pyqlib`

🌟 Feature Description

Add CI for pip install pyqlib

Motivation

Currently, the CI clone the latest qlib and test the scripts.
Besides, we think we should test if the latest scripts still works for the latest version installed by pip install pyqlib (Many users install the qlib by pip install pyqlib and run the examples directly)

I think only part of scripts is required to be tested.

Supporting Graph Data

Currently, Qlib has designed a dedicated data structure for time-series finance data.
The graph data is very popular recently. Supporting graph data will support more research directions.

`expanding_rsquare` always return `nan`

Reproduce the bug:

import numpy as np
import statsmodels.api as sm

from qlib.data._libs.rolling import rolling_rsquare, rolling_resi, rolling_slope, rolling_mean
from qlib.data._libs.expanding import expanding_rsquare, expanding_resi, expanding_slope, expanding_mean

x = np.random.randn(100)

print('test rolling')
# window size = 20
assert np.isclose(x[-20:].mean(), rolling_mean(x, 20)[-1])
model = sm.OLS(x[-20:], sm.add_constant(np.arange(20))).fit()
assert np.isclose(model.params[1], rolling_slope(x, 20)[-1])
assert np.isclose(model.resid[-1], rolling_resi(x, 20)[-1])
assert np.isclose(model.rsquared, rolling_rsquare(x, 20)[-1])

print('test expanding')
model = sm.OLS(x, sm.add_constant(np.arange(len(x)))).fit()
assert np.isclose(model.params[1], expanding_slope(x)[-1])
assert np.isclose(model.resid[-1], expanding_resi(x)[-1])
assert np.isclose(model.rsquared, expanding_rsquare(x)[-1]) # the bug one
print(expanding_rsquare(x)) # all nan

It can be fixed by changing this line

qlib/qlib/data/_libs/expanding.pyx

Line 120 in 9895106

self.x2_sum += size

to

self.x2_sum += size * size

API for appending data

❓ Questions and Help

According to the paper, appending data should be super easy using qlib data format.
Is there any documentation about how to use this feature?

BTW:
Is there any api to write dataframes into the data store directly?
We would like to use the data store in a more interactive way, e.g., read some data from the data store, transform it, then write the result back to the data store using a different identifier.

Thanks in advance!

D.features, redis_lock.AlreadyAcquired: Already acquired from this Lock instance.

instruments = ['SH600000'] fields = ['close', ] df = D.features(instruments, fields, start_time='2010-01-01', end_time='2012-12-31', freq='day') print(df)

Traceback (most recent call last): File "test.py", line 127, in <module> df = D.features(instruments, fields, start_time='2010-01-01', end_time='2012-12-31', freq='day') File "F:\anaconda3\lib\site-packages\qlib-0.5.0-py3.6-win-amd64.egg\qlib\data\data.py", line 968, in features return DatasetD.dataset(instruments, fields, start_time, end_time, freq, disk_cache) File "F:\anaconda3\lib\site-packages\qlib-0.5.0-py3.6-win-amd64.egg\qlib\data\cache.py", line 301, in dataset return self._dataset(instruments, fields, start_time, end_time, freq, disk_cache) File "F:\anaconda3\lib\site-packages\qlib-0.5.0-py3.6-win-amd64.egg\qlib\data\cache.py", line 677, in _dataset with CacheUtils.writer_lock(self.r, "dataset-%s" % _cache_uri): File "F:\anaconda3\lib\contextlib.py", line 82, in __enter__ return next(self.gen) File "F:\anaconda3\lib\site-packages\qlib-0.5.0-py3.6-win-amd64.egg\qlib\data\cache.py", line 200, in writer_lock current_cache_wlock.acquire() File "F:\anaconda3\lib\site-packages\redis_lock\__init__.py", line 223, in acquire raise AlreadyAcquired("Already acquired from this Lock instance.") redis_lock.AlreadyAcquired: Already acquired from this Lock instance.

Add CI/CD

Hi all, I suggest we should add CI/CD as soon as possible. It will be necessary to do so before we could bring more features into the project. Here are some most popular CI tools that I know:

• Travis CI
• Circle CI

Github also provides a CI/CD tool: GitHub Actions. It is another good alternative for us to set up a workflow.

Besides, we could add some issue templates that can help users make bug reports and feature requests more clearly.

Derek

Did not get graph in jupyternotebook

Hi,

I ran the jupyter notebook in my local, but did not see any graphs,
I downloaded US data using python scripts/get_data.py qlib_data --target_dir ../.qlib/qlib_data/us_data --region us

and set:

market = "sp500" 

benchmark = "_^ndx"

Can anyone give me some guidance how to get the graph show up in jupyternotebook?
Any help/hint will be appreciated! Thanks.

No module named 'qlib.tests'

❓ Questions and Help

!git clone https://github.com/microsoft/qlib
!python ./qlib/scripts/get_data.py qlib_data_cn --target_dir ./qlib/qlib_data/cn_data

-> Traceback (most recent call last):
File "./qlib/scripts/get_data.py", line 4, in
from qlib.tests.data import GetData
ModuleNotFoundError: No module named 'qlib.tests'

WeChat or QQ group

❓ Questions and Help

Do you have a WeChat or QQ group ?

A scaled down approach to examples

🌟 Feature Description

Provide settings to test the functionality of the package quickly.

Motivation

When I study a new package, the quality of its analysis is of little importance: the main goal is to research the pipeline, tools, find out valid inputs, decision points, etc. I.e. I don't need valid results, just any results: I will check the data on the next step as documentation suggests.
Current requirement to run examples is 5G/16G, which I will need to substitute for another volume of data anyway. I suggest you add option to get a limited-size dataset to test-run the package, look up valid data format and decompose the pipeline into wrapper functions

Cannot install pyqlib

install latest version of python 3.9, but it cannot install pyqlib, report not find version.

Redesign the Trainer module

The trainer module is not flexible enough now.
In the future, we will redesign the trainer module.

Futures Trading

🌟 Feature Description

Futures trading support

Do you have any plans to add the futures trading support to the framework?

Removal of Sacred and redesign of experiment API

Hi, due to the incompatibility of the sacred package we use for experiment, the examples will fail on some of the machines. Besides, sacred package is no longer actively updated (#761). We should remove it in the future and redesign the API for experiment.

和BigQuant的接口设计和代码命名都是一样的，这个不侵犯知识产权么？

• 看了几个数据接口和变量命名，都是一样的
• BigQuant 16年就就发布了，这个应该是来自BigQuant的
• 微软是否有考虑知识产权问题。我们公司在商业中使用，是否会有连带知识产权问题？

python setup.py install failure

qlib/data/_libs/rolling.cpp:630:10: fatal error: 'ios' file not found
#include "ios"
^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
We sincerely suggest you to carefully read the documentation of our library as well as the official paper. After that, if you still feel puzzled, please describe the question clearly under this issue.

ModuleNotFoundError: No module named 'qlib.config'

Both on Windows and Linux, when I run the command: python run_all_model.py --models=lightgbm, I got the following error message:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'qlib.config'

What is exp_info.json?

examples/estimator/analyze_from_estimator

with CUR_DIR.joinpath(estimator_name, 'exp_info.json').open() as fp:
    latest_id = json.load(fp)['id']

Thanks

ai量化是趋势啊

📖 Documentation

Erroneous float calculations.

🐛 Bug Description

When an expression is being evaluated, I think the series is changed into np.float32 and when returning back its being changed back to np.float64. Calculation results are in fact erroneous.

To Reproduce

d = get_quote_data([ticker], fields=['$close', '$volume', 'Mul($close, $volume)'], start_time='2020-10-23', end_time='2020-10-30')

Expected Behavior

The actual value should be 3989893260.0

Screenshot

Environment

Note: One could run python scripts/collect_info.py under the qlib directory to get the following information.

• Qlib version:
• Python version:
• OS (Windows, Linux, MacOS):
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

No module named 'qlib.tests

❓ Questions and Help

(TF2..1) D:\winAI\qlib-main>python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn
Traceback (most recent call last):
File "scripts/get_data.py", line 4, in
from qlib.tests.data import GetData
ModuleNotFoundError: No module named 'qlib.tests'
I have already pip install pyqlib, why this happened？

Refactor the log system

Current log design is too complicated.

Replace it with existing multi-process(threading)-safe logger modules (Such as loguru)

Is there any buy or sell function for simulation trading?

📖 Documentation

Perferct work and thanks all the contributors.
In the documentation, i can not find the buy or sell function for simulation trading , is there any possbile solution ?
Anymore, is there any solution to provide the functions such as before the stock market operation, after the stock market opertion ?

Is there any tools provided to trun the qlib data format to csv format?

❓ Questions and Help

Your work helps me a lot. I would like to ask whether these has tools to turn the qlib data format to csv. Since I find it is a good format to use, but sometimes, I may need to trun it to csv to show it to my coworkers who may never use qlib before. I find Converting CSV Format into Qlib Format in your documentation but don't find the opposite.

Is tensorflow model available ?

Examples show that pytorch models can be used in qlib.

I wonder is tensorflow models available in this framework?

Ref ops creating an issue with non existing instruments

🐛 Bug Description

querying 'Ref' fields for non existing instruments fails

To Reproduce

Steps to reproduce the behavior:

D.features(tickers, fields, start_time=start_time, end_time=end_time, freq=freq)
passing non existing instruments to the above tickers will fail, where fields = ['$close/Ref($close, 1)-1']

Expected Behavior

should ideally return an empty data

Screenshot

Environment

Note: One could run python scripts/collect_info.py under the qlib directory to get the following information.

• Qlib version:
• Python version:
• OS (Windows, Linux, MacOS):
• Commit number (optional, please provide it if you are using the dev version):

Additional Notes

How to define custom OPS like EMA in ops.py ?

How to define custom OPS like EMA in ops.py ?

Thanks

Refactor model management

Currently, we use sacred to manage the models and experiments results.
In the future, this part may be refactored.