philipperemy / fx-1-minute-data

HISTDATA - Dataset composed of all FX trading pairs / Crude Oil / Stock Indexes. Simple API to retrieve 1 Minute data (and tick data) Historical FX Prices (up to date).

FX 1-Minute Dataset (+ Crude Oil and Stock indexes e.g. SP500)

API and dataset download for histdata.com. Tick data is also available with the API.

Download the dataset

• Click Here to download it: All instruments - 1Minute - 2000 / 2023 - Google Drive link (3GB).

• You can also re-download the whole dataset (up to date) by yourself:

pip install -r requirements.txt
python download_all_fx_data.py

Expect it to take around 10 minutes if you have a fast internet connection.

API

pip install histdata

Examples

from histdata import download_hist_data as dl
from histdata.api import Platform as P, TimeFrame as TF

• Download tick data for 2019/06:

dl(year='2019', month='6', pair='eurusd', platform=P.GENERIC_ASCII, time_frame=TF.TICK_DATA)

• Other possible calls:

dl(year='2019', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_LAST)
dl(year='2019', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_ASK)
dl(year='2019', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_BID)
dl(year='2019', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.ONE_MINUTE)
dl(year='2019', month='6', pair='eurusd', platform=P.GENERIC_ASCII, time_frame=TF.TICK_DATA)
dl(year='2019', month='6', pair='eurusd', platform=P.EXCEL, time_frame=TF.ONE_MINUTE)
dl(year='2019', month='6', pair='eurusd', platform=P.META_TRADER, time_frame=TF.ONE_MINUTE)
dl(year='2019', month='6', pair='eurusd', platform=P.META_STOCK, time_frame=TF.ONE_MINUTE)
dl(year='2018', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_LAST)
dl(year='2018', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_ASK)
dl(year='2018', month='6', pair='eurusd', platform=P.NINJA_TRADER, time_frame=TF.TICK_DATA_BID)

Data specification

This repository contains:

• A dataset of all the FX prices (1-minute data) from 2000, in Generic ASCII.
  • More than 66 FX pairs
• Contains some commodities:
  • WTI/USD = WEST TEXAS INTERMEDIATE in USD
  • BCO/USD = BRENT CRUDE OIL in USD
• Contains some indexes:
  • SPX/USD = S&P 500 in USD
  • JPX/JPY = NIKKEI 225 in JPY
  • NSX/USD = NASDAQ 100 in USD
  • FRX/EUR = FRENCH CAC 40 in EUR
  • UDX/USD = US DOLLAR INDEX in USD
  • UKX/GBP = FTSE 100 in GBP
  • GRX/EUR = DAX 30 in EUR
  • AUX/AUD = ASX 200 in AUD
  • HKX/HKD = HAN SENG in HKD E - TX/EUR = EUROSTOXX 50 in EUR
• A set of functions to download the historical prices yourself.

All the data is retrieved from: http://www.histdata.com/

Any file in a dataset is zipped and contains:

• a CSV (semicolon separated file).
• a status report (containing some meta data such as gaps).

Any CSV file looks like this:

20120201 000000;1.306600;1.306600;1.306560;1.306560;0
20120201 000100;1.306570;1.306570;1.306470;1.306560;0
20120201 000200;1.306520;1.306560;1.306520;1.306560;0
20120201 000300;1.306610;1.306610;1.306450;1.306450;0
20120201 000400;1.306470;1.306540;1.306470;1.306520;0
[...]

Headers are not included in the CSV files. They are:

DateTime Stamp;Bar OPEN Bid Quote;Bar HIGH Bid Quote;Bar LOW Bid Quote;Bar CLOSE Bid Quote;Volume

DateTime Stamp

Format: YYYYMMDD HHMMSS

Legend:

• YYYY – Year
• MM – Month (01 to 12)
• DD – Day of the Month
• HH – Hour of the day (in 24h format)
• MM – Minute
• SS – Second, in this case it will be always 00

TimeZone: Eastern Standard Time (EST) time-zone WITHOUT Day Light Savings adjustments

OPEN Bid Quote

The open (first) bid quote of the 1M bin.

HIGH Bid Quote

The highest bid quote of the 1M bin.

LOW Bid Quote

The lowest bid quote of the 1M bin.

CLOSE Bid Quote

The close (last) bid quote of the 1M bin.

Volume

Number of lots. Looks like it's always 0.

fx-1-minute-data's Issues

download all fx data with api?

Hey, what would be the most efficient way to download all of the FX pair data using the API?

Expired Certificate

Max retries exceeded with URL.

I think the Hisdata website's certificate expired.

csv is not really csv here

Thanks for the dataset, but shouldn't there be , instead of ; in the CSV files?

Use 7zip for compression: ~50% smaller than current one with zip files

Out of curiosity, tested compressing files since this repo hitting storage limit. Used the GBPUSD folder as a test: https://github.com/philipperemy/FX-1-Minute-Data/tree/master/2000-Jun2019/gbpusd

1. Extracted contents of each zip file
2. Compressed all the CSV and TXT files using 7zip CLI: 7z a -r -mx=9 gbpusd *.csv *.txt
3. New zip file: 27mb
4. Original folder size: 52mb

There are other compression algorithms/tools that have higher compression ratios, but found them to be unstable or too slow.

Add 2018 data

Add data for 2018.