stas-prokopiev / binance_historical_data Goto Github PK

View Code? Open in Web Editor NEW

72.0 72.0 27.0 78 KB

python PYPI package to dump all needed crypto historical data from binance just by 2 lines of code

License: MIT License

Python 100.00%

binance_historical_data's People

Contributors

Stargazers

Watchers

binance_historical_data's Issues

Problem downloading data for 29 Feb 2024

There is problem downloading data (1 minute, BTCUSDT) for Feb 2024. 29.02.204 is missing. If you delete all daily data for Feb 2024 and reload there is no monthly data downloaded for Feb 2024 (there is monthly data for Jan 2024 and daily data for Mar 2024 but no data for Feb 2024).

FileNotFoundError: [Errno 2] No such file or directory: 'data_dump/um/daily/aggTrades'

Hello

When I call data_dumper.delete_outdated_daily_results()

I get this error:

FileNotFoundError: [Errno 2] No such file or directory: 'data_dump/um/daily/aggTrades'

The correct path is: 'data_dump/futures/um/daily/aggTrades'

Add option to NOT download unavaiable coins.

Hi,

After download for 10 minutes, last month of 1INCH is 1INCHDOWNUSDT-1h-2021-12.csv

But we can see this coin is always trade:
https://www.binance.com/fr/trade/1INCH_USDT?theme=light&type=spot

Can you help me to understand ?

Подскажите,как запустить.

Здравствуйте.Подскажите,как запустить ,я не понял,если честно.Новичок в питоне.Спасибо.

OSError: [Errno 24] Too many open files

hi - this package seems really good. I know this might be outside of your remit but do you have any advice as to how to best handle this error?

apple m1 pro 32 gb

ulimit -n
256
ulimit -u
5333

many thanks

it does not download as per mentioned start_date

Hello, when I tried to download file by providing only start date, it did data from start of the current month.
To reproduce the problem use below code:

from binance_historical_data import BinanceDataDumper
from datetime import date

data_dumper = BinanceDataDumper(
    path_dir_where_to_dump=".",
    asset_class="um",  # spot, um, cm
    data_type="klines",  # aggTrades, klines, trades
    data_frequency="1m",
)

data_dumper.dump_data(
    tickers=None,
    date_start=date(day=23,month=6,year=2023),
    date_end=None,
    is_to_update_existing=False,
    tickers_to_exclude=["UST"],
)

maybe replacing below code:

# 2) Download all daily date
if self._data_type == "metrics":
    date_start_daily = date_start
else:
    date_start_daily = date_end_first_day_of_month

with:

# 2) Download all daily date
date_start_daily = date_start

will do the job.

Will there be a tickers_to_include

I am looking for ~30 of the top symbols. Can there be a tickers_to_include variable?

The first run of data_dumper downloaded only partial data

Hello,

I have used this script:

#!/usr/bin/env python3

from binance_historical_data import BinanceDataDumper

data_dumper = BinanceDataDumper(
    path_dir_where_to_dump=".",
    asset_class="spot",
    data_type="klines",
    data_frequency="3m",
)

data_dumper.dump_data()

Only partial data has been downloaded:

---> Found overall tickers: 2189
---> Filter to USDT tickers
------> Tickers left: 439
Download full data for 439 tickers: 
---> Data Frequency: 3m
---> Start Date: 20170101
---> End Date: 20230420
Tickers: 100%
Tried to dump data for 439 tickers:                                                                                                                                                                                                                                                                                                                                      
---> General stats:
------> NEW Data WAS dumped for 352 trading pairs
------> NEW Data WASN'T dumped for 87 trading pairs
---> New months saved:
------> For 438 tickers saved: 0 months
------> For 1 tickers saved: 68 months
---> New days saved:
------> For 348 tickers saved: 19 days
------> For 87 tickers saved: 0 days
------> For 2 tickers saved: 6 days
------> For 2 tickers saved: 18 days

Especially most of the monthly data is completely missing (only BTCUSDT has been downloaded) Why is that?

RuntimeError:

Hi, it gives me an error while doing this:

from binance_historical_data import BinanceDataDumper

data_dumper = BinanceDataDumper(
    path_dir_where_to_dump="d:\\",
    asset_class="spot",  # spot, um, cm
    data_type="klines",  # aggTrades, klines, trades
    data_frequency="30m",
)

data_dumper.dump_data()


<frozen runpy>", line 291, in run_path
......
  File "C:\Users\Stanislav\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 138, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Feature request: tickers_to_include

Hi, thank you for building this wonderful tool.
I'm wondering if there's any way to only download a particular trading pair by data_dumper.dump_data.
For example BTCUSD.

HTTPError: HTTP Error 451:

I am getting the error message: HTTPError: HTTP Error 451: when running the examples in your package homepage.

idle data_dumper.dump_data

Hi I tried running the below code and gave it some time but it does not download any data.

`import os

import datetime

from binance_historical_data import BinanceDataDumper

os.getcwd()

data_dumper = BinanceDataDumper(
path_dir_where_to_dump= ".", # /data/binance
asset_class="spot", # spot, um, cm
data_type="klines", # aggTrades, klines, trades
data_frequency="1h"
)

print(data_dumper.get_list_all_trading_pairs())

data_dumper.dump_data()

data_dumper.dump_data(
tickers = ["BTCUSDT"] ,
date_start = None,
date_end = None,

# date_start=datetime.date(year=2022, month=1, day=1),
# date_end=datetime.date(year=2022, month=1, day=3),
is_to_update_existing=False

console:
---> Found overall tickers: 2215
---> Filter to asked tickers: 1
------> Tickers left: 1
Download full data for 1 tickers:
---> Data Frequency: 1h
---> Start Date: 20170101
---> End Date: 20230514
Tickers: 0%| | 0/1 [00:00<?, ?it/s]
monthly files to download: 0%| | 0/69 [00:00<?, ?files/s]

running your tests/test_binance_historical_data.py gives the same results

Store historical data to a timeseries database

Hello,

It could be nice to provide an example to store Binance historical data into a timeseries database such as TimescaleDB or InfluxDB (and TICK stack)
Ideally a Docker environment could be create to automatically update data into database

Kind regards

since last update (0.1.8), BinanceDataDumper might not been properly working

Since last update,

every time I try to import BinanceDataDumper,

from binance_historical_data import BinanceDataDumper
import datetime

data_dumper = BinanceDataDumper(
    path_dir_where_to_dump="./test/",
    data_type="klines",  # aggTrades, klines, trades
    data_frequency="5m",  # argument for data_type="klines"
)

I keep getting the following error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-124f350d6565> in <module>
----> 1 from binance_historical_data import BinanceDataDumper
      2 import datetime
      3 
      4 data_dumper = BinanceDataDumper(
      5     path_dir_where_to_dump="./teste/",

/opt/conda/lib/python3.8/site-packages/binance_historical_data/__init__.py in <module>
      6 # Local imports
      7 from . import logger
----> 8 from .data_dumper import BinanceDataDumper
      9 
     10 # Global constants

/opt/conda/lib/python3.8/site-packages/binance_historical_data/data_dumper.py in <module>
     23 
     24 
---> 25 class BinanceDataDumper():
     26 
     27     _ASSET_CLASSES = ("spot")  # , "futures"

/opt/conda/lib/python3.8/site-packages/binance_historical_data/data_dumper.py in BinanceDataDumper()
    229             is_to_update_existing=False,
    230             int_max_tickers_to_get=None,
--> 231             tickers_to_exclude : Optional[list[str]] = None,
    232     ):
    233         """Main method to dump new of update existing historical data

TypeError: 'type' object is not subscriptable

VPN usage

If I use a VPN along this package, would that cause unexpected issues?

HTTP 451 and TimeoutError

if i open vpn , i get a http 451 error.
if directly connect without vpn, i get "TimeoutError: [WinError 10060] The connection attempt failed because the connecting party did not respond correctly after a period of time or the connected host did not respond.

How to avoid creating daily folder?

When I download the one-minute kline data of BTCUSDT, a monthly folder and a daily folder are created simultaneously. The monthly folder has all the data I need, whereas the daily folder has only ten days' data. How to avoid creating the daily folder? I cannot find that in the document. Thanks.

stas-prokopiev / binance_historical_data Goto Github PK

binance_historical_data's People

Contributors

Stargazers

Watchers

Forkers

binance_historical_data's Issues

data_dumper.dump_data()

Recommend Projects

Recommend Topics

Recommend Org