Issue with too many subscriptions using watch_tickers() about ccxt

[jabranzahid-dev]

thanks, I will keep an eye on this thread for any updates, in the meanwhile I will test with other exchanges

[carlosmiei]

Hello @jabranzahid-dev, Does that happen immediately? the duplicated values

[jabranzahid-dev]

Hello @jabranzahid-dev, Does that happen immediately? the duplicated values

yes immediately. I mean the values do eventually change but after a while, so basically I am not getting prices in realtime, i.e. the frequency at which the values change depend on how many symbols I am subscribed to, for example, if I increase the symbols in the regions of 50-100, the values update every few seconds (for BTC/USDT I expect the values to change in every message right?), but I increase the symbols to 500, the values will change may be after 45 seconds BUT I will keep receiving new messages for every symbol. Hope I have explained correctly :)

[carlosmiei]

@jabranzahid-dev I was trying to debug this issue by running the snippet below, but although some messages have the same last price and some other common fields, I don't see duplicated timestamps (which would be a red flag)

    cache = {}
    markets = await exchange.load_markets(True)
    # exchange.verbose = True
    spot_markets = [market['symbol'] for market in markets.values() if market['spot']][:200]
    while True:
        tickers = await exchange.watch_tickers(spot_markets)
        for ticker_symbol in tickers:
            ticker = tickers[ticker_symbol]
            if ticker_symbol not in cache:
                cache[ticker_symbol] = ticker
            else:
                if ticker['last'] == cache[ticker['symbol']]['last']:
                    print('No change in price for', ticker['symbol'])
                    print('Timestamps:', ticker['timestamp'], cache[ticker['symbol']]['timestamp'])
                if ticker['timestamp'] == cache[ticker['symbol']]['timestamp']:
                    print('No change in timestamp for', ticker['symbol'])
                cache[ticker['symbol']] = ticker

[carlosmiei]

@jabranzahid-dev Actually after some time I got a duplicated timestamp as well but that was sent by the exchange in two different messages

2024-05-07T10:01:54.302Z message {"e":"24hrTicker","E":1715076113164,"s":"XVGETH","p":"0.00000000","P":"0.000","w":"0.00000173","x":"0.00000175","c":"0.00000174","Q":"2091.00000000","b":"0.00000173","B":"40395.00000000","a":"0.00000174","A":"65980.00000000","o":"0.00000174","h":"0.00000177","l":"0.00000170","v":"11705621.00000000","q":"20.27069206","O":1714989713163,"C":1715076113163,"F":11609979,"L":11611339,"n":1361}



2024-05-07T10:01:54.302Z message {"e":"24hrTicker","E":1715076113164,"s":"XVGETH","p":"0.00000000","P":"0.000","w":"0.00000173","x":"0.00000175","c":"0.00000174","Q":"2091.00000000","b":"0.00000173","B":"40395.00000000","a":"0.00000174","A":"65980.00000000","o":"0.00000174","h":"0.00000177","l":"0.00000170","v":"11705621.00000000","q":"20.27069206","O":1714989713163,"C":1715076113163,"F":11609979,"L":11611339,"n":1361}

[jabranzahid-dev]

yes maybe if you subscribe to 1000 symbols, you will see duplicated messages straight away. So yes, the messages are coming in from the exchanges but with same content in them, do you think this is issue with exchange or ccxt is not able to handle that many messages coming in ? I tested this few months ago and it was fine back then and I recently picked this project(reinstalled ccxt) and facing this issue so I am not sure if something changed on the exchange end or with CCXT library

[carlosmiei]

@jabranzahid-dev I don't think this is an issue with ccxt, the exchange is sending those duplicated messages not ccxt. We will reach out to them to try to clarify this scenario.

[carlosmiei]

@jabranzahid-dev Yes I can confirm that the duplicated timestamps come from the exchange directly, as you can see using this snippet that just uses the raw websockets package:

Nonetheless, the messages are not 100% equal, some of the fields like A are different even though the timestamp is the same.

ws_demo2.txt

[jabranzahid-dev]

Hi, Sorry it was an error in my code, I was subscribing to too many symbols too many times (incorrect exchange/symbol loops) so I think there is no issue. Sorry for inconvenience. Closing the issue. Regards