BT Proxy extremely unreliable about eq3btsmart

[BIGFAAT]

@CMMenzel @BIGFAAT Could you please enable debug mode on the thermostats and post a screenshot of the history of the "Connected" sensor entity so we can see in which interval the thermostats disconnect for you?

Done, will post the history and also try your script on my desktop tomorrow :)

[dbuezas]

@moscito1010 see here https://opensource.stackexchange.com/questions/11240/is-it-wrong-to-demand-features-in-open-source-projects

[dbuezas]

Hi!
Happy to hear you find it useful!

Can you try v5.0.3?
I have the suspicion a change related to pairing may have caused issues with btproxies

[EuleMitKeule]

@CMMenzel @BIGFAAT Could you please enable debug mode on the thermostats and post a screenshot of the history of the "Connected" sensor entity so we can see in which interval the thermostats disconnect for you?

[moscito1010]

The problem with the BLE proxies has existed for around 18 months.
The proxies are fine, it looks like it is the BLE driver used for esphome that is causing problems.

There is a workaround for using tasmota on the BLE proxies.
Tasmota sends the data to an mqtt broker and with an entry in the configuration.yaml you regulate the data exchange between HA and the EQ-3's
There is an adapted tasmota version with a MAC filter

https://github.com/Eroli/Tasmota/tree/master

If you're interested, I'll write instructions here. Thanks to “Eroli” I now have some practice :-)

[EuleMitKeule]

Actually the BLE proxies work perfectly fine without any disconnects at all, you simply have to send a message to the thermostat more often (like every 10 seconds) for the thermostat to not shut down its radio. I have fixed this problem and tested it out successfully using esp32 bluetooth proxies over long periods of time. The updated integration will be part of the next release of the Home Assistant core.

[EuleMitKeule]

I experience the same behaviour with D1 Mini ESP32 bluetooth proxies. I will try out v5.0.3 and will report if the connection is more stable.

[EuleMitKeule]

As you can see the valve still frequently goes unavailable.

I get this error whenever the thermostats go offline:
[custom_components.dbuezas_eq3btsmart.climate] [Büro Heizung] Error updating: Device not found

Also sometimes this error gets logged:

2024-01-13 20:44:50.020 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/dbuezas_eq3btsmart/sensor.py", line 122, in fetch_serial
    await self._thermostat.async_query_id()
  File "/config/custom_components/dbuezas_eq3btsmart/python_eq3bt/eq3bt/eq3btsmart.py", line 160, in async_query_id
    await self._conn.async_make_request(value)
  File "/config/custom_components/dbuezas_eq3btsmart/python_eq3bt/eq3bt/bleakconnection.py", line 164, in async_make_request
    await self._async_make_request_try(value, retries)
  File "/config/custom_components/dbuezas_eq3btsmart/python_eq3bt/eq3bt/bleakconnection.py", line 203, in _async_make_request_try
    raise ex
  File "/config/custom_components/dbuezas_eq3btsmart/python_eq3bt/eq3bt/bleakconnection.py", line 182, in _async_make_request_try
    await asyncio.wait_for(
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError

[BIGFAAT]

I can relate to this issue as well so please tell me if you need any specific log or test done.

My esp config:

esphome:
  name: internetdesp
  friendly_name: internetdesp

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended
    # Custom sdkconfig options
    sdkconfig_options:
      COMPILER_OPTIMIZATION_PERFORMANCE: y

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "***"

ota:
  password: "***"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  passive_scan: true
  enable_btm: true
  enable_rrm: true
  fast_connect: true
  power_save_mode: high
  manual_ip:
    static_ip: ***
    gateway: ***
    subnet: ***
    dns1: ***

esp32_ble_tracker:
  scan_parameters:
    active: false
    interval: 1100ms
    window: 1100ms

bluetooth_proxy:
  active: true
  cache_services: true
    

[BIGFAAT]

In the meantime i tried to set up ble_client on the esp32 with:

ble_client:
  - mac_address: ***
    id: Fenster
    on_passkey_request:
      then:
        - ble_client.passkey_reply:
            id: Fenster
            passkey: ***

I get this as error within the esp:

[21:10:37][D][esp32_ble_client:053]: [3] [00:1A:22:1B:99:6A] Found device
[21:10:37][D][esp32_ble_tracker:215]: Pausing scan to make connection...
[21:10:37][I][esp32_ble_client:069]: [3] [00:1A:22:1B:99:6A] 0x00 Attempting BLE connection
[21:10:38][I][bluetooth_proxy:278]: [0] [00:1A:22:1B:99:6A] Connecting v3 with cache
[21:10:43][I][esp32_ble_client:201]: [3] [00:1A:22:1B:99:6A] Connected
[21:10:43][I][esp32_ble_client:069]: [0] [00:1A:22:1B:99:6A] 0x00 Attempting BLE connection
[21:10:43][D][esp-idf:000]: W (109245) BT_GATT: gatt_connect wrong state 4

[21:10:43][I][esp32_ble_client:270]: [0] [00:1A:22:1B:99:6A] auth complete. remote BD_ADDR: ***
[21:10:43][E][esp32_ble_client:273]: [0] [00:1A:22:1B:99:6A] auth fail reason = 0x50
[21:10:43][I][esp32_ble_client:270]: [3] [00:1A:22:1B:99:6A] auth complete. remote BD_ADDR: ***
[21:10:43][E][esp32_ble_client:273]: [3] [00:1A:22:1B:99:6A] auth fail reason = 0x50
[21:10:43][I][esp32_ble_client:149]: [0] [00:1A:22:1B:99:6A] Connected
[21:10:43][D][esp32_ble_tracker:266]: Starting scan...
[21:10:43][W][bluetooth_proxy.connection:096]: [0] [***] Error writing char/descriptor at handle 0x430, status=5
[21:10:43][D][esp-idf:000]: E (109614) BT_BTM: BTM_GetSecurityFlags false

I also did reset the thermostat and double checked the pin. Second bluetooth thermostat from another brand works as intended with same config.

[CMMenzel]

Sure thing - I have also updated to v5.0.3 - no changes in behavior
Here is the screenshot:

@EuleMitKeule do you believe it is connected to the HW (D1 Mini ESP32)?

[EuleMitKeule]

What scan interval did you configure in Home Assistant?

I believe the thermostats disable their bluetooth radio if no command is sent after X minutes. For me the radio stays active about 1:30 minutes and the gets disabled for about 30 seconds, but for you it might be different.

If you are familiar with Python could you try out the following two scripts on a machine that has an internal or external bluetooth adapter and that is close to the thermostat and tell us what the output is?

You'd have to do pip install bleak and change the MAC_ADDRESS to the address of your thermostat.

This script should lead to disconnects:

import asyncio
import logging

from bleak import BleakClient

MAC_ADDRESS = "00:1A:22:0C:E7:E3"  # Replace with the MAC address you want to connect to
NOTIFY_UUID = "d0e8434d-cd29-0996-af41-6c90f4e0eb2a"
WRITE_UUID = "3fa4585a-ce4a-3bad-db4b-b8df8179ea09"

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")


class Client:
    def __init__(self):
        self._client = BleakClient(
            MAC_ADDRESS, disconnected_callback=self._on_disconnect
        )
        self._is_connected = False

    @property
    def is_connected(self):
        return self._is_connected

    async def run(self):
        while True:
            while not self._client.is_connected:
                try:
                    await self._client.connect()
                    self._is_connected = True
                except Exception as e:
                    logging.error(e)
                    await asyncio.sleep(1)

            await self._client.start_notify(NOTIFY_UUID, self._on_notify)

            while self._client.is_connected:
                pass
                # await self._client.write_gatt_char(WRITE_UUID, bytearray([0x02]))
                await asyncio.sleep(10)

    def _on_disconnect(self, client: BleakClient):
        self._is_connected = False

    def _on_notify(self, sender: int, data: bytearray):
        # log the received data in this format: Received: 0x00 0x01 0x02 ...
        logging.debug(f"Received: {' '.join([f'0x{b:02x}' for b in data])}")


class Monitor:
    def __init__(self, client: Client):
        self._client = client
        self._last_is_connected: bool | None = None

    async def run(self):
        while True:
            if self._last_is_connected != self._client.is_connected:
                self._last_is_connected = self._client.is_connected
                logging.info(
                    f"Status changed: {'Connected' if self._last_is_connected else 'Disconnected'}"
                )

            await asyncio.sleep(1)


async def main():
    client = Client()
    monitor = Monitor(client)

    await asyncio.gather(client.run(), monitor.run())


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        logging.info("KeyboardInterrupt")

This should keep an active connection without any disconnects:

import asyncio
import logging

from bleak import BleakClient

MAC_ADDRESS = "00:1A:22:0C:E7:E3"  # Replace with the MAC address you want to connect to
NOTIFY_UUID = "d0e8434d-cd29-0996-af41-6c90f4e0eb2a"
WRITE_UUID = "3fa4585a-ce4a-3bad-db4b-b8df8179ea09"

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")


class Client:
    def __init__(self):
        self._client = BleakClient(
            MAC_ADDRESS, disconnected_callback=self._on_disconnect
        )
        self._is_connected = False

    @property
    def is_connected(self):
        return self._is_connected

    async def run(self):
        while True:
            while not self._client.is_connected:
                try:
                    await self._client.connect()
                    self._is_connected = True
                except Exception as e:
                    logging.error(e)
                    await asyncio.sleep(1)

            await self._client.start_notify(NOTIFY_UUID, self._on_notify)

            while self._client.is_connected:
                await self._client.write_gatt_char(WRITE_UUID, bytearray([0x02]))
                await asyncio.sleep(10)

    def _on_disconnect(self, client: BleakClient):
        self._is_connected = False

    def _on_notify(self, sender: int, data: bytearray):
        # log the received data in this format: Received: 0x00 0x01 0x02 ...
        logging.debug(f"Received: {' '.join([f'0x{b:02x}' for b in data])}")


class Monitor:
    def __init__(self, client: Client):
        self._client = client
        self._last_is_connected: bool | None = None

    async def run(self):
        while True:
            if self._last_is_connected != self._client.is_connected:
                self._last_is_connected = self._client.is_connected
                logging.info(
                    f"Status changed: {'Connected' if self._last_is_connected else 'Disconnected'}"
                )

            await asyncio.sleep(1)


async def main():
    client = Client()
    monitor = Monitor(client)

    await asyncio.gather(client.run(), monitor.run())


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        logging.info("KeyboardInterrupt")

[CMMenzel]

@EuleMitKeule Thanks a lot for the code. All thermostats are configured with the standard 1 minute. Unfortunately I don't have a linux machine that is close enough to the thermostats. I will try to do this with my mac on sunday.
In the meantime I figured out that the log of the BT proxy is giving me some errors - however, researching these on the internet has not brought me any further.. Any thoughts?

[15:46:28][I][app:102]: ESPHome version 2023.12.9 compiled on Jan 26 2024, 15:45:38
[15:46:28][I][app:104]: Project esphome.bluetooth-proxy version 1.0
[15:46:28][C][wifi:573]: WiFi:
[15:46:28][C][wifi:405]:   Local MAC: E8:9F:6D:BB:04:38
[15:46:28][C][wifi:410]:   SSID: [redacted]
[15:46:28][C][wifi:411]:   IP Address: 192.168.178.35
[15:46:28][C][wifi:413]:   BSSID: [redacted]
[15:46:28][C][wifi:414]:   Hostname: 'esp32-bluetooth-proxy-bb0438'
[15:46:28][C][wifi:416]:   Signal strength: -65 dB ▂▄▆█
[15:46:28][C][wifi:420]:   Channel: 1
[15:46:28][C][wifi:421]:   Subnet: 255.255.255.0
[15:46:28][C][wifi:422]:   Gateway: 192.168.178.1
[15:46:28][C][wifi:423]:   DNS1: 192.168.178.1
[15:46:28][C][wifi:424]:   DNS2: 0.0.0.0
[15:46:28][C][logger:439]: Logger:
[15:46:28][C][logger:440]:   Level: DEBUG
[15:46:28][C][logger:441]:   Log Baud Rate: 115200
[15:46:28][C][logger:443]:   Hardware UART: UART0
[15:46:28][C][bluetooth_proxy:088]: Bluetooth Proxy:
[15:46:28][C][bluetooth_proxy:089]:   Active: YES
[15:46:28][C][safe_mode.button:022]: Safe Mode Button 'Safe Mode Boot'
[15:46:28][C][safe_mode.button:022]:   Icon: 'mdi:restart-alert'
[15:46:28][C][esp32_ble:374]: ESP32 BLE:
[15:46:28][C][esp32_ble:376]:   MAC address: E8:9F:6D:BB:04:3A
[15:46:28][C][esp32_ble:377]:   IO Capability: none
[15:46:28][C][esp32_ble_tracker:645]: BLE Tracker:
[15:46:28][C][esp32_ble_tracker:646]:   Scan Duration: 300 s
[15:46:28][C][esp32_ble_tracker:647]:   Scan Interval: 1100.0 ms
[15:46:28][C][esp32_ble_tracker:648]:   Scan Window: 1100.0 ms
[15:46:28][C][esp32_ble_tracker:649]:   Scan Type: ACTIVE
[15:46:28][C][esp32_ble_tracker:650]:   Continuous Scanning: True
[15:46:28][C][mdns:115]: mDNS:
[15:46:28][C][mdns:116]:   Hostname: esp32-bluetooth-proxy-bb0438
[15:46:28][C][ota:097]: Over-The-Air Updates:
[15:46:28][C][ota:098]:   Address: esp32-bluetooth-proxy-bb0438.local:3232
[15:46:28][C][api:139]: API Server:
[15:46:28][C][api:140]:   Address: esp32-bluetooth-proxy-bb0438.local:6053
[15:46:28][C][api:142]:   Using noise encryption: YES
[15:46:28][C][improv_serial:032]: Improv Serial:
[15:46:37][I][bluetooth_proxy:278]: [0] [00:1A:22:11:F4:32] Connecting v3 with cache
[15:46:37][D][esp32_ble_tracker:215]: Pausing scan to make connection...
[15:46:37][I][esp32_ble_client:069]: [0] [00:1A:22:11:F4:32] 0x00 Attempting BLE connection
[15:46:39][I][esp32_ble_client:149]: [0] [00:1A:22:11:F4:32] Connected
[15:46:39][D][esp32_ble_tracker:266]: Starting scan...
[15:46:39][D][esp-idf:000]: E (28288) BT_L2CAP: slave connection parameters update failed, the parameters are out of range

[15:46:39][D][esp-idf:000]: E (28305) BT_APPL: service change write ccc failed

[15:46:40][W][bluetooth_proxy.connection:096]: [0] [00:1A:22:11:F4:32] Error writing char/descriptor at handle 0x430, status=5
[15:47:40][W][bluetooth_proxy:246]: [0] [00:1A:22:11:F4:32] Connection already established
[15:47:40][W][bluetooth_proxy.connection:096]: [0] [00:1A:22:11:F4:32] Error writing char/descriptor at handle 0x430, status=5
[15:48:40][W][bluetooth_proxy:246]: [0] [00:1A:22:11:F4:32] Connection already established
[15:48:40][W][bluetooth_proxy.connection:096]: [0] [00:1A:22:11:F4:32] Error writing char/descriptor at handle 0x430, status=5
[15:49:41][W][bluetooth_proxy:246]: [0] [00:1A:22:11:F4:32] Connection already established
[15:49:41][W][bluetooth_proxy.connection:096]: [0] [00:1A:22:11:F4:32] Error writing char/descriptor at handle 0x430, status=5

[EuleMitKeule]

I can only guess, maybe the bluetooth proxies think they are connected and are trying to write to the thermostat even though the thermostat already turned off its radio?
You could also try out the updated integration on the develop branch in Home Assistant and see if the disconnects stop. You would have to manually copy the custom_components/eq3btsmart directory into your Home Assistant's config directory, do a manual pip install git+https://github.com/dbuezas/eq3btsmart@develop from within your Home Assistant environment and delete and reconfigure the config entries for your thermostats.

Once the changes are merged into main this process will be much simpler of course.

[orgebar]

If you're interested, I'll write instructions here. Thanks to “Eroli” I now have some practice :-)

Thanks! Instructions would be awesome! :)

Maybe you can put it as a new topic in the discussion section for better visibility?

To be honest, ESP32-Setup and Usage comes too short inside the documentation. There is a lot of Information on the ESP32-topic spread over many tickets and could be mentioned in the readme. Its hard to determine relevant steps to get it working or even if there are any, especially if you are not a very experienced user.

[moscito1010]

#4 (comment)

Of course an ESPhome solution is preferable but as long as it doesn't work well this is a route you can take.

Besides, a solution where you force the thermostat to be active every 10 seconds isn't really a good solution. The thermostat batteries are used up 3 to 4 times faster. It makes little sense to keep a passive BLE unit permanently active. So this can at most be a workaround that doesn't solve the actual problem.
So you haven't really fixed the problem.

A doctor would say that you have treated the symptom but not eliminated the cause.
The driver is generally good, but apparently not or no longer suitable for passive BLE devices like the EQ-3 units. I don't know what was changed in the driver but it must be something.
The internet is full of people who despair about the problem :-)

I'll put it more drastically. Of course you can cover a coffee stain on a tablecloth with a fruit bowl but the stain is still there. Triggering the thermostats a lot is more like putting a fruit bowl on a coffee stain.
Check out the original Android app. You cannot address the units for 10 days (smartphone BLE off) but after switching on BLE and opening the app you will receive data from the units in no time and can also send data. You simply wait for a signal to be sent and contact is made and that's exactly how it was with the ESPhome BLE proxies and the EQ-3 units until about 2 years ago.

[dbuezas]

You simply wait for a signal to be sent and contact is made

These thermostats unfortunately don't seem to broadcast their state, one has to connect first. As to why the connection is so unreliable with btproxys, it is a bit of a mystery to me, particularly those with old firmware, since those don't have pairing with pin.

[moscito1010]

The new firmware with the 6 characters does not require any pairing.
Original statement from equiva support: the only difference between the two firmware versions is the number of characters for the pin.

I can also confirm that, I use 18 EQ-3s. 11 Units have 6 character pins the other 4 character pins.
They work much worse with the Tasmota solution without pairing and with ESPhome.
Of course Tasmota also uses an open source BLE driver, but a different one :-)
Why don't you just replace the driver and the whole thing would be over?

Some time ago I reset all EQ-3 and only switched on BLE. I haven't done any pairing. I haven't used any apps on my smartphone since the Tasmota solution worked reliably.
It should be mentioned in passing. The ESP32 modules that I use have an external antenna, which increases the range of the BLE and WIFI enormously. The normal ESP32 modules without an external antenna are significantly worse when it comes to BLE. This is due to the shape and arrangement of the PCB antenna.

By the way, only the MAC is required. Pairing is only necessary for the smartphone app. I can connect and manipulate any EQ-3 unit with my Linux computer. I tested it with my laptop at the neighbor's doorstep and manipulated its units, it worked great. He was probably a bit warm that day :-)
I haven't tried it with Windows yet, I don't have an MS OS.

Entering the pairing code in the ESPhome code is completely pointless because it is not queried. It's the app that asks for it and not the thermostat's firmware. This is supposedly to prevent abuse by neighbors :-)
If you use a Raspi4 (Home Assistant) it works like clockwork even without pairing.

It should be mentioned that the HA team has known about the problem for a long time. I am therefore not surprised that they are willing to adopt the solution with the timing even though it is known that this solution is a dead end. I'm just assuming that they have absolutely no desire to take care of a few heating nerds with EQ-3 units :-) They hope that the things will "die out" and that the problem will solve itself.
I've gotten a bloody nose several times on the Discord when I brought up the topic.
The most sincere sentence was: "Don't bother, buy some decent stuff with zigbee and don't expect that your bad purchase will be supported"
Oh well.......

[moscito1010]

lol

I think you're misjudging me.
I didn't make any bold demands on the developers, but politely and objectively pointed out a reproducible error and suggested solutions. Much more detailed than what you have read from me here so far.

I am very familiar with the way we work in open source projects and I am aware that the people behind the project sacrifice their free time for it. Every year I donate a not inconsiderable sum of money to various open source projects that I use myself in order to at least financially recognize the work behind them.
Subtly accusing me of just making demands is very superficial.

It's possible that I get on someone's nerves with the way I talk, but problems aren't solved by talking about them every now and then, but rather by addressing the problem again and again.

As a user, I am not a bit maker but an important link in the open source chain. Without information from users for whom these projects are, there is no improvement or progress.

But it's not like I didn't understand your point.
Since it obviously bothers you, I will never write anything on the topic again.

passt schon :-)

[dbuezas]

The new firmware with the 6 characters does not require any pairing

Not my experience. The previous fw had fake pairing, the new one has real Bluetooth paring, but it looks like it has a bug where it will still take commands while pairing. A client that respects the standard can't communicate without pairing.

It's the app that asks for it and not the thermostat's firmware

Not in the new fw

Why don't you just replace the driver and the whole thing would be over?

Probably the same reason you don't, but to a leaser extent

[maciekelga]

ESP Bt Proxy is stable only with 2 TRV. I'm using BT Proxy on ESP32 for Home Assistant for almost 1.5 year for my 4 pcs TRV. As long as I have active 2 BT proxy device all working very well, until i disconnect one of them. Maximum limit is 2pcs TRV per one BT Proxy.

[orgebar]

There is an adapted tasmota version with a MAC filter

https://github.com/Eroli/Tasmota/tree/master

If you're interested, I'll write instructions here. Thanks to “Eroli” I now have some practice :-)

Could you please provide the instructions? Thanks! 😊