vanviegen / hue-thief

Factory reset Philips Hue bulbs using an EZSP-based Zigbee USB stick. After a reset, bulbs can easily join any type of compatible bridge.

Hue Thief

, required=True

Factory reset Philips Hue bulbs using an EZSP-based Zigbee USB stick. After a reset, bulbs can easily join any type of compatible bridge.

EZSP-based Zigbee USB sticks

These are devices based on a Silicon Labs EM351, EM357 or EM358 chip that communicates with the host PC using the EZSP protocol over a virtual COM port. Which protocol is used depends on which firmware is flashed.

• Silicon Labs/Linear/Nortek/GoControl HubZ/QuickStick Combo (HUSBZB-1)/Elelabs Raspberry Pi shield and USB adapter. These devices should come with an EZSP firmware preloaded.
• Home Assistant Skyconnect. Reportedly also works, and comes with the right EZSP firmware preloaded.
• Silicon Labs/Telegesis ETRX357USB/ETRX3USB/ETRX3. These devices come with an AT-command based firmware preloaded. You may be able to flash it with compatible firmware by doing something like this: https://community.home-assistant.io/t/eu-usb-sticks-for-the-new-zigbee-component/16718/10?u=frank

Installation

Make sure you have python 3.5 or newer and pip. (sudo apt-get install python3-pip)

git clone https://github.com/vanviegen/hue-thief
cd hue-thief
pip3 install --user -r requirements.txt

Usage

Bring the bulb(s) you want to factory reset close to your EZSP device. Shutdown any other applications (home assistant, perhaps?) that may be using the EZSP device. Power on the bulb(s) and immediately:

python3 hue-thief.py /dev/ttyUSB0

or if you need to change the baudrate to 115200 for the Home Assistant SkyConnect:

python3 hue-thief.py /dev/ttyUSB0 --baudrate 115200

/dev/ttyUSB0 should be your EZSP device. You should have full permissions on this device file.

In case you're using Ubuntu on Windows (WSL) you'll want to do something like this, assuming the EZSP device is mapped to COM4:

sudo python3 hue-thief.py /dev/ttyS4

Hue Thief will now scan all Zigbee channels for ZLL-compatible bulbs that are associated with any Zigbee network. When a bulb is found, it will blink a couple of times, and the application will ask if you want to factory reset this bulb. (If you didn't see any blinking, you may be doing your neighbours a favour by choosing 'N' here. :-))

That's it. Your now factory clean bulbs should be discoverable through whatever means your bridge/software offers.

Docker

If you already have Docker set up, then it's extremely simple to build a hue-thief image using the provided Dockerfile.

Docker build

git clone https://github.com/vanviegen/hue-thief
cd hue-thief
docker build -t hue-thief .

Docker run

Now that you have built the image, here's what you use every time you want to run it. Alternative devices include ttyUSB0 or ttyAMA0.

DEV=/dev/ttyUSB1 docker run --rm --device=$DEV:$DEV -it hue-thief python hue-thief/hue-thief.py $DEV

This will create a container from the image you built, and run it with the Zigbee device available inside the container. It will automatically run the hue-thief code from this project, then remove the container on completion.

Docker troubleshooting

If you want to run a container with a shell for diagnostics use:

DEV=/dev/ttyUSB1 docker run --rm --device=$DEV:$DEV -it hue-thief bash

A common issue is that the device is already in use by another process. Stop other containers that might be using the Zigbee radio and try again.

Problems

I will not be held responsible if you brick any hardware or do other awful things with this. On the bright side: I really don't see how that could ever happen, but still...

This script is kind of a hack, as it tries to implement about a zillion layers of Zigbee protocol in just a few lines of code. :-) So things will only work if everything goes exactly according to plan.

If no devices are found, there is no blinking or the factory reset doesn't work, the generated log.pcap file should be the first place to look for a clue. (Wireshark has decent Zigbee analyzers, though the ZLL commissioning part is still missing.)

hue-thief's Issues

Sonoff Dongle E

Is this supposed to work with the Sonoff Dongle E with the Silicon Labs EFR32MG21 chip?

No output, never exists

Running hue-thief I never see any output nor does the process ever exit. I run the docker container on a host that sees the cc2531 as /dev/ttyACM0:

[~] # dmesg | grep ACM
[426686.432044] cdc_acm 2-2:1.0: ttyACM0: USB ACM device
[~] # ls -l /dev/ttyACM0
crw------- 1 admin administrators 166, 0 2020-01-10 09:54 /dev/ttyACM0
[~] #

and run the container like this:

`docker run --rm --device=/dev/ttyACM0:/dev/ttyACM0 -it hue-thief:latest python hue-thief/hue-thief.py /dev/ttyACM0``

Inside the container, the device file shows like

root@2e13d5a444df:/# ls -l /dev/ttyACM0
crw------- 1 root root 166, 0 Jan 10 10:02 /dev/ttyACM0

so from a permissions perspective it should be ok.

After 30min of waiting in front of an empty terminal, I decided to open this issue. I read the troubleshooting section, but I can assure there's no other container using the device.

Works with home assistant skyconnect if baudrate is changed to 115200

Update bellows to latest version and change baudrate to 115200.

Range is about 10 cm

macOS compatibility?

I'm trying to run this on macOS with a Sonoff ZBDongle-E (CH9102F EZSP serial interface, Silicon Labs EFR32MG21 Chip).
The device got added as /dev/tty.usbserial-202208161433041.

Followed the installation instructions and got two warning at the end that the /Users/stefan/.local/bin folder is not in my PATH. Added that and then tried to run hue-thief.
Got this:

╭─    ~/hue-thief  on   master ······································································································································· 1 ✘  took 1m 28s   base   at 17:54:47  ─╮
╰─ python3 hue-thief.py /dev/tty.usbserial-202208161433041                                                                                                                                                                  ─╯
/Users/stefan/hue-thief/hue-thief.py:126: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(steal(sys.argv[1]))
Traceback (most recent call last):
  File "/Users/stefan/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/stefan/hue-thief/hue-thief.py", line 126, in <module>
    asyncio.get_event_loop().run_until_complete(steal(sys.argv[1]))
  File "/Users/stefan/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/stefan/hue-thief/hue-thief.py", line 26, in steal
    s = await util.setup(device, baudrate=57600)
  File "/Users/stefan/.local/lib/python3.10/site-packages/bellows/cli/util.py", line 108, in setup
    await s.reset()
  File "/Users/stefan/.local/lib/python3.10/site-packages/bellows/ezsp.py", line 84, in reset
    await self._gw.reset()
  File "/Users/stefan/.local/lib/python3.10/site-packages/bellows/uart.py", line 221, in reset
    return await asyncio.wait_for(self._reset_future, timeout=RESET_TIMEOUT)
  File "/Users/stefan/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Factory-resetting a Hue bulb didn't work.

Any other macOS users here? Any advice?
(Interesting that miniconda3 is used here, too ... 🤔)

Specify license

I would like to reimplement hue-thief to make it work with my zigbee sticks.
For that it would be great for hue-thief to have an explicitly specified license under which it can be (re)used.

fails at line 2: No module named 'pure_pcapy'

I have ensured that pure_pcapy is installed but it still throws this error

Traceback (most recent call last):
File "/home/henri/hue-thief/hue-thief.py", line 2, in
import pure_pcapy
ModuleNotFoundError: No module named 'pure_pcapy'

Install guide

On Ubuntu on WSL (probably on all newer ubuntus) I had to install python3-pip instead of python-pip3

Timeout Error during Communication with Bulb

Scanning on channel 20
NCP entered failed state. Requesting APP controller restart
Traceback (most recent call last):
  File "/home/colter/.local/lib/python3.10/site-packages/bellows/ezsp/protocol.py", line 152, in command
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/colter/hue-thief/hue-thief.py", line 135, in <module>
    asyncio.get_event_loop().run_until_complete(steal(args.device, args.baudrate, args.channel))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/colter/hue-thief/hue-thief.py", line 105, in steal
    await dev.mfglibSendPacket(frame)
  File "/home/colter/.local/lib/python3.10/site-packages/bellows/ezsp/protocol.py", line 151, in command
    async with asyncio_timeout(EZSP_CMD_TIMEOUT):
  File "/home/colter/.local/lib/python3.10/site-packages/async_timeout/__init__.py", line 141, in __aexit__
    self._do_exit(exc_type)
  File "/home/colter/.local/lib/python3.10/site-packages/async_timeout/__init__.py", line 228, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

I had my hue bulbs paired to a usb sonoff zigbee adaptor via Home Assistant but had a usb path break and need to reset all the bulbs. Some hue bulbs accepted the power cycle reset method, but a handful will not.

This touchlink reset method is promising because the bulbs are detected and do a flash, but the process seems to fail to get a response back from the bulb before it can ask to commense a reset?

I tried bridging the code to just send the reset packet automatically but get a similar error.

Bellows upgrade required when using newer firmware on HUSBZB-1

After upgrading my HUSBZB-1 firmware to 6.7.8, hue-thief stopped working. Upgrading bellows to 0.34.1 fixed the issue - pip3 install bellows==0.34.1.

NCP entered failed state. Requesting APP controller restart

First off, thanks so much for this script and keeping it updated! I know these things can become a burden, but its much appreciated!

I've got a few original Hue E27 bulbs, trying to migrate them over to another bridge of my own. When running the script I get the error below. I'm on a fresh Debian buster (64bit) install on a PI 3, with a USB conected Sonoff ZBDongle-E running a newer EZSP version (think its v7+). Not knowing an awful lot of Zigbee I don't know where to start to debug my issue :(

The bulb does flash twice when the script reaches channel 20, but then turns back on and the script fails. I have switched off the Hue (original) hub and the Zigbee integration on Home Assistant (also powered off its WiFi connected Zigbee bridge).

Any help will be greatly appreciated! Thanks!

pi@mypi:/opt/hue-thief $ sudo python3 hue-thief.py /dev/ttyACM0 --baudrate 115200
Scanning on channel 11
Scanning on channel 12
Scanning on channel 13
Scanning on channel 14
Scanning on channel 15
Scanning on channel 16
Scanning on channel 17
Scanning on channel 18
Scanning on channel 19
Scanning on channel 20
NCP entered failed state. Requesting APP controller restart
Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/bellows/ezsp/protocol.py", line 152, in command
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hue-thief/hue-thief.py", line 135, in <module>
    asyncio.get_event_loop().run_until_complete(steal(args.device, args.baudrate, args.channel))
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/opt/hue-thief/hue-thief.py", line 105, in steal
    await dev.mfglibSendPacket(frame)
  File "/root/.local/lib/python3.9/site-packages/bellows/ezsp/protocol.py", line 152, in command
    return await future
  File "/root/.local/lib/python3.9/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/root/.local/lib/python3.9/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

Question: How do you confirm that the philips hue bulb was reset?

Hi, thanks for this cool little tool!!

I received a used 3rd gen hue color bulb. The thief finds the bulb, it blinks, and asks if it should be reset. After I answer yes, should the bulb give any feedback? I don't get any feedback from the bulb after answering yes. Does that mean the factory reset has failed?

Thanks,
Peter

Failure to start due to assertion error

Using a Go Control stick (Silicon Labs chip), been running with Home Assistant just fine. Run into the following:

root@f2e832ada290:/# python3 hue-thief/hue-thief.py /dev/ttyUSB1 
Exception in callback SerialTransport._read_ready()
handle: <Handle SerialTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.9/site-packages/serial_asyncio/__init__.py", line 119, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.9/site-packages/bellows/uart.py", line 64, in data_received
    self.frame_received(frame)
  File "/usr/local/lib/python3.9/site-packages/bellows/uart.py", line 86, in frame_received
    self.data_frame_received(data)
  File "/usr/local/lib/python3.9/site-packages/bellows/uart.py", line 107, in data_frame_received
    self._application.frame_received(self._randomize(data[1:-3]))
  File "/usr/local/lib/python3.9/site-packages/bellows/ezsp.py", line 169, in frame_received
    assert expected_id == frame_id
AssertionError

Not sure if this is a bellows issue or how hue-thief is trying to use it. Any pointers appreciated!

compatible device

Is there any other device that compatible with hue-thief? Sonoff Bridge or Sonoff USB Stick? Cause the one listed is not available to purchase on my country.

Python asyncio loop argument deprecated

This is early warning of a feature that will be unavailable in Python 3.10

hue-thief/hue-thief.py:15: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  self.q = asyncio.Queue(loop=self.loop)

For examples of how to do asyncio queues now please see https://docs.python.org/3.8/library/asyncio-queue.html#examples

Python 3.10 is not likely to be out before the end of 2021 so there's no rush, but if anyone finds themselves inside this code then its something to consider resolving.

"TypeError: cannot 'yield from' a coroutine object in a non-coroutine generator"

I'm getting the following error when I attempt to run hue-thief using the following command
python3 hue-thief.py /dev/ttyUSB0

Any ideas as to the cause?

hue-thief.py:28: RuntimeWarning: coroutine 'setup' was never awaited
s = yield from util.setup(device, baudrate=57600)
Traceback (most recent call last):
File "hue-thief.py", line 124, in
loop.run_until_complete(steal(sys.argv[1]))
File "/usr/lib/python3.5/asyncio/base_events.py", line 466, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "hue-thief.py", line 28, in steal
s = yield from util.setup(device, baudrate=57600)
TypeError: cannot 'yield from' a coroutine object in a non-coroutine generator

Syntax Error

When I try to run hue-thief I get a syntax error:

File "hue-thief.py", line 21
async def __call__(self, msg, end-'\n', flush=False):
        ^

click.exceptions.ClickException: Unable to start mfglib

Hi,

trying both Linux and WSL, I fail to successfully start hue-thief.

I'm using a Home Assistant SkyConnect and set the Baud Rate in the Device Manager in Windows to 115200 and used the command option to set it there as well.

After a couple of seconds I just get this Traceback:

Traceback (most recent call last):
  File "hue-thief.py", line 135, in <module>
    asyncio.get_event_loop().run_until_complete(steal(args.device, args.baudrate, args.channel))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "hue-thief.py", line 33, in steal
    util.check(res[0], "Unable to start mfglib")
  File "/root/.local/lib/python3.8/site-packages/bellows/cli/util.py", line 150, in check
    raise click.ClickException(message)
click.exceptions.ClickException: Unable to start mfglib

baudrate error

install on Raspbian-Stretch using Nortec HUSBZB-1
i get
Traceback (most recent call last):
File "hue-thief.py, line 124, in
loop.run_until_complete(steal(sys.argv[1]))
File "user/lib/python3.5/asyncio/base_events.py", line 293, in result
rase self._exception
File "user/lib/python3.5/asyncio/tasks.py", line 239, in_step
result = coro.send(none)
File "hue-thief.py", line 28 in steal
s= yield from util.setup(device)
TypeError: setup() missing 1 required positional argument: 'baudrate'

Timeout error during setup

I'm running this on my Ubuntu laptop and get a timeout error during setup:

$  python hue-thief.py /dev/ttyUSB0

Traceback (most recent call last):
  File "/home/stu/.virtualenvs/hue-thief/lib/python3.11/site-packages/bellows/cli/util.py", line 113, in setup
    await s._startup_reset()
  File "/home/stu/.virtualenvs/hue-thief/lib/python3.11/site-packages/bellows/ezsp/__init__.py", line 103, in _startup_reset
    await self.reset()
  File "/home/stu/.virtualenvs/hue-thief/lib/python3.11/site-packages/bellows/ezsp/__init__.py", line 132, in reset
    await self._gw.reset()
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/stu/projects/external/iot/hue/hue-thief/hue-thief.py", line 135, in <module>
    asyncio.get_event_loop().run_until_complete(steal(args.device, args.baudrate, args.channel))
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/stu/projects/external/iot/hue/hue-thief/hue-thief.py", line 28, in steal
    dev = await util.setup(device_path, baudrate)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stu/.virtualenvs/hue-thief/lib/python3.11/site-packages/bellows/cli/util.py", line 117, in setup
    raise click.Abort()
click.exceptions.Abort

$ ❯ lsusb | grep CP210x
Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge

Continuous Loop of exceptions on start

Installing per README.md, I end up in a loo (below) that I need to interrupt.

I've tried it on both Ubuntu 18.04 and on a Pi running Raspbian Desktop 2020-05-27, as well as within docker. I'm trying to use a Nortek HZUSB-1 if that matters.

Scanning on channel 11
Exception running handler
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/bellows/ezsp.py", line 194, in handle_callback
    handler(*args)
  File "hue-thief.py", line 54, in cb
    resp = interpanZll.ScanResp.deserialize(data)[0]
  File "/home/pi/2020-06-19_HueThief/interpanZll.py", line 40, in deserialize
    v, data = field[1].deserialize(data)
  File "/home/pi/.local/lib/python3.7/site-packages/zigpy/types/basic.py", line 287, in deserialize
    item, data = r._itemtype.deserialize(data)
  File "/home/pi/.local/lib/python3.7/site-packages/zigpy/types/basic.py", line 17, in deserialize
    raise ValueError("Data is too short to contain %d bytes" % cls._size)
ValueError: Data is too short to contain 1 bytes
Exception running handler
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/bellows/ezsp.py", line 194, in handle_callback
    handler(*args)
  File "hue-thief.py", line 54, in cb
    resp = interpanZll.ScanResp.deserialize(data)[0]
  File "/home/pi/2020-06-19_HueThief/interpanZll.py", line 40, in deserialize
    v, data = field[1].deserialize(data)
  File "/home/pi/.local/lib/python3.7/site-packages/zigpy/types/basic.py", line 287, in deserialize
    item, data = r._itemtype.deserialize(data)
  File "/home/pi/.local/lib/python3.7/site-packages/zigpy/types/basic.py", line 17, in deserialize
    raise ValueError("Data is too short to contain %d bytes" % cls._size)
ValueError: Data is too short to contain 1 bytes

Thanks for any help and I'll be happy to run any commands or troubleshoot further if you think it's on my end?

Finally, sorry about the weird double-tap; I don't know how I created an issue accidentally.