Read EPs about zha-toolkit

[mdeweerd]

A lot of functions come from @Adminiuga but here they're document and access to the results is facilitated and I made some updates to follow ZHA evolutions.

'scan_device' indeed limits the scan to endpoints provided by ZHA.
I am adding the 'endpoint' option.

[mdeweerd]

You may be able to cheat like that, but another way would be to also add the cluster parameters and scan the clusters in that list without getting the info from ZHA data.

[mdeweerd]

handle_join is supposed to reinterogate the device as if it newly joined .
The minimal parameter is "ieee" (which can be the entity name)

[mdeweerd]

Unfortunately, the current method relies on ZHA for:

1. Getting the list of endpoints - easy to work around (and done, but not released yet);
2. Knowing the clusters available on the endpoint - more complex as this requires an unimplemented discovery.

So this will be on "standby" for a while until I or somebody else implements the code to discover which clusters are on an endpoint.

[MattWestb]

Is it possible "cheating" ZHA by adding EP and cluster in zigbee.db with SQL editor for the device and toolkit is reading it from the DB and using it ?

[MattWestb]

I was trying version 0.5.5 and the system is still sending on leave but without the flag rejoin setted:

service: zha_toolkit.execute
data:
  command: rejoin
  ieee: 60:a4:23:ff:fe:fd:dc:14

And the sniff is giving the command:

IEEE 802.15.4 Data, Dst: 0x4b2d, Src: 0x0000
ZigBee Network Layer Data, Dst: 0x6b7f, Src: 0x0000
    Frame Control Field: 0x0608, Frame Type: Data, Discover Route: Suppress, Security, Source Route Data
    Destination: 0x6b7f
    Source: 0x0000
    Radius: 30
    Sequence Number: 150
    [Extended Source: SiliconL_ff:fe:0f:54:89 (84:2e:14:ff:fe:0f:54:89)]
    [Origin: 2]
    Source Route, Length: 1
    ZigBee Security Header
ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Leave Request (Cluster ID: 0x0034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 73
ZigBee Device Profile, Leave Request, Device: SiliconL_ff:fe:fd:dc:14
    Sequence Number: 97
    Extended Address: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    .0.. .... = Remove Children: False
    0... .... = Rejoin: False

= the device is deleting its network setting but is not trying rejoining its old network then is not knowing it.
That ending with the device is leaving and deleting all network creantshals and must being reseted for coming back in the network and i losing the possibility getting the device being in the network and also being some seconds in INIT mode = dont working as intended if i not have misunderstanding the function of the command.

My thinking is that the "rejoining" is sending one leave to one device and have rejoining flag setted and not need open the network for joining then the device have not deleting its network setting and only doing on "soft reboot" and going back in the network.

I think only the command await device.zdo.Mgmt_Leave_req(device.ieee, 1) is needed like puddl was writing without open the network and doing other things.

Perhaps i have misunderstand the purpose of the rejoining command you have implantation then please can you implanting one " leave with rejoin" without all other not useful things.

PS: I was adding 3 EP in Zigbee.db and the ZHA can see them. Then adding OnOff cluster on them and i can see the commands from EP 2-4 if have getting the device in INIT mode and sending mode switch to it and its working until i taking the battery out. But i have only manage getting it working OK one time then rejoining is not working as intended.

Thanks in advance for great work !!

[mdeweerd]

When I tested, the rejoin bit was set (sniffer log). So I am quite surprised that it's not the case for you.

I wonder if there is an issue with the wireshark/tshark decoder which is different for your version than the version I have to use on the system where I run it.
The Rejoin bit should be the rightmost bit of the byte.
In wireshark you can verify the value of the byte itself rather than how it's decoded to ensure that its value is not just "1" and badly decoded.

You can also edit the code on your system to make some tests.
I recommend to add or modify a LOGGER.debug message in the function you're modifying to ensure that the edited code was taken into account.

https://github.com/mdeweerd/zha-toolkit/blob/main/custom_components/zha_toolkit/__init__.py#L310-L314

For instance you could change the debug message:

LOGGER.debug("TEST1 %s: leave and rejoin result: %s", src, ieee, res)

or you can add something to event_data if you are listening to the event that you defined (Example: event_done: my_rejoin_event) by adding a line like:

  event_data['debug_info']="TEST1"

And you can of course do both: add event_data and modify/add a debug message.

Normally there is no need to restart HA in this case, and if the debug message/event data changes as you defined it then you confirm that you're running the modified code.

[mdeweerd]

I add that await device.zdo.Mgmt_Leave_req(device.ieee, 1) is called by calling await src.zdo.leave(remove_children=False, rejoin=True): https://github.com/zigpy/zigpy/blob/77dec68ab51a9205535a0b47ae2f6f27f27ec73f/zigpy/zdo/__init__.py#L192-L199

[MattWestb]

OK i trying it.

I was looking on one user sniff from tuya ZBGW that was having the remote ad direct child and its looks like this:

Frame 1: 47 bytes on wire (376 bits), 45 bytes captured (360 bits) on interface \\.\pipe\zboss_sniffer_COM6, id 0
IEEE 802.15.4 Data, Dst: 0xbaaf, Src: 0x0000
    Frame Control Field: 0x8861, Frame Type: Data, Acknowledge Request, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 249
    Destination PAN: 0x8e35
    Destination: 0xbaaf
    Source: 0x0000
    [Extended Source: SiliconL_ff:fe:85:ee:07 (84:71:27:ff:fe:85:ee:07)]
    [Origin: 1]
    [Ack In: 2]
ZigBee Network Layer Command, Dst: 0xbaaf, Src: 0x0000
    Frame Control Field: 0x1209, Frame Type: Command, Discover Route: Suppress, Security, Extended Source Command
    Destination: 0xbaaf
    Source: 0x0000
    Radius: 1
    Sequence Number: 97
    Extended Source: SiliconL_ff:fe:85:ee:07 (84:71:27:ff:fe:85:ee:07)
    ZigBee Security Header
    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..1. .... = Rejoin: True
        .1.. .... = Request: True
        0... .... = Remove Children: False



Frame 3: 47 bytes on wire (376 bits), 45 bytes captured (360 bits) on interface \\.\pipe\zboss_sniffer_COM6, id 0
IEEE 802.15.4 Data, Dst: 0x0000, Src: 0xbaaf
    Frame Control Field: 0x8861, Frame Type: Data, Acknowledge Request, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 158
    Destination PAN: 0x8e35
    Destination: 0x0000
    Source: 0xbaaf
    [Extended Source: SiliconL_ff:fe:f6:fc:e2 (60:a4:23:ff:fe:f6:fc:e2)]
    [Origin: 3]
    [Ack In: 4]
ZigBee Network Layer Command, Dst: Broadcast, Src: 0xbaaf
    Frame Control Field: 0x1209, Frame Type: Command, Discover Route: Suppress, Security, Extended Source Command
    Destination: 0xfffd
    Source: 0xbaaf
    Radius: 1
    Sequence Number: 118
    Extended Source: SiliconL_ff:fe:f6:fc:e2 (60:a4:23:ff:fe:f6:fc:e2)
    ZigBee Security Header
    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..1. .... = Rejoin: True
        .0.. .... = Request: False
        0... .... = Remove Children: False

And ZHA:

Frame 4: 119 bytes on wire (952 bits), 119 bytes captured (952 bits) on interface \Device\NPF_Loopback, id 0
Null/Loopback
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 17754, Dst Port: 17754
ZigBee Encapsulation Protocol, Channel: 16, Length: 55
IEEE 802.15.4 Data, Dst: 0x0a3e, Src: 0xc203
    Frame Control Field: 0x8841, Frame Type: Data, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 155
    Destination PAN: 0xc005
    Destination: 0x0a3e
    Source: 0xc203
    [Extended Source: SiliconL_ff:fe:a2:2c:6b (bc:33:ac:ff:fe:a2:2c:6b)]
    [Origin: 4]
    TI CC24xx-format metadata: FCS OK
ZigBee Network Layer Data, Dst: 0x0a3e, Src: 0x0000
    Frame Control Field: 0x0248, Frame Type: Data, Discover Route: Enable, Security Data
    Destination: 0x0a3e
    Source: 0x0000
    Radius: 29
    Sequence Number: 65
    [Extended Source: SiliconL_ff:fe:0f:54:89 (84:2e:14:ff:fe:0f:54:89)]
    [Origin: 1]
    ZigBee Security Header
ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Leave Request (Cluster ID: 0x0034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 151
ZigBee Device Profile, Leave Request, Device: SiliconL_ff:fe:fd:dc:14
    Sequence Number: 202
    Extended Address: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    .0.. .... = Remove Children: False
    0... .... = Rejoin: False



Frame 5: 111 bytes on wire (888 bits), 111 bytes captured (888 bits) on interface \Device\NPF_Loopback, id 0
Null/Loopback
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 17754, Dst Port: 17754
ZigBee Encapsulation Protocol, Channel: 16, Length: 47
IEEE 802.15.4 Data, Dst: 0xc203, Src: 0x0a3e
    Frame Control Field: 0x8841, Frame Type: Data, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 114
    Destination PAN: 0xc005
    Destination: 0xc203
    Source: 0x0a3e
    [Extended Source: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)]
    [Origin: 5]
    TI CC24xx-format metadata: FCS OK
ZigBee Network Layer Data, Dst: 0x0000, Src: 0x0a3e
    Frame Control Field: 0x2248, Frame Type: Data, Discover Route: Enable, Security, End Device Initiator Data
    Destination: 0x0000
    Source: 0x0a3e
    Radius: 30
    Sequence Number: 45
    ZigBee Security Header
ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Leave Response (Cluster ID: 0x8034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 109
ZigBee Device Profile, Leave Response, Status: Success
    Sequence Number: 202
    Status: Success (0)



Frame 6: 111 bytes on wire (888 bits), 111 bytes captured (888 bits) on interface \Device\NPF_Loopback, id 0
Null/Loopback
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 17754, Dst Port: 17754
ZigBee Encapsulation Protocol, Channel: 16, Length: 47
IEEE 802.15.4 Data, Dst: 0xc203, Src: 0x0a3e
    Frame Control Field: 0x8841, Frame Type: Data, PAN ID Compression, Destination Addressing Mode: Short/16-bit, Frame Version: IEEE Std 802.15.4-2003, Source Addressing Mode: Short/16-bit
    Sequence Number: 115
    Destination PAN: 0xc005
    Destination: 0xc203
    Source: 0x0a3e
    [Extended Source: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)]
    [Origin: 5]
    TI CC24xx-format metadata: FCS OK
ZigBee Network Layer Command, Dst: Broadcast, Src: 0x0a3e
    Frame Control Field: 0x1209, Frame Type: Command, Discover Route: Suppress, Security, Extended Source Command
    Destination: 0xfffd
    Source: 0x0a3e
    Radius: 1
    Sequence Number: 46
    Extended Source: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    ZigBee Security Header
    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..0. .... = Rejoin: False
        .0.. .... = Request: False
        0... .... = Remove Children: False

One different thing that tuya was sending the command in the network layer and not in the application layer but i think it shall not being any problem if getting the rejoin flag OK

[MattWestb]

In the tuya ZBGW sniff the device is sending beacons and rejoining in some millisecond after receiving command and broadcasting left to the network and is back with one device acuntsment.

ZHA (not the last update) is only silent after left and wireshark is showing all OK with the rejoin flag if looking on both sniffs.

[MattWestb]

One problem i running HA 2021.12.X and i dont have the last commit in my container so i think i cant doing your patching so easy :-((

[mdeweerd]

One is ZDO cluster 0x0004 (Leave) and the other is ZDO cluster 0x0034 (Leave Request).

I retested with a TI Samplelight:

    Leave Request (Cluster ID: 0x0034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 33
ZigBee Device Profile, Leave Request, Device: TexasIns_00:01:dd:7a:d7
    Sequence Number: 150
    Extended Address: TexasIns_00:01:dd:7a:d7 (00:12:4b:00:01:dd:7a:d7)
    .0.. .... = Remove Children: False
    1... .... = Rejoin: True

The sample light does not support it (it's an old TI stack), but that doesn't matter:

    Leave Response (Cluster ID: 0x8034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 33
ZigBee Device Profile, Leave Response, Status: Not Supported
    Sequence Number: 150
    Status: Not Supported (132)
Data (3 bytes)

0000  00 00 00                                          ...
    Data: 000000
    [Length: 3]

[mdeweerd]

Maybe a restart was required to have the latest version for rejoin & handle_rejoin. I moved those to misc.py so that their core code is reloaded on each call.

You'll have to restart after updating this.

[MattWestb]

im getting:

  File "/config/custom_components/zha_toolkit/__init__.py", line 280, in command_handler_rejoin
    await misc.rejoin(*args, **kwargs)
AttributeError: module 'custom_components.zha_toolkit.misc' has no attribute 'rejoin'

Then using:

service: zha_toolkit.execute
data:
  command: rejoin
  ieee: 60:a4:23:ff:fe:fd:dc:14

Is i doing the parameters wrong or is they changed ?

PS I wad deleting all in the folder before restart HA so the old cash is gone.

[mdeweerd]

Ok, fixed that. You should be able to update that and use the new code without ha core restart. I forgot to rename the rejoin method after moving it.

[MattWestb]

Now i dont getting the error but i cant getting it doing one "leave with joining flag".

Was trying little more low level and moved the TS004f to my IKEA controllers network from Sirlabs WSTK with one cooked Zigbee Control Bridge and testing ZDO leave command:

zdo leave 
Usage:
<int>: 123 or 0x1ABC
<string>: "foo" or {0A 1B 2C}

zdo leave (args) 
   <uint16_t>  Target node ID
   <uint8_t>  Remove children
   <uint8_t>  Rejoin after leave

 - Send a ZDO Management Leave command to the target device.

Sending: zdo leave 0x56ff 0 1
and getting:

ZBCB10>Leave Request: 0x00
ZBCB10>Processing message: len=12 profile=0000 cluster=0013
RX: ZDO, command 0x0013, status: 0x00
Device Announce: 0x5171

I think the cluster 13 is one cosmetick thing but the device is leaving and rejoining the network without have opening for joining as is shall doing and HA is logging:

2022-01-26 11:02:56 INFO (MainThread) [zigpy.application] Device 0x5171 (60:a4:23:ff:fe:fd:dc:14) joined the network
2022-01-26 11:02:56 INFO (MainThread) [zigpy.application] Device 0x5171 (60:a4:23:ff:fe:fd:dc:14) joined the network
2022-01-26 11:02:56 INFO (MainThread) [bellows.zigbee.application] ZDO Device announce: 0x5171, 60:a4:23:ff:fe:fd:dc:14
2022-01-26 11:02:56 INFO (MainThread) [zigpy.application] Device 0x5171 (60:a4:23:ff:fe:fd:dc:14) joined the network
2022-01-26 11:02:56 DEBUG (MainThread) [zigpy.zdo] [0x5171:zdo] ZDO request ZDOCmd.Device_annce: [0x5171, 60:a4:23:ff:fe:fd:dc:14, 128]

So Silabs Zigbee stack (6.10.1 GA build 216) is working OK and the device also.
But i cant understand way ZHA cant doing one simple ZDO command with 3 parameters in the right way ;-(((

All made from the WSTK looks like puddly was writing in zigpy/zigpy#831 (comment) but i dont have the knowledge implanting it in working code for sending it to one device in ZHA.

[mdeweerd]

In my case the leave (with rejoin) command has the flag set correctly.
So it may be a bellows (zigpy) limitation? I have a znp (CC2531) radio.

I would still have a look at the byte (not the decoded bits) that is sent over the air.

And maybe activate more debug message for the modules involved.

[mdeweerd]

But i cant understand way ZHA cant doing one simple ZDO command with 3 parameters in the right way

The different parameters are encoded in one byte:
https://github.com/zigpy/zigpy/blob/77dec68ab51a9205535a0b47ae2f6f27f27ec73f/zigpy/zdo/__init__.py#L192-L199

The parameter types of the ZDO command are defined here:
https://github.com/zigpy/zigpy/blob/a19a2a063d5edacb46722aeb56d538badd37fcc0/zigpy/zdo/types.py#L575

Also, the log that you shared above is shown command 0x04, I do not see where this is defined as a zigbee command.

    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..1. .... = Rejoin: True
        .0.. .... = Request: False
        0... .... = Remove Children: False

[MattWestb]

With Zigbee Control Bridge sending zdo leave 0x56ff 0 1 its sending and the device is repaying with:

IEEE 802.15.4 Data, Dst: 0x56ff, Src: 0x881d
ZigBee Network Layer Data, Dst: 0x56ff, Src: 0x881d
    Frame Control Field: 0x0208, Frame Type: Data, Discover Route: Suppress, Security Data
    Destination: 0x56ff
    Source: 0x881d
    Radius: 30
    Sequence Number: 20
    [Extended Source: SiliconL_ff:fe:c3:90:1a (08:6b:d7:ff:fe:c3:90:1a)]
    [Origin: 4]
    ZigBee Security Header
ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Leave Request (Cluster ID: 0x0034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 15
ZigBee Device Profile, Leave Request, Device: 00:00:00_00:00:00:00:00
    Sequence Number: 1
    Extended Address: 00:00:00_00:00:00:00:00 (00:00:00:00:00:00:00:00)
    .0.. .... = Remove Children: False
    1... .... = Rejoin: True

IEEE 802.15.4 Data, Dst: 0x881d, Src: 0x56ff
ZigBee Network Layer Command, Dst: Broadcast, Src: 0x56ff
    Frame Control Field: 0x1209, Frame Type: Command, Discover Route: Suppress, Security, Extended Source Command
    Destination: 0xfffd
    Source: 0x56ff
    Radius: 1
    Sequence Number: 132
    Extended Source: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    ZigBee Security Header
    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..1. .... = Rejoin: True
        .0.. .... = Request: False
        0... .... = Remove Children: False

HA:

service: zha_toolkit.execute
data:
  command: rejoin
  ieee: 60:a4:23:ff:fe:fd:dc:14

With disabled permit joining in your code.

IEEE 802.15.4 Data, Dst: 0x56ff, Src: 0x881d
ZigBee Network Layer Data, Dst: 0x56ff, Src: 0x0000
    Frame Control Field: 0x0208, Frame Type: Data, Discover Route: Suppress, Security Data
    Destination: 0x56ff
    Source: 0x0000
    Radius: 29
    Sequence Number: 42
    [Extended Source: SiliconL_ff:fe:d5:92:9c (58:8e:81:ff:fe:d5:92:9c)]
    [Origin: 1]
    ZigBee Security Header
ZigBee Application Support Layer Data, Dst Endpt: 0, Src Endpt: 0
    Frame Control Field: Data (0x40)
    Destination Endpoint: 0
    Leave Request (Cluster ID: 0x0034)
    Profile: ZigBee Device Profile (0x0000)
    Source Endpoint: 0
    Counter: 223
ZigBee Device Profile, Leave Request, Device: SiliconL_ff:fe:fd:dc:14
    Sequence Number: 43
    Extended Address: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    .0.. .... = Remove Children: False
    0... .... = Rejoin: False


ZigBee Encapsulation Protocol, Channel: 15, Length: 47
IEEE 802.15.4 Data, Dst: 0x881d, Src: 0x56ff
ZigBee Network Layer Command, Dst: Broadcast, Src: 0x56ff
    Frame Control Field: 0x1209, Frame Type: Command, Discover Route: Suppress, Security, Extended Source Command
    Destination: 0xfffd
    Source: 0x56ff
    Radius: 1
    Sequence Number: 187
    Extended Source: SiliconL_ff:fe:fd:dc:14 (60:a4:23:ff:fe:fd:dc:14)
    ZigBee Security Header
    Command Frame: Leave
        Command Identifier: Leave (0x04)
        ..0. .... = Rejoin: False
        .0.. .... = Request: False
        0... .... = Remove Children: False

Is it one addressing problem the Silabs stack is using short address and you long address ?Perhaps ZHA /Zigpy / bllows is truncating the command if using long address in this command so missing the rejoin flag.

[MattWestb]

One PM the last response is from the remote but catched from its parent that is doing the broadcast of the left network message (the end device is sending it in unicat to its parent and the parent is doing the broadcast).

[mdeweerd]

It works for me with zigpy/zigpy-znp, so I think there is a problem somewhere in zigpy/bellows .

[MattWestb]

Sorry for bombing you with bad requests and annoying things !!
The leave flags was wrong and Puddly have getting it working by adding one class with the right bits setted but its not implanted in Zigpy zigpy/zigpy#831 (comment).

I having problem getting it working in the quirk (not knowing how to coding it) but i think the flags is OK in the last posting.

[mdeweerd]

It seems to be complex because the Zigbee specifciation is counterintuitive resulting in implementations that are "not compliant" with zigpy*.

I updated the code in misc.py which still works as before with the default code, but that you can easily adapt + you have examples.

You can set method to 0,1,2,3 or 4 and it will execute the Leave with a different method and/or paramters.
For instance method 0 and 2 use the same function but put the rejoin bit on the opposite ends.
Method 4 sets the children remove bit in the lower bits of the byte, but not in the upper bit.

You could change the code to send 0x81 to avoid setting the children remove bit.

You can update your zha-toolkit without reloading and modify and test the code without reloading.

Hopefully you find the right combination.
Then this code may need to depend on HA Version, and radio type to select the best method.

    method=1
    res = "Not executed, no valid 'method' defined in code"
    if method==0:
        # Works on HA 2021.12.10 & ZNP - rejoin is 1:
        res = await src.zdo.request(0x0034, src.ieee, 0x01)
    elif method==1:
        # Works on ZNP but apparently not on bellows:
        res = await src.zdo.leave(remove_children=False, rejoin=True)
    elif method==2:
        # Results in rejoin bit 0 on ZNP
        LOGGER.debug("Using Method 2 for Leave")
        res = await src.zdo.request(0x0034, src.ieee, 0x80)
    elif method==3:
        # Results in rejoin and leave children bit set on ZNP
        LOGGER.debug("Using Method 3 for Leave")
        res = await src.zdo.request(0x0034, src.ieee, 0xFF)
    elif method==4:
        # Results in rejoin and leave children bit set on ZNP
        LOGGER.debug("Using Method 4 for Leave")
        res = await src.zdo.request(0x0034, src.ieee, 0x83)

[MattWestb]

Great thanks !!

I think the problem is the first implementation was made with TI coordinators that is doing one "workaround" and was not being tested with other ones so the wrong implementation have always being in Zigpy.

I hope ZHA is fixing the bug so users can using it as "normal ZHA service" but A is little tricky then saying its somthing wrong and hi dont like doing it. Our P is great and is helping very much also user like my with no coding experience.

I have getting the quirk sending one leave after 15 seconds the quirk i loades and have the joining only open for 10 seconds and the device is rejoining nicels only making one beacon request and jumping back to the network.

I have deleting all cluster in the quirk and only using basc and identify for not getting long init time then testing and shall adding power config cluster for the next test.

Then getting it working its 2 ways implanting the procedure. 1 in the normal quirk then the device only have one active EP and running the script and getting 4 EP. 2 Implanting it in your "collection" its better but then users must installing it so its not out of the box.

For the first way is also possible making one local quirk doing the magick and then using the ZHA standard one. Or implanting triggering the script with with one cluster command.

Great thanks for helping and supporting !!!

I reporting back then i have getting little more good function and perhaps stealing little of your code ;-)

[MattWestb]

Ops how is the syntax for the updated command in service XML ?

[mdeweerd]

Same - you can use the UI mode if you want - that's what I do to select a device and the command, and then I switch back to yaml mode.

service: zha_toolkit.execute
data:
  command: rejoin
  ieee: light.texasinstruments_ti_samplelight_d77add01_level_light_color_on_off
  event_done: zha_done

[MattWestb]

Setting method = 4 the code and executing the the command and the system is kicking the device 3 times and the last with timeout.

Timeout i can understand then the device is 3 time offline and cant answer the request but its not good then its not possible sending commands to the device for around 20 seconds.

2022-01-31 20:44:32 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.execute (c:5458cada15b151ffba28851ad168f274): command=rejoin, ieee=60:a4:23:ff:fe:fd:dc:14, event_done=zha_done>
.
.
2022-01-31 20:44:34 INFO (MainThread) [zigpy.application] Device 0x2203 (60:a4:23:ff:fe:fd:dc:14) joined the network
. 
.
2022-01-31 20:44:43 INFO (MainThread) [zigpy.application] Device 0x2203 (60:a4:23:ff:fe:fd:dc:14) joined the network
. 
.
2022-01-31 20:44:53 INFO (MainThread) [zigpy.application] Device 0x2203 (60:a4:23:ff:fe:fd:dc:14) joined the network
.
.
2022-01-31 20:45:02 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'ieee_org': '60:a4:23:ff:fe:fd:dc:14', 'ieee': '60:a4:23:ff:fe:fd:dc:14', 'command': 'rejoin', 'start_time': '2022-01-31T19:44:32.876572+00:00', 'errors': ["DeliveryError('[0x2203:0:0x0034]: Message send failure')"], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'code': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': 0, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': 'zha_done', 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': False}
2022-01-31 20:45:02 DEBUG (MainThread) [custom_components.zha_toolkit] Fire zha_done -> {'ieee_org': '60:a4:23:ff:fe:fd:dc:14', 'ieee': '60:a4:23:ff:fe:fd:dc:14', 'command': 'rejoin', 'start_time': '2022-01-31T19:44:32.876572+00:00', 'errors': ["DeliveryError('[0x2203:0:0x0034]: Message send failure')"], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'code': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': 0, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': 'zha_done', 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': False}
2022-01-31 20:45:02 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_toolkit.execute (c:5458cada15b151ffba28851ad168f274): command=rejoin, ieee=60:a4:23:ff:fe:fd:dc:14, event_done=zha_done>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1511, in catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1530, in _execute_service
    await handler.job.target(service_call)
  File "/config/custom_components/zha_toolkit/__init__.py", line 141, in toolkit_service
    raise handler_exception
  File "/config/custom_components/zha_toolkit/__init__.py", line 94, in toolkit_service
    await handler(
  File "/config/custom_components/zha_toolkit/__init__.py", line 280, in command_handler_rejoin
    await misc.rejoin(*args, **kwargs)
  File "/config/custom_components/zha_toolkit/misc.py", line 137, in rejoin
    res = await src.zdo.request(0x0034, src.ieee, 0x83)
  File "/usr/local/lib/python3.9/site-packages/zigpy/device.py", line 287, in request
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: [0x2203:0:0x0034]: Message send failure

I have cutting away much then pressing one key on the remote for keeping it awake for trying not getting timeout.

[mdeweerd]

Probably the end device did not reply to the leave+rejoin request, letting zigpy think the command timed out. The logs provide only a partial view on what happens, sniffing will show if the rejoin command was really sent. If it is sent, then the reply from the device is missing or not understood.

I updated (not released) https://github.com/mdeweerd/zha-toolkit/blob/main/custom_components/zha_toolkit/misc.py#L158-L161 . That code uses the "tries" parameter that can be provided in the service call - the implementation depends on whether the underlying function is itself retryable or not.

As far as I understand from the code, the default value for the retryable command is 1 try .

I tries on my TI Sample light which replies with extra zeros in the payload for which I see a ValueError in the log, but that conclude to a timeout, the result of the zha_done event (using the updated code):

{
    "event_type": "zha_done",
    "data": {
        "ieee_org": "light.texasinstruments_ti_samplelight_d77add01_level_light_color_on_off",
        "ieee": "00:12:4b:00:01:dd:7a:d7",
        "command": "rejoin",
        "start_time": "2022-01-31T21:52:42.209980+00:00",
        "errors": [
            "TimeoutError()",
            "TimeoutError()",
            "TimeoutError()"
        ],
        "params": {
            "cmd_id": null,
            "endpoint_id": null,
            "cluster_id": null,
            "attr_id": null,
            "attr_type": null,
            "attr_val": null,
            "code": null,
            "min_interval": null,
            "max_interval": null,
            "reportable_change": null,
            "dir": 0,
            "manf": null,
            "tries": 3,
            "expect_reply": true,
            "args": [],
            "state_id": null,
            "state_attr": null,
            "allow_create": false,
            "event_success": null,
            "event_fail": null,
            "event_done": "zha_done",
            "read_before_write": true,
            "read_after_write": true,
            "write_if_equal": false
        },
        "success": false,
        "result": null
    },
    "origin": "LOCAL",
    "time_fired": "2022-01-31T21:53:27.407533+00:00",
    "context": {
        "id": "70b30d90f2b19806e51496a6993813dc",
        "parent_id": null,
        "user_id": null
    }
}

Service call:

service: zha_toolkit.execute
data:
  command: rejoin
  ieee: light.texasinstruments_ti_samplelight_d77add01_level_light_color_on_off
  tries: 3
  event_done: zha_done

[mdeweerd]

@MattWestb I am leaving this open because of the retries (that was updated) and to confirm the command to run in case of bellows (if I understook correctly, method 4).

[MattWestb]

OK i was not having time trying it the have updating HA and digging in sniffs for 2 issues.
And i shall testing if the leave lag is fixed in Zigpy so we can getting done.

Thanks for great coding and support !!

[MattWestb]

I was testing reading attributes on my TS004F and was putting in all information i have like this:

service: zha_toolkit.attr_read
data:
  endpoint: 1
  cluster: 6
  attribute: 32772
  ieee: 60:a4:23:ff:fe:fd:dc:14

But getting error Cluster 0x0006 not found for '60:a4:23:ff:fe:fd:dc:14', endpoint 1 = false, plus 2 more errors all with KeyError: 6.

I think i need attr_type: 0x41 but i dont knowing witch parameter i need to putting in.
The device signature is:

{
  "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.EndDevice: 2>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress: 128>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0820",
      "in_clusters": [
        "0x0000",
        "0x0001",
        "0x0003",
        "0x0004",
        "0x1000"
      ],
      "out_clusters": [
        "0x0003",
        "0x0004",
        "0x0005",
        "0x0006",
        "0x0008",
        "0x000a",
        "0x0019",
        "0x0300",
        "0x1000"
      ]
    }
  },
  "manufacturer": "_TZ3000_xabckq1v",
  "model": "TS004F",
  "class": "ts004f3B.TuyaSmartRemote004F1A"
}

And the OnOff is one out cluster.

[mdeweerd]

Only in_clusters have readable attributes. out_clusters mostly send commands to other devices and in some cases can also receive data (attribute reports) from other clusters of the same type on other devices (for example, for remote temperature sensing).

I think that your device is some kind of remote control, so it's sending on/off commands (and other commands) to devices that receive them. So the out_cluster 0x0006 is sending commands to an in_cluster on a light for instance.

There is a key error because the cluster can not be found in the list of in_clusters. There's still a lot of opportunity to improve zha-toolkit with error reporting to the user.

[mdeweerd]

(Also: when the topic changes, it's better to create a new one - maybe it's still about reading EEPs according to the title, but it was more about rejoining, and now its about another type of issue.

I am not in favor of creating a topic for each "microissue".

Multiple issues makes the thread less long and makes it easier for other visitors to read and understand the thread - even find an old thread about the topic they have an issue with.

[MattWestb]

OK i open one new issue for not making one bigger mess that is now.
If you like closing it its OK but i shall making more testy later.

By the way is the command supported for requesting the device active EPs ?
Its normal used the joining the device so knowing witch EP shall being initiated and getting its cluster on.

Its little on the topic then some devices (tuya) is hiding some EP and cant scanning them if not defining them in one quirk.

I think its little to much for your nice tool but if possible reading active EP and then all cluster on them and then doing the scan so getting all possible information from the device is one ultimate method.

I open one new issue for the problem maker.

[mdeweerd]

Closing this because there is a new Issue to follow up on the new subject.