Consider using netlink.AttributeDecoder about nftables

[stapelberg]

I gave this a shot, but hit an issue: https://godoc.org/github.com/mdlayher/netlink#AttributeDecoder.Uint32 seems to decode an uint32 using the host’s endianness: https://github.com/mdlayher/netlink/blob/3d8cc9af89458b8b6a9ce675d4dfa50a724e9025/nlenc/int.go#L78

For nftables, however, values are stored in big endian most of the time, only the resource id in the extraHeader field is stored in native endian.

A specific place where this causes issues is:

nftables/nftables.go

Line 519 in 121db0b

objectType = binaryutil.BigEndian.Uint32(attr.Data)

With the current code, objectType will be set to 1, but when using ad.Uint32(), it gets set to 16777216.

[mdlayher]

Interesting! Every netlink family I've worked with has used native endianness for attributes so far. I suppose it wouldn't hurt for me to add an option to set the endianness on the decoder!

That said, if nftables uses different endianness in the same attribute lists a lot, this could be a hassle. As a workaround, you could use Do to add special decoding logic for big endian fields.

[stapelberg]

From what I can tell, it’s all big endian for nftables.

Let me know what decision you end up with, and I can give this another shot.

[mdlayher]

I've added a new tunable as of mdlayher/netlink@ff3050a: https://godoc.org/github.com/mdlayher/netlink#AttributeDecoder.

I considered adding it to the constructor as well, but figured it'd be cleaner to allow overriding the default after construction. Let me know what you think.

[stapelberg]

Yeah, this worked. Are you planning to add an AttributeEncoder, too? :)

[mdlayher]

Yeah; starting to think about that problem as well due to the situations I've gotten myself into with: https://github.com/mdlayher/wireguardctrl/blob/master/internal/wgnl/configure_linux.go and its tests. I imagine I'll do something reasonably similar.