Optimize data payload about bthome.io

[balloob]

The data types are not fixed, they are preferred.

[Ernst79]

A fixed byte length will limit the temperature range to -327,68 - + 327,68°C, which can be insufficient for some users/devices. E.g. BBQ sensors. A workaround is adding more object id's for temperature, e.g. one with sint16, with factor 0.1, which will give -3276,8 - + 3276,8°C as range.

However, keep in mind that we only have 255 object id's.

[triochi]

You are not limited to integer values. There are other formats too. Like for temperature - you can use instead FP16 https://en.wikipedia.org/wiki/Half-precision_floating-point_format

[Ernst79]

yes, but the first byte is telling something about the length, but also about the format (uint/sint/float/string). If we remove that first byte, we have to stick to one format, and one length.

[triochi]

Yes, this is the proposal. For those types that are expected to have more than one scale factors - there are the floating point formats. Well for the integer types - there no other options except for defining few more types: counter_8, counter_16, counter_24 for example.

[KevinCathcart]

If the length prefixed format is removed that would cause forward compatibility issues. Any receiving device that sees an unknown object_id would need to discard the rest of the message, since they have no way of knowing how long it is, to be able to skip the object and read any remaining data they can understand.

It would be preferable for receiving devices to be able to tolerate and ignore unknown object ids. Obviously they won't be able to handle new major versions of the protocol, but being able to "handle" (by ignoring) minor new additions seems desirable.

This is not to say there is no room for criticism here. For example, the spec does not specify how to interpret floats.[1] Similarly, it does not make a ton of sense to be able to specify a size of 0 bytes (plus length/data-type byte), since the object id will take at least one byte [2], and even a size of 1 seems odd, since the only sensible use case I can see for that is the string data type where a zero byte string can be entirely valid. But, for example, an int0, or float0 would not make much sense. Even criticize some of the ranges. For example the two decimal digits for temperate readings seem questionable[3].

But I don't think the the redundancy of specifying the data size outweighs the downsides of preventing forward compatibility if it were not present.

Footnotes:
[1] Like there are well-specified formats in IEEE 754 (ISO/IEC 60559) for float16, float32, float64, float128, and float256. But any other sizes? shrug who can say, the spec also fails to clarify if a value sent of a float should have the indicated scaling factor applied, or if in that case it should be interpreted with a scale of 1, since floats can specify their own scale.

[2] Technically the current scheme could support extending it to more than one byte if needed, for example by having 0xff mean "read next byte, and add 254 to get the overall object id", or by using a varint encoding of some form. If we assume readers that don't know the object id will simply skip over it, such approaches causes no real problems.

[3] I don't see terribly many sensors with sufficient precision to support more than one decimal digit. Like plenty may be connected to ADCs that can cover more digits, but the design just won't support those digits being meaningful, rather than just random noise. And that is just precision. Over on the accuracy front, if you read the manuals, you will often find a stated accuracy of ±2°C to ±1°C (maybe ±0.5°C at best). Sensors that can have better accuracy (like ±0.1°C) are not super hard to find, but tend to be at least slightly more expensive than common general purpose sensors, have more limited ranges, etc, so they are not particularly common in practice.

[Ernst79]

Thanks for this detailed analysis/response.

You have a very good point about the issue that will arise if a BTHome receiver, like Home Assistant, is running a lower (minor) BTHome release than a sensor, and the sensor sends a new object id. Hadn't thought about that. I had already removed the length and format byte in the V2 branch, but we might need to reconsider.

I see one alternative option (alternative to bringing the length and format byte back in) to remain compatibility with minor BTHome changes. If we prescribe that the object_ids have to be in a low to high order in the BLE advertisement (service data), it will always first parse the lower object id's. When it arrives at an unknown object id (with a higher object id), we can tell the parser to stop parsing the rest of the BLE advertisement. New object id's have to be added with a higher number in minor updates, but that is something that is already being done.

About your footnotes
[1] BTHome only supports float16 (2 bytes), float32 (4 bytes) and float64(8 bytes), which can be decoded with struct.unpack e, f and d. It will just multiply it with a factor, but 1 makes most sense for floats, I agree

[2] Yes, we also thought of that. for now, we will go to object id 127, and will see how fast it goes. If we reach this number of objects very fast, we can also consider using the first 127 object ids as single byte ids, and the remaining 128 bytes as double byte ids with 128 * 256 = 32768 additional bytes. Should be more than enough. But we can make this decision later, when we reach 0x7F = 127.

[3] That is totally right, but with 1 byte and a factor 0.1, we can reach temperatures between -12.8 and +12.8°C, which is too limited for most sensors. With factor 1, you can reach -128 - + 128°C, but that is too little precision. We can consider adding a 2 byte temperature with factor 0.1, but one can also just send it with 2 digit precision and only send 25.10, 25.20, 25.30 etc. and let the receiver deal with it. I think e.g. (not 100% sure) Home Assistant will display this as 25.1, 25.2, 25.3. However, there is no objection adding an object id for temperature with factor 0.1.

[Ernst79]

Added a warning when not using a numerical order (from low to high) for the object id in the payload and added a break when it finds a wrong object id is this PR (for BTHome V2).

Bluetooth-Devices/bthome-ble#18

[KevinCathcart]

Yeah, specifying numerical order and ignoring any later numbers makes perfectly good sense as a forward compatibility strategy without needing a length prefix for every item.

My comments were mostly about the protocol as currently specified (I.e. that which is in this repository), rather than what the reference implementation supports. The v1 protocol as specified would permit other float sizes, even if the reference implementation would reject them. With the changes in the not yet fully specified v2, those concerns would be mostly mooted, although the specification probably should mention how to interpret the values.

For temperature, I certainly agree that 1 byte simply cannot provide both reasonable precision and range at the same time. It is also a fair enough point that once you go to two bytes you totally can afford to support two decimal places of precision, since that still leaves plenty of range for basically everything except kilns/metal working furnaces. (-327.68°C to 327.67°C covers the range from absolute zero, to past normal oven temperature ranges). And it is not like no temperature sensors have sufficient precision for the second decimal to contain meaningful data. Meaningful precision can (and typically does) exceed accuracy (e.g. many thermometers specified as being accurate to ±2°C still show .0 or .5, and the .5 value still refers to a temperature greater than the .0 one even if the whole might be off by 1 degree.)

At the same time it is probably not unreasonable for the specification to non-normatively mention that the precision available for some object ids may exceed what a sensor can reasonably provide, and in that case the sensor should probably provide a value rounded to its meaningful level of precision. (Obviously it is up to the device to choose whether to err on the side of including some random noise to ensure all meaningful precision is captured, as is common on some scientific devices, or choosing to avoid showing random noise, at possible loss of some meaningful precision, as most consumer devices tend to do.)

[Ernst79]

The PR for the V2 release is #22, in case you are interested.

[Ernst79]

@triochi As you can see in the V2 PR's for the code and docs, we have adopted your suggestion and will remove the first byte. We will now use a fixed length and format. If someone needs a different format or length, we can add an extra data type (like for humidity and battery).

#22
Bluetooth-Devices/bthome-ble#13

[Ernst79]

V2 has been released.