I2C reads always returning 0xFF (as of 2.7.4+rev1) about balena-raspberrypi

[agherzan]

Thanks, @ericbarch for investigating this. It was hard on our side as we lack the specific hardware. Update us if we can help in any way.

[floion]

Hi,
can you write down a testcase for this issue please?

[ericbarch]

Of course 😀

We have been testing against an Atlas Scientific RTD module: https://www.atlas-scientific.com/product_pages/circuits/ezo_rtd.html connected to a Raspberry Pi 3.

Testing under 2.9.7+rev1 (same result w/ 2.7.4+):

balena run --rm -it --privileged resin/armv7hf-debian:stretch /bin/bash

apt update && apt install -f i2c-tools

root@7c4a31740ab4:/# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

root@7c4a31740ab4:/# i2cget -y 1 104 0 b
0xff

Testing under 2.7.2+rev1:

docker run --rm -it --privileged resin/armv7hf-debian:stretch /bin/bash

apt update && apt install -f i2c-tools

root@3b5b9cf66681:/# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

root@3b5b9cf66681:/# i2cget -y 1 104 0 b
0x05

Note that this first register (address 0) is the device identification register. It should always return 0x05.

[ericbarch]

After performing some additional testing, it appears as though the new releases only cause issues when communicating with our Atlas Scientific I2C devices. We also tested against a BME280 sensor and did not notice any issues when upgrading to the latest resinOS.

I grabbed a few captures via our logic analyzer of the Atlas Scientific transactions.

Note that the top (BAD) test was performed on resinOS 2.12.3+rev1. The bottom (GOOD) test was performed on resinOS 2.7.2+rev1.

It appears as though the newer resinOS builds are attempting to perform a repeated start. The Atlas Scientific module we are using does not support repeated start.

In fact, if you look closely at the (BAD) analog waveforms (below the digital waveforms), you can see a slight voltage increase in SCL after SDA goes high (indicating that the Pi is attempting to perform a repeated start). Because the change is ~150mV, this would seem to indicate that the slave device is holding the clock low to prevent the repeated start.

Is it possible the underlying I2C drivers have changed to perform repeated starts? And if so, is there any way to prevent this?

[ericbarch]

We've now tracked this issue down to a change in the official RPi kernel. It appears that when the bcm2835 I2C module was made the default (with repeated starts enabled), it broke compatibility with our Atlas Scientific board. This change can be found here:

raspberrypi/linux@fe92705

Will continue to investigate on our own. But closing as this is not a resinOS specific issue.