NeoPXL8 (& HDR) strandtest hangs USB serial on RP2040 SCORPIO about adafruit_neopxl8

[PaintYourDragon]

Thanks for your detective work on this, and thanks for your patience as I’ve been cycling through other projects. I’ve been able to reproduce the problem, and had encountered something similar on another project (though receiving rather than sending serial data). Been looking into it, have not yet found the culprit…not even certain yet whether it’s NeoPXL8 breaking Serial, or the other way ’round. Testing continues…

[PaintYourDragon]

Question: does the choice of USB stack (from Arduino “Tools” menu) make any difference? It might be a fluke, but I’m seeing different outcomes for each (at random point when the phenomenon occurs):
TinyUSB: full lockup — no animation, no serial output
Pico SDK: animation stalls for about 1 second, then resumes, serial output stops and never recovers

[PaintYourDragon]

Just did a new release 1.2.4, should percolate through the Arduino Library Manager within a couple hours. Seems like it might’ve simply been related to GPIO drive strength. Let me know if this fixes things for you! Thanks.

[digitalcircuit]

For anyone following this issue here, after waiting several days I also posted about the problem on the official Adafruit forums, where I've been continuing some troubleshooting:
https://forums.adafruit.com/viewtopic.php?t=198134

So far, the issue happens even with nothing connected to the SCORPIO asides from the USB-C plug, and it happens on an AMD desktop and Intel laptop (unlikely to be the computer at fault).

I'll follow up here if any solution is found.

[digitalcircuit]

Sure thing, and no worries! I understand being busy, and I'm simply relieved to know it's not just me 🙂

I've posted an update on the forum thread too, in case anyone else subscribed there.

In the meantime, while not ideal, CircuitPython's NeoPXL8 library can be used as a workaround. Serial will still rarely drop out (sometimes runs for 20+ hours no issues, sometimes it'll drop a couple times over a couple minutes), but it so far CircuitPython NeoPXL8 has always reconnected and resumed functioning, no interruption to code.

If there's any other testing you'd find useful, let me know and I'll see what I can do!

[digitalcircuit]

✅ 1.2.4 success 🎉

Thank you! NeoPXL8 version 1.2.4 appears to fix the issue (continued working for over 2 hours).

It did lock up once, but despite trying multiple times, I haven't reliably recreated that issue (previously, I could easily get it to drop Serial within 5 minutes), and it seems much more stable now.

❓ EDIT 2023-2-9 - one more hang

I had it hang USB Serial once more after 7 hours (25814 seconds) of trouble-free operation. In the off-hand chance my larger project's use of a digitalRead() interferes, I threw a gpio_set_drive_strength(…) on that pin before calling pinMode(…, INPUT_PULLUP).

I'll leave my desktop on overnight, and if it fails again, I'll revert back to the simple test code in this issue.

Even with this, it's still a significant improvement!

Further thoughts

I can also confirm your USB findings for NeoPXL8 version 1.2.3:

• Adafruit TinyUSB used to fully lock up
• Pico SDK (default) used to stall for about 1 second, then resumed animation but serial never recovered

In both cases on 1.2.3, the Serial output stalled roughly 5-15 seconds before the animation itself stalled or hung entirely.

I wonder if the GPIO drive strength adjustment should be made in CircuitPython's NeoPXL8, too?

I sometimes - not always! - saw CircuitPython's NeoPXL8 briefly (sub-second) stall both Serial and animation, then fully recover via serial dis/connecting (code did not reset, seconds since start kept going up). It wasn't consistent - sometimes I got over 20 hours without issue, other times it dis/connected a few times over a few minutes, always fully recovering.

Regardless, I'll close this issue for now and share the good news on the forum thread too. If the problem resurfaces, I can comment here or file a new issue.

[digitalcircuit]

Good news: The Serial breakage happens a lot less often. I've had a run of 7.2 hours before Serial stopped, then a separate run of 6.2 hours.

Not-so good news: Sometimes Serial breaks quickly (still not as quick as before though!)… I just had a run of 407 seconds (6 minutes) before Serial stopped.

It's possible there were multiple reasons Serial broke, so I'm retrying disconnecting the LEDs alongside other troubleshooting steps to attempt to narrow this down.

The GPIO drive current change appears to bring it in line with CircuitPython at least, which could go for hours or quickly dis/connect after a few minutes (CircuitPython always successfully reset the serial connection while keeping the code running).

Let me know if there's anything you'd find useful!

EDIT 2023-2-10:
With the LEDs disconnected, I had it hang after 4904 seconds (1.3 hours). I'll keep investigating.

EDIT 2023-2-11:
On my laptop (less noisy USB power) with the LEDs disconnected, I had it hang after 978 seconds, 416 seconds, and… 3 seconds. No real consistency to this at all.

I'll try repeating the 100µF filter capacitor test before reopening this issue.

EDIT 2023-2-12 part A:
With a 100µF filter capacitor across the USB (5v) and GND pins, Serial still hung at 762 seconds then at 3528 seconds, no discernible improvement.

EDIT 2023-2-12 part B:
With a 100µF filter capacitor across the 3.3V and GND pins, Serial still hung at 3335 seconds then at 117 seconds, no discernible improvement.

[digitalcircuit]

⚠ 1.2.4 … is actually only partial success

Unfortunately, as described in my edited comment, some interaction between Serial and the NeoPXL8 library continues to break the Serial connection.

I repeated the troubleshooting steps from the forum thread, namely disconnecting the LEDs, then using a different computer, adding a filter capacitor across USB/GND or 3.3V/GND, to no avail.

To be clear, the situation seems to be improved!

It's now reliable enough that if I had the board hard-reset itself 5 minutes after not receiving Serial data, it would hack around the problem most of the time. But it would be nicer to actually fix this.

This now appears to mimic the CircuitPython NeoPXL8 behavior where that would inconsistently reset the Serial connection too (including the mass storage device), however CircuitPython always reconnected the Serial line without interrupting the code.

Please let me know if there's any further information that would help!

[digitalcircuit]

I'm continuing to re-test this with the updated Raspberry Pi Pico SDK and other libraries. Results seem promising so far with over 3 days (268179 seconds and counting) of functioning USB serial connection, even handling the computer being shut down for hours at a time.

I'll close this issue within a few weeks if I don't notice it reoccurring.

Edit 2023-3-13:
5.7 days (500004 seconds) and counting. I'll wait for around 7 days of serial uptime, then remove power from the SCORPIO then try again in case this was just lucky.

Edit 2023-3-14:
7 days (609345 seconds) and counting without hang. I'll fully power off the SCORPIO and computer (disconnect the power supply), then restart the test to see if this succeeds for another 7 days.

[digitalcircuit]

✅ Issue resolved with Pico SDK 3.0.0, etc

Good news: Over the past two weeks, I have successfully gotten an entire week of uptime with the SCORPIO USB connection. The first time went to 7 days (609345 seconds), then I fully powered off the SCORPIO and computer (disconnecting power supply), then restarted the test, running it for another 7 days (667833 seconds).

At the end of both tests, USB serial continued to respond, the Arduino IDE automatically reset the SCORPIO into bootloader mode, etc.

I've been using the RP2040 Pico SDK version 3.0.0, and I'll continue updating (SDK 3.1.0 is out now, also Adafruit SPIFlash got updated from 4.0.0 to 4.1.1, Adafruit TinyUSB got updated from 2.0.0 to 2.0.1) and confirming this.

EDIT 2023-3-23: I had a hang within a few hours after the above board and library updates. I'll continue testing to see if this was a one-off, or if it's repeatable.

[digitalcircuit]

Okay, after a few days with Pico SDK 3.1.0 and the other updates mentioned, I've had it lock up again (I wasn't actively monitoring USB serial at the time, so I don't have the exact seconds). That's at least twice.

It's at least been a few hours, sometimes a week+ before the serial connection hangs. It's long enough between hangs that I'm not sure of the right techniques to troubleshoot this - perhaps I can set up a debugging probe of sorts.

I'll follow up if I can find any more details. For now, given how tricky this is to track down, I might leave this issue closed.

I apologize for the back-and-forth. This seems to be a proper heisenbug.