toverainc / willow Goto Github PK
View Code? Open in Web Editor NEWOpen source, local, and self-hosted Amazon Echo/Google Home competitive Voice Assistant alternative
Home Page: https://heywillow.io/
License: Apache License 2.0
willow's People
Contributors
Stargazers
Watchers
Forkers
artisr suryatmodulus suchoudh tinklern antelligent-app kappuchino kenhayward tiamat-tech ai-jie01 hamishcunningham rj williamzhou2018 1perfscepmapfu 9gravretozo biobunaa tufebramga 7quigranoviuka madcowmoomoo 0sumpleaxide 1dalliavrene 1graphaptioso cremacwhezo kustomzone 1nistmonesto 0atemxmorro capriaanshi anbifanhe 7intaudina riepecjeuloc 9moporsancchi presbes0rueno 9esodchade 8spirpenvgratpe 8conftutada flicutmcosyu 0georepfarcdo soon14 1fragammultpo 8gilmiageofu simpcauyusdo hhy5277 ulgeqfruspu tyldev ticucuito 0irboflubku pistmafexgu shenxiaozheng kloodhu consmoesperri 8quecepcongi 7tiadigrangi 8flagfowcastbo 0curumpuncho 3falsatiori kunwon1 huangweiboy2 qqq-tech ssurovich mineshaftgap blakadder raybbb dkzdev stealthcopter jake-c-s meeech hcole223 a6blpka hbcbh1999 zzh-527 robbles nikito jeffphillipssss myxor skorokithakis suglider wemersiveadmin espressif2022 rcythr jesusoctavioas tangyiyong randomact5 mranv prynix kelciwang daniel-7d mjhewitt1 kevdliu chorty jak12-3 edustackwillow's Issues
Home Assistant flasher
We should do something similar to esphome (or use esphome) where users can configure and then flash devices directly from Home Assistant.
Add "support" argument to utils to collect user environment info for debugging and user support
As users start seeing ESP BOXes ship we are going to run into issues with different environments, etc.
We should add a support option to utils.sh to collect a range of (non-identifying) info so they can provide it in GH issues.
New Feature? countdown timer
One of the things I currently us a lot on Alexa is a countdown timer so in order to try replace it, I need that functionality. I wanted to see if there was interest in having this be part of the project? If so I can look to see what it would take to implement it.
Better Readme
I went through readme and got project setup e2e. Some notes / questions jotted in this issue to follow up on. Some of them are probably obvious and included for completeness
- The only supported hardware currently is the ESP32-S3-Korvo-2. Is this a single device or a family? Link to a couple of sources where it can be purchased?
- Add an intro section. I have no clue what this project does by README.
- I would love to just pull some prebuilt docker image ideally that takes me right to config.
- KK gave me these instructions which are ~current and a lot better than README. Update README with them?
PORT=the USB port on your system
./utils.sh build-docker
./utils.sh docker
(now in container)
./utils.sh install
./utils.sh config
(Go to "Sallow Configuration") - pretty obvious options there
./utils.sh build
./utils.sh flash (edited)
- If we want wider adoption mac & ubuntu specific directions for getting USB device name and flashing?
- What is default wake word?
Support wake word
`./utils.sh config` results in Failed to resolve component 'esp-adf-libs'.
When trying to build, running the step ./utils.sh config produces:
...
[6/6] lvgl/lvgl (8.3.7)
CMake Error at /opt/esp/idf/tools/cmake/build.cmake:201 (message):
Failed to resolve component 'esp-adf-libs'.
Call Stack (most recent call first):
/opt/esp/idf/tools/cmake/build.cmake:236 (__build_resolve_and_add_req)
/opt/esp/idf/tools/cmake/build.cmake:237 (__build_expand_requirements)
/opt/esp/idf/tools/cmake/build.cmake:242 (__build_expand_requirements)
/opt/esp/idf/tools/cmake/build.cmake:242 (__build_expand_requirements)
/opt/esp/idf/tools/cmake/build.cmake:518 (__build_expand_requirements)
/opt/esp/idf/tools/cmake/project.cmake:393 (idf_build_process)
CMakeLists.txt:10 (project)
-- Configuring incomplete, errors occurred!
See also "/willow/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1
Cleanly handle server failures
We should display an error message and more gracefully timeout if Home Assistant or the inference server are unavailable.
Implement stream/VAD timeout
I am still seeing cases where VAD/wake end never happens.
In testing I had a device stream continuously for 12 hours before I finally pulled the power...
It's very difficult to reproduce but:
-
It makes testing/development frustrating for me. I believe my issues are related to wifi environment but I have experienced this in multiple completely different environments - Wisconsin (two Unifi APs) and California (Qualcomm and Meditek AP). FWIR stintel has yet to observe this issue.
-
We don't (currently) have a use case where endless streaming makes sense anyway.
We should implement a timeout based on either seconds (10, maybe?), number of audio chunks, something to timeout streaming no matter what.
It seems that if I wait for a few seconds after the display indicates ready it's much less likely to occur.
speech_commands/generate_commands.py errors with Multinet + fix
(i flashed and tested sucessfully without multinet btw)
Enabled multinet in config, and running build errors like so:
root@770980441ff8:/willow# ./utils.sh build
MULTINET: Generate speech commands: Attempting to fetch your ['light.', 'switch.'] entities from Home Assistant...
Traceback (most recent call last):
File "speech_commands/generate_commands.py", line 84, in <module>
numbers = re.search(r'(\d{1,})', friendly_name)
File "/usr/lib/python3.8/re.py", line 201, in search
return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
root@770980441ff8:/willow#
It's because I get a roguefriendly_name = None back from home assistant. maybe a HA config issue or something.
to make it more robust i just skip entities with a None friendy_name:
(base) rj@rjms:~/src/willow$ git diff
diff --git a/speech_commands/generate_commands.py b/speech_commands/generate_commands.py
index 4fd9368..31fcb43 100644
--- a/speech_commands/generate_commands.py
+++ b/speech_commands/generate_commands.py
@@ -81,6 +81,9 @@ for type in entity_types:
if entity_id.startswith(type):
attr = entity.get('attributes')
friendly_name = attr.get('friendly_name')
+ if friendly_name is None:
+ # in case of blank or misconfigured HA entities
+ continue
numbers = re.search(r'(\d{1,})', friendly_name)
if numbers:
for number in numbers.groups():
(base) rj@rjms:~/src/willow$ Great job btw, looking forward to playing around with this!
Multinet slower than Alexa
While Wakenet is only trivially slower than Alexa, Multinet is perceptible slower.
I timed a very unscientific video of all three turning on my living room lights and here is what I have found:
I have confirmed that the command TURN ON LIVING ROOM LIGHTS is in speech_commands/commands_en.txt.
What else do I need to do in order to improve my multinet speeds?
EDIT: removed movies from automatically showing due to bringing browser to a crawl, hoping links above will still work.
Support flashing from Windows
As learned from #60 (again)
Currently Windows users are somewhat "left out in the cold" and passing USB through VMs, etc is a mess.
We should create a Powershell or similar script so Windows users can flash a willow-dist.bin from the build process using esptool natively within Windows.
Replace beeps with chimes
We currently beep once for success and twice for failure. Not the worst beep in the world but not the most pleasant either. There are a variety of options online for low-cost royalty chimes.
Ideally we could do a poll or similar on this issue to determine what the community likes the most for success and error and incorporate them in flash for playback.
Set Willow HTTP User-Agent header
...is there more to say?
Cant Flash - cant find ttyACM0?
Everything seems to go well, up to erasing the flash - It cant find the port, but dmesg shows the Espressif and that it is on /dev/ttyACM0. Looking at /dev - I can see ttyACM0 is there. If I disconnect the USB cable, ttyACM0 does go away...
Using venv for esptool
esptool.py v4.5.1
Serial port /dev/ttyACM0
A fatal error occurred: Could not open /dev/ttyACM0, the port doesn't exist
dmesg when plugged in:
[ 3364.126378] usb 2-2.1: new full-speed USB device number 8 using uhci_hcd
[ 3364.434428] usb 2-2.1: New USB device found, idVendor=303a, idProduct=1001, bcdDevice= 1.01
[ 3364.434435] usb 2-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3364.434437] usb 2-2.1: Product: USB JTAG/serial debug unit
[ 3364.434439] usb 2-2.1: Manufacturer: Espressif
[ 3364.434440] usb 2-2.1: SerialNumber: 7C:DF:A1:E8:22:F8
[ 3364.438405] cdc_acm 2-2.1:1.0: ttyACM0: USB ACM device
[ 3408.337353] usb 2-2.2: USB disconnect, device number 7
[ 3412.010379] usb 2-2.1: USB disconnect, device number 8
[ 3412.923174] usb 2-2.1: new full-speed USB device number 9 using uhci_hcd
Only potential issue - this is a VM, but Im passing the USB through, and I never had issues in the past doing this on any flashing (ESP / etc...)
Any ideas?
exported PORT is not supplied to docker container?
Ran export PORT=/dev/ttyACM0 as documented but that is not passed into docker container as an ENV var or passed to --device ?
Decrease multinet timeout
The default multinet timeout of 5.8s feels like an eternity compared to the rest of Willow.
Suggest decreasing it to 2 seconds, which is twice the target response time and more than enough time for users to issue commands within the 63 character command limit of multinet.
User build/run time configuration
We should have reasonable defaults for things like wake word engine, VAD aggressiveness, etc but at a minimum we should move them into the Kconfig/config system (under Willow Configuration).
When we have a Home Assistant component working we can specify the configuration there and have ESP Boxes push/pull the preferred configuration from our Home Assistant API endpoint as JSON to configure the ESP Boxes dynamically on startup or on demand.
Investigate display timeout variability
The display timeout isn't completely reliable.
The timers in ESP-IDF are being re-worked - save this for that.
homeassistant conversation API not understanding text when ending with a "."
I've noticed that the public inference server produces sentences with a "." character at the end. It seems that HASS's conversation API isnt able to parse these and just responds with "no_intent_match".
If you manually Curl the API endpoint with two sentences (for example "turn on the big lights") it wont understand if the sentence ends with a period:
`
curl http://IP:8123/api/conversation/process -H "..." --data-ascii '{"text":"turn off the big lights.","language":"en"}'
-> {"response":{"speech":{"plain":{"speech":"Sorry, I didn't understand that","extra_data":null}},"card":{},"language":"en","response_type":"error","data":{"code":"no_intent_match"}},"conversation_id":null}%
but:
curl http://IP:8123/api/conversation/process -H "..." --data-ascii '{"text":"turn off the big lights","language":"en"}'
is fine
`
HA version: 2023.1.4
Show configured wake word on display
Related to #16 - because we support a variety of wake words it's not immediately obvious to the user what the wake word is. We should show the configured wake word on the display.
WebSocket support
Establishing a HTTPS connection seems to take ~1s. If both the AIA and HA are using HTTPS, this adds quite some latency. Implement WebSocket support to improve this.
Stack Overflow when using multinet
When using multinet, booting willow results in a stack overflow. After disabling multinet, willow works correctly. I tried enabling the willow debug options for more info with no luck.
MC Quantized wakenet9: wakeNet9_v1h24_hiesp_3_0.63_0.635, tigger:v3, mode:2, p:0, (May 5 2023 20:32:52)
I (5752) AFE_SR: wake num: 3, mode: 1, (May 5 2023 20:32:52)
I (6211) wifi:<ba-add>idx:1 (ifx:0, 32:ee:52:44:b0:6a), tid:1, ssn:0, winSize:64
I (23:17:17.205) WILLOW: SNTP client synchronized time to 1684689437
Build fst from commands.
***ERROR*** A stack overflow in task main has been detected.
Backtrace: 0x4037a192:0x3fcace10 0x40383ff5:0x3fcace30 0x40387a22:0x3fcace50 0x40385d70:0x3fcacec0 0x403840ac:0x3fca
cee0 0x403840a2:0x00000000 |<-CORRUPTED
Use Pool NTP servers if no server is provided via DHCP
We currently have an ifdef if get NTP servers from DHCP is defined.
However, it's defined by default and should be but we should actually be checking to see if the DHCP server provides an NTP server. If not, we should use pool.
Allow users to configure multinet for on-device speech command recognition
LCD SPI Transmit Failure
While torture testing I observed an SPI transmit failure that essentially breaks the device:
I (07:18:50.153) SALLOW: AUDIO_REC_WAKEUP_START
I (07:18:50.448) SALLOW: AUDIO_REC_VAD_START
W (07:18:50.449) AUDIO_PIPELINE: Pipeline already started, state:3
E (07:18:50.449) AUDIO_ELEMENT: [http_stream_writer] Element already stopped
E (07:18:50.458) AUDIO_ELEMENT: [raw_stream_writer_to_api] Element already stopped
I (07:18:50.458) AUDIO_ELEMENT: [encoder] AEL_MSG_CMD_RESUME,state:5
I (575614) AMRWB_ENCODER: amrwb open
W (07:18:50.478) AUDIO_EVT: There is no space in external queue
I (07:18:51.021) AUDIO_THREAD: The http_stream_writer task allocate stack on external memory
I (07:18:51.022) AUDIO_ELEMENT: [http_stream_writer-0x3d814040] Element task created
I (07:18:51.028) AUDIO_ELEMENT: [raw_stream_writer_to_api-0x3d814170] Element task created
I (07:18:51.037) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:7077243 Bytes, Inter:67187 Bytes, Dram:67187 Bytes
I (07:18:51.050) AUDIO_ELEMENT: [http_stream_writer] AEL_MSG_CMD_RESUME,state:1
I (07:18:51.059) SALLOW: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, length=0
I (07:18:51.059) AUDIO_PIPELINE: Pipeline started
audio_pipeline_run(hdl_ap_to_api) - uri: 'https://infer.tovera.io/api/sallow?model=medium&detect_language=False&beam_size=1&save_audio=True'
I (07:18:51.633) SALLOW: AUDIO_REC_VAD_END
I (576784) AMRWB_ENCODER: amrwb close
W (07:18:51.642) AUDIO_EVT: There is no space in external queue
W (07:18:51.643) AUDIO_EVT: There is no space in external queue
W (07:18:51.643) AUDIO_RECORDER: Not in speeching, return 0
at_read() ret leq 0
I (07:18:51.656) AUDIO_ELEMENT: OUT-[raw_stream_writer_to_api] AEL_IO_DONE,0
MALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000800:
At 0x3fcb1894 len 16383 free 7 allocated 15912 min_free 7
largest_free_block 0 alloc_blocks 12 free_blocks 0 total_blocks 12
At 0x3fcabde0 len 252208 free 27368 allocated 221748 min_free 4420
largest_free_block 15360 alloc_blocks 538 free_blocks 11 total_blocks 549
At 0x3fce9710 len 22308 free 4 allocated 20636 min_free 4
largest_free_block 0 alloc_blocks 233 free_blocks 0 total_blocks 233
At 0x600fe000 len 8192 free 7812 allocated 0 min_free 7812
I (07:18:51.733) SALLOW: AUDIO_REC_WAKEUP_END
largest_free_block 7680 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 24663 allocated 268620 min_free 12243 largest_free_block 14336
MALLOC_CAP_SPIRAM:
Heap summary for capabilities 0x00000400:
At 0x3d800000 len 8388607 free 7017259 allocated 1368096 min_free 7003247
largest_free_block 6946816 alloc_blocks 238 free_blocks 4 total_blocks 242
Totals:
free 7017259 allocated 1368096 min_free 7003247 largest_free_block 6946816
Total bytes written: 4096
Total bytes written: 4218
I (07:18:52.168) AUDIO_ELEMENT: IN-[http_stream_writer] AEL_IO_DONE,-2
I (07:18:52.169) SALLOW: [ + ] HTTP client HTTP_STREAM_POST_REQUEST, write end chunked marker
I (07:18:52.442) HTTP_CLIENT: Body received in fetch header state, 0x3fcda529, 48
I (07:18:52.443) SALLOW: [ + ] HTTP client HTTP_STREAM_FINISH_REQUEST
I (07:18:52.447) SALLOW: Got HTTP Response = {"text":"Turn off dining room.","language":"en"}
I (07:18:52.457) SALLOW: sending '{"text":"Turn off dining room.","language":"en"}' to Home Assistant API on 'http://hass:8123/api/conversation/process'
I (07:18:52.538) HTTP_CLIENT: Body received in fetch header state, 0x3fcd8a48, 270
I (07:18:52.539) SALLOW: HTTP POST status='200' content_length='270'
I (07:18:52.543) SALLOW: home assistant response_type: action_done
I (07:18:52.550) AUDIO_THREAD: The play_tone_ok task allocate stack on external memory
I (07:18:52.558) SALLOW: HTTP POST response body:
{
"response": {
"speech": {
"plain": {
"speech": "Turned off light",
"extra_data": null
}
},
"card": {
},
"language": "en",
"response_type": "action_done",
"data": {
"targets": [],
"success": [{
"name": "Dining Room",
"type": "entity",
"id": "light.dining_room"
}],
"failed": []
}
},
"conversation_id": null
}
E (07:18:52.642) lcd_panel.io.spi: panel_io_spi_tx_color(385): spi transmit (queue) color failed
I (07:18:54.599) AUDIO_ELEMENT: [http_stream_writer] AEL_MSG_CMD_PAUSE
I (07:18:55.595) SALLOW: AUDIO_REC_WAKEUP_START
MALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000800:
At 0x3fcb1894 len 16383 free 7 allocated 15912 min_free 7
largest_free_block 0 alloc_blocks 12 free_blocks 0 total_blocks 12
At 0x3fcabde0 len 252208 free 59036 allocated 190316 min_free 4420
largest_free_block 17408 alloc_blocks 479 free_blocks 34 total_blocks 513
At 0x3fce9710 len 22308 free 36 allocated 20612 min_free 4
largest_free_block 0 alloc_blocks 231 free_blocks 2 total_blocks 233
At 0x600fe000 len 8192 free 7812 allocated 0 min_free 7812
largest_free_block 7680 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 66891 allocated 226840 min_free 12243 largest_free_block 17408
MALLOC_CAP_SPIRAM:
Heap summary for capabilities 0x00000400:
At 0x3d800000 len 8388607 free 7023407 allocated 1361952 min_free 7003247
largest_free_block 6946816 alloc_blocks 237 free_blocks 4 total_blocks 241
Totals:
free 7023407 allocated 1361952 min_free 7003247 largest_free_block 6946816
MALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000800:
At 0x3fcb1894 len 16383 free 7 allocated 15912 min_free 7
largest_free_block 0 alloc_blocks 12 free_blocks 0 total_blocks 12
At 0x3fcabde0 len 252208 free 59036 allocated 190316 min_free 4420
largest_free_block 17408 alloc_blocks 479 free_blocks 34 total_blocks 513
At 0x3fce9710 len 22308 free 36 allocated 20612 min_free 4
largest_free_block 0 alloc_blocks 231 free_blocks 2 total_blocks 233
At 0x600fe000 len 8192 free 7812 allocated 0 min_free 7812
largest_free_block 7680 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 66891 allocated 226840 min_free 12243 largest_free_block 17408
MALLOC_CAP_SPIRAM:
Heap summary for capabilities 0x00000400:
At 0x3d800000 len 8388607 free 7023407 allocated 1361952 min_free 7003247
largest_free_block 6946816 alloc_blocks 237 free_blocks 4 total_blocks 241
Totals:
free 7023407 allocated 1361952 min_free 7003247 largest_free_block 6946816
W (587564) AFE_SR: ERROR! rb_out slow!!!
W (587794) AFE_SR: ERROR! rb_out slow!!!
W (587834) AFE_SR: ERROR! rb_out slow!!!
W (587864) AFE_SR: ERROR! rb_out slow!!!
W (588834) AFE_SR: ERROR! rb_out slow!!!
W (588874) AFE_SR: ERROR! rb_out slow!!!
W (588894) AFE_SR: ERROR! rb_out slow!!!
W (588914) AFE_SR: ERROR! rb_out slow!!!
The LCD garbles and the while wake still works no audio is transmitted and the rb_out slow messages loop.
Seems to be related to this issue.
Apparently the fix is an SPI flag to keep CS (cable select) active.
Enforce running of erase-flash
As learned from #60 we should create a flag file or similar to prevent flash if the user hasn't run erase-flash at least once.
I get to idf.py build and receive these errors on the stintel/wip
bash-3.2$ idf.py build
Executing action: all (aliases: build)
Running cmake in directory /Users/brian/kk/sallow/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DIDF_TARGET=esp32s3 -DCCACHE_ENABLE=0 /Users/brian/kk/sallow"...
CMake Error at CMakeLists.txt:5 (include):
include could not find requested file:
/CMakeLists.txt
-- Found Git: /opt/homebrew/bin/git (found version "2.39.1")
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /Users/brian/.espressif/tools/xtensa-esp32s3-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Users/brian/.espressif/tools/xtensa-esp32s3-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Users/brian/.espressif/tools/xtensa-esp32s3-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building ESP-IDF components for target esp32s3
Processing 1 dependencies:
[1/1] idf (4.4.4)
-- DEBUG: Use esp-modbus component folder: /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/freemodbus.
-- Project sdkconfig file /Users/brian/kk/sallow/sdkconfig
-- Found Python3: /opt/homebrew/Frameworks/Python.framework/Versions/3.11/bin/python3.11 (found version "3.11.2") found components: Interpreter
-- Performing Test C_COMPILER_SUPPORTS_WFORMAT_SIGNEDNESS
-- Performing Test C_COMPILER_SUPPORTS_WFORMAT_SIGNEDNESS - Success
-- App "sallow" version: 1
-- Adding linker script /Users/brian/kk/sallow/build/esp-idf/esp_system/ld/memory.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_system/ld/esp32s3/sections.ld.in
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.api.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.libgcc.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.newlib.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.version.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom/esp32s3/ld/esp32s3.rom.newlib-time.ld
-- Adding linker script /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/soc/esp32s3/ld/esp32s3.peripherals.ld
-- DEBUG: Use esp-modbus component folder: /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/freemodbus.
-- Components: app_trace app_update asio audio_flash_tone bootloader bootloader_support bt cbor cmock coap console cxx driver efuse esp-tls esp32s3 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_hid esp_http_client esp_http_server esp_https_ota esp_https_server esp_hw_support esp_ipc esp_lcd esp_local_ctrl esp_netif esp_phy esp_pm esp_ringbuf esp_rom esp_serial_slave_link esp_system esp_timer esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos hal heap idf_test ieee802154 jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl openthread partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter tinyusb ulp unity usb vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/app_trace /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/app_update /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/asio /Users/brian/kk/sallow/components/audio_flash_tone /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/bootloader /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/bootloader_support /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/bt /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/cbor /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/cmock /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/coap /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/console /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/cxx /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/driver /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/efuse /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp-tls /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp32s3 /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_adc_cal /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_common /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_eth /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_event /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_gdbstub /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_hid /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_http_client /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_http_server /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_https_ota /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_https_server /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_hw_support /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_ipc /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_lcd /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_local_ctrl /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_netif /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_phy /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_pm /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_ringbuf /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_rom /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_serial_slave_link /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_system /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_timer /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_websocket_client /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esp_wifi /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/espcoredump /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/esptool_py /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/expat /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/fatfs /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/freemodbus /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/freertos /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/hal /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/heap /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/idf_test /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/ieee802154 /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/jsmn /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/json /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/libsodium /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/log /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/lwip /Users/brian/kk/sallow/main /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/mbedtls /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/mdns /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/mqtt /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/newlib /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/nghttp /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/nvs_flash /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/openssl /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/openthread /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/partition_table /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/perfmon /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/protobuf-c /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/protocomm /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/pthread /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/sdmmc /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/soc /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/spi_flash /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/spiffs /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/tcp_transport /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/tcpip_adapter /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/tinyusb /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/ulp /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/unity /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/usb /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/vfs /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/wear_levelling /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/wifi_provisioning /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/wpa_supplicant /Users/brian/kk/sallow/deps/esp-adf/esp-idf/components/xtensa
-- Configuring incomplete, errors occurred!
cmake failed with exit code 1
OTA Firmware updates
ESP-IDF and ESP32 devices have robust support for OTA (other the air) firmware updates.
At a point you have a bunch of these things going around re-flashing them is a huge pain. Will also benefit from #11.
S3 Box: Use touchscreen
Verify AFE, AEC, AGC, etc
ESP-BOX lite freezes
After a restart the device, it picks up the first "Hi E.S.P." just fine, before tio reports "Wake LCD timeout, tunring off LCD" - which never happes - LCD is on all the time. But device is not responding. I assume it's sleeping.
It does wakes up by pressing the middle button and then speak the wake-word.
Is this intended to behave like this - or is it only the ESP=BOX Lite ?
Improve error logging on display
Per discussion
Fix hardware mute button
It works to mute the microphones when pressed.
However, unmuting the microphone doesn't seem to work.
esp tool error while flashing, invalid choice: 'esp32s3'
I already build the project in docker sucessfully. I use macOS. But when I want to flash it from my host machine I get the following error:
Using venv for esptool
esptool.py --chip esp32s3 -p /dev/tty.usbmodem14201 -b 2000000 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 16MB 0x0 bootloader/bootloader.bin 0x10000 willow.bin 0x8000 partition_table/partition-table.bin 0x210000 srmodels/srmodels.bin 0x710000 audio.bin
usage: esptool [-h] [--chip {auto,esp8266,esp32}] [--port PORT] [--baud BAUD]
[--before {default_reset,no_reset,no_reset_no_sync}]
[--after {hard_reset,soft_reset,no_reset}] [--no-stub]
[--trace] [--override-vddsdio [{1.8V,1.9V,OFF}]]
{load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,version}
...
esptool: error: argument --chip/-c: invalid choice: 'esp32s3' (choose from 'auto', 'esp8266', 'esp32')
I used ./utils.sh flash or ./utils.sh erase-flash to perform the operation.
Better config management
Instead of depending on using one gigantic sdkconfig.willow we should probably have at least three:
- Defaults for everything (from espressif)
- Overrides for Willow configs
- Use overrides for things like the Willow Configuration options
Undefined behavior of type type_mismatch_v1 in cJSON_ArrayForEach
When building with UndefinedBehaviorSanitizer enabled we hit a type_mismatch_v1 in cJSON_ArrayForEach (https://github.com/toverainc/willow/blob/main/main/hass.c#L210).
diff --git a/main/CMakeLists.txt b/main/CMakeLists.txt
index f84414b..fd2fefe 100644
--- a/main/CMakeLists.txt
+++ b/main/CMakeLists.txt
@@ -16,3 +16,5 @@ set(COMPONENT_PRIV_REQUIRES
register_component()
spiffs_create_partition_image(audio ../spiffs/audio FLASH_IN_PROJECT)
+
+string(APPEND CMAKE_C_FLAGS -fsanitize=undefined)I (16:11:55.905) HTTP_CLIENT: Body received in fetch header state, 0x3fcc81e1, 367
0x3fcc81e1 - _btdm_bss_start
at ??:??
Undefined behavior of type type_mismatch_v1
Backtrace: 0x4037a106:0x3fcb33f0 0x40384569:0x3fcb3410 0x403845af:0x3fcb3430 0x403845bf:0x3fcb3490 0x4200a195:0x3fcb34b0 0x4200a29a:0x3fcb34e0 0x42009677:0x3fcb3500 0x42128c00:0x3fcb35a0 0x40388046:0x3fcb35c0
0x4037a106 - panic_abort
at /opt/esp/idf/components/esp_system/panic.c:408
0x3fcb33f0 - _btdm_bss_start
at ??:??
0x40384569 - esp_system_abort
at /opt/esp/idf/components/esp_system/esp_system.c:137
0x3fcb3410 - _btdm_bss_start
at ??:??
0x403845af - __ubsan_default_handler
at /opt/esp/idf/components/esp_system/ubsan.c:166
0x3fcb3430 - _btdm_bss_start
at ??:??
0x403845bf - __ubsan_handle_type_mismatch_v1
at /opt/esp/idf/components/esp_system/ubsan.c:180
0x3fcb3490 - _btdm_bss_start
at ??:??
0x4200a195 - hass_check_assist_pipeline
at /willow/main/hass.c:210
0x3fcb34b0 - _btdm_bss_start
at ??:??
0x4200a29a - init_hass
at /willow/main/hass.c:383
0x3fcb34e0 - _btdm_bss_start
at ??:??
0x42009677 - app_main
at /willow/main/main.c:761
0x3fcb3500 - _btdm_bss_start
at ??:??
0x42128c00 - main_task
at /opt/esp/idf/components/freertos/port/port_common.c:141
0x3fcb35a0 - _btdm_bss_start
at ??:??
0x40388046 - vPortTaskWrapper
at /opt/esp/idf/components/freertos/port/xtensa/port.c:142
0x3fcb35c0 - _btdm_bss_start
at ??:??
S3 Box: Control LCD
watchdog triggered after wake word
Hi,
first of all, thank you for this very interesting project. I just got an S3-Box, and flashed Willow configured for local detection and Alexa keyword.
It detects the wakeword fine, but after saying my command, it stays on the "Recording command..." screen, until the watchdog fires and reboots the device.
The log up to this point is:
I (22:22:43.343) WILLOW: AUDIO_REC_WAKEUP_START
I (22:22:43.512) WILLOW: AUDIO_REC_VAD_START
I (22:22:49.746) WILLOW: AUDIO_REC_VAD_END
I (22:22:49.846) WILLOW: AUDIO_REC_WAKEUP_END
I (22:22:53.629) RECORDER_SR: MN dect quit
E (489720) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (489720) task_wdt: - IDLE (CPU 1)
E (489720) task_wdt: Tasks currently running:
E (489720) task_wdt: CPU 0: IDLE
E (489720) task_wdt: CPU 1: afe_mase
E (489720) task_wdt: Aborting.
abort() was called at PC 0x420871e0 on core 0
Backtrace: 0x4037a17e:0x3fc9fe50 0x40383c7d:0x3fc9fe70 0x4038bd62:0x3fc9fe90 0x420871e0:0x3fc9ff00 0x4037b1e9:0x3fc9ff30 0x4211b86b:0x3fcb16d0 0x42003ca9:0x3fcb16f0 0x4038506f:0x3fcb1710
The device is connected to a Wifi network without access to the internet (only access to the Homeassistant instance).
Any ideas what might be causing this?
Thanks in advance,
Henning
Better support for audio
We currently POST/stream audio and get an HTTP 200 of just the output text. We then come back to a static URL to grab a single file from the endpoint.
This is totally broken, slow, etc.
We should be able to get the JSON output of something like the /api/asr in the HTTP response WITH the audio all in the same single request.
"One shot" install script
As learned from #60 and elsewhere users can sometimes miss crucial steps provided in the dense documentation.
We should create a "one shot" install script that executes the steps in the proper order where the only remaining "manual" step is the configuration menu.
Willows in earshot of one another should not all wake and stream on wake word
Willow far-field audio and wake word detection is good enough that in many cases, when installed in typical environments, multiple Willows will wake with a given wake word. Nothing terrible happens, it's just annoying to get multiple confirmation beeps and beat up your Willow Command Endpoint (such as Home Assistant).
Generally there are two main ways to go about addressing this:
- Multicast. When devices wake they send a packet to a multicast group the other Willows have joined. It kind of turns into a race but without getting into that now a Willow would win and the other Willows would silently back off.
Pros:
- All local. Works with any speech recognition mode (server, local).
- We should be able to read the amplitude of the audio input signal, include that in the message, and ensure that the Willow "closest" to the speech wins the election.
Cons:
- I've had really spotty experiences with multicast on wifi and I'm a little scared of it in the real world with so many diverse environments. It's almost guaranteed to be problematic for some users.
- Quite a bit of work.
- A lot of extra CPU on device.
- Let the Willow Inference Server figure it out. All devices wake and start early streaming. I have a branch (referenced below) that includes the ability to generate and use an anonymous random identifier to group Willows within the same group/installation/proximity/etc. This identifier is provided to the Willow Inference Server. In this case the Willow Inference Server would essentially handle the election, drop all but the preferred source within the group, and the dropped Willows would silently deactivate just like the multicast case above.
Pros:
- Doesn't depend on anything special regarding Wifi. Universally compatible.
- We can still read and include the amplitude of the audio input signal to pick the closest Willow.
- Using the Willow Inference Server's processing abilities opens up all kinds of other interesting possibilities like potentially being able to do speaker identification to allow multiple simultaneous activations within a group if the speakers (people) are different.
Cons:
- Could be considered intrusive by those using the Tovera community hosted WIS. I insist again we don't log anything and we'll be documenting this formally soon. Of course this doesn't apply when you are hosting your own WIS (after we release next week you absolutely should)!
Manage wake word and audio pipeline hardware access
The wake word engine listens on the mic constantly (obviously). When wake word engine activates or push to record is triggered the ADF audio pipelines don't have access to hardware and thus nothing is recorded or played back.
We need to manage audio pipelines and wwe (wake word engine) so that we can support both functions seamlessly.
Code formatting
Code is currently very messy and has no formatting.
Use clang-format
Home Assistant auto discovery/configuration
Some helpful 3D prints
I have made two very crude 3D prints in order to achieve 2 things:
- Block out the power LED glow without having to desolder it. For now I have it block the mute LED but not the button, if there is interest I can make another version that doesn't block the mute LED.
- Stop ESP32-S3-Box from falling with an add on weighted base. I used the approach many have which is to have a place to insert pennies for weight, this right now is geared toward US pennies. If there is interest I can make other diameters if people provide them.
I made designed both to not need supports.
Since Github does not allow uploads of STL files I zipped them up
Here you can see what it looks like. There are a ton of ways it can be improved on, but it already makes it so much more pleasant:
Use VAD
Use VAD (voice activity detection) to know when the speaker is done talking so we can end the audio stream
memory leak / task_wdt triggered
With every cycle the min_free value of internal memory goes down. Eventually it goes so low that task_wdt is triggered.
ALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000800:
At 0x3fca7b30 len 269280 free 79536 allocated 186608 min_free 32
largest_free_block 18944 alloc_blocks 341 free_blocks 28 total_blocks 369
At 0x3fce9710 len 22308 free 36 allocated 20624 min_free 4
largest_free_block 0 alloc_blocks 228 free_blocks 2 total_blocks 230
At 0x600fe000 len 8192 free 7812 allocated 0 min_free 7812
largest_free_block 7680 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 87384 allocated 207232 min_free 7848 largest_free_block 18944
MALLOC_CAP_SPIRAM:
Heap summary for capabilities 0x00000400:
At 0x3d800000 len 8388607 free 7058735 allocated 1326420 min_free 7050215
largest_free_block 6946816 alloc_blocks 288 free_blocks 5 total_blocks 293
Totals:
free 7058735 allocated 1326420 min_free 7050215 largest_free_block 6946816
I (18:35:49.285) SALLOW: AUDIO_REC_WAKEUP_START
I (18:35:49.954) SALLOW: AUDIO_REC_VAD_START
E (18:35:49.955) AUDIO_ELEMENT: [http_stream_writer] Element already stopped
E (18:35:49.957) AUDIO_ELEMENT: [raw_stream_writer_to_api] Element already stopped
I (18:35:49.965) AUDIO_THREAD: The http_stream_writer task allocate stack on external memory
I (18:35:49.975) AUDIO_ELEMENT: [http_stream_writer-0x3d8140b8] Element task created
I (18:35:49.983) AUDIO_ELEMENT: [raw_stream_writer_to_api-0x3d8141e8] Element task created
I (18:35:49.991) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:7139623 Bytes, Inter:87036 Bytes, Dram:87036 Bytes
I (18:35:50.004) AUDIO_ELEMENT: [http_stream_writer] AEL_MSG_CMD_RESUME,state:1
I (18:35:50.012) AUDIO_PIPELINE: Pipeline started
audio_pipeline_run(hdl_ap_to_api) - uri: 'https://infer.tovera.io/api/sallow?model=medium&detect_language=False&beam_size=1&save_audio=True'
I (18:35:50.031) SALLOW: [ + ] HTTP client HTTP_STREAM_PRE_REQUEST, length=0
I (18:35:51.307) SALLOW: AUDIO_REC_VAD_END
W (18:35:51.334) AUDIO_RECORDER: Not in speeching, return 0
at_read() ret leq 0
I (18:35:51.335) AUDIO_ELEMENT: OUT-[raw_stream_writer_to_api] AEL_IO_DONE,0
I (18:35:51.407) SALLOW: AUDIO_REC_WAKEUP_END
E (532543) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (532543) task_wdt: - IDLE (CPU 0)
E (532543) task_wdt: Tasks currently running:
E (532543) task_wdt: CPU 0: http_stream_wri
E (532543) task_wdt: CPU 1: fetch_task
E (532543) task_wdt: Print CPU 0 (current core) backtrace
Backtrace: 0x420896AE:0x3FC9BF20 0x403772B1:0x3FC9BF50 0x420E606C:0x3D943370 0x420C56A7:0x3D943390 0x420C34D0:0x3D9433D0 0x420C3559:0x3D943400 0x420BCD49:0x3D943420 0x420BCF3A:0x3D943440 0x420AFEBD:0x3D943460 0x420AFF79:0x3D9434F0 0x420B06DD:0x3D943520 0x420B307E:0x3D943690 0x420B0F95:0x3D943740 0x420B0FB8:0x3D943760 0x420B0FDE:0x3D943780 0x420F0CC1:0x3D9437A0 0x420F05E1:0x3D9437C0 0x420F08FA:0x3D943840 0x420EF9E2:0x3D943870 0x4214019D:0x3D943890 0x420AD56A:0x3D9438B0 0x420AE2ED:0x3D9438D0 0x4200A085:0x3D9438F0 0x42011831:0x3D943A20 0x42011D77:0x3D943A50 0x42011EB2:0x3D943A70 0x42012BC1:0x3D943AA0 0x42011FAE:0x3D943AE0
E (532543) task_wdt: Print CPU 1 backtrace
Backtrace: 0x4037B0E9:0x3FC9C530 0x403772B1:0x3FC9C550 0x400559DD:0x3FCACA30 |<-CORRUPTED
E (537543) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (537543) task_wdt: - IDLE (CPU 0)
E (537543) task_wdt: Tasks currently running:
E (537543) task_wdt: CPU 0: http_stream_wri
E (537543) task_wdt: CPU 1: afe_mase
E (537543) task_wdt: Print CPU 0 (current core) backtrace
Backtrace: 0x420896AE:0x3FC9BF20 0x403772B1:0x3FC9BF50 0x420E6067:0x3D943370 0x420C56A7:0x3D943390 0x420C34D0:0x3D9433D0 0x420C3559:0x3D943400 0x420BCD49:0x3D943420 0x420BCF3A:0x3D943440 0x420AFEBD:0x3D943460 0x420AFF79:0x3D9434F0 0x420B06DD:0x3D943520 0x420B307E:0x3D943690 0x420B0F95:0x3D943740 0x420B0FB8:0x3D943760 0x420B0FDE:0x3D943780 0x420F0CC1:0x3D9437A0 0x420F05E1:0x3D9437C0 0x420F08FA:0x3D943840 0x420EF9E2:0x3D943870 0x4214019D:0x3D943890 0x420AD56A:0x3D9438B0 0x420AE2ED:0x3D9438D0 0x4200A085:0x3D9438F0 0x42011831:0x3D943A20 0x42011D77:0x3D943A50 0x42011EB2:0x3D943A70 0x42012BC1:0x3D943AA0 0x42011FAE:0x3D943AE0
E (537543) task_wdt: Print CPU 1 backtrace
Backtrace: 0x4037B0E9:0x3FC9C530 0x403772B1:0x3FC9C550 0x400559DD:0x3FCACA30 |<-CORRUPTED
E (542543) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (542543) task_wdt: - IDLE (CPU 0)
E (542543) task_wdt: Tasks currently running:
E (542543) task_wdt: CPU 0: http_stream_wri
E (542543) task_wdt: CPU 1: fetch_task
E (542543) task_wdt: Print CPU 0 (current core) backtrace
Backtrace: 0x420896AE:0x3FC9BF20 0x403772B1:0x3FC9BF50 0x420E606C:0x3D943370 0x420C56A7:0x3D943390 0x420C34D0:0x3D9433D0 0x420C3559:0x3D943400 0x420BCD49:0x3D943420 0x420BCF3A:0x3D943440 0x420AFEBD:0x3D943460 0x420AFF79:0x3D9434F0 0x420B06DD:0x3D943520 0x420B307E:0x3D943690 0x420B0F95:0x3D943740 0x420B0FB8:0x3D943760 0x420B0FDE:0x3D943780 0x420F0CC1:0x3D9437A0 0x420F05E1:0x3D9437C0 0x420F08FA:0x3D943840 0x420EF9E2:0x3D943870 0x4214019D:0x3D943890 0x420AD56A:0x3D9438B0 0x420AE2ED:0x3D9438D0 0x4200A085:0x3D9438F0 0x42011831:0x3D943A20 0x42011D77:0x3D943A50 0x42011EB2:0x3D943A70 0x42012BC1:0x3D943AA0 0x42011FAE:0x3D943AE0
E (542543) task_wdt: Print CPU 1 backtrace
Backtrace: 0x4037B0E9:0x3FC9C530 0x403772B1:0x3FC9C550 0x400559DD:0x3FCACA30 |<-CORRUPTED
E (547543) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (547543) task_wdt: - IDLE (CPU 0)
E (547543) task_wdt: Tasks currently running:
E (547543) task_wdt: CPU 0: http_stream_wri
E (547543) task_wdt: CPU 1: IDLE
E (547543) task_wdt: Print CPU 0 (current core) backtrace
Backtrace: 0x420896AE:0x3FC9BF20 0x403772B1:0x3FC9BF50 0x420E606C:0x3D943370 0x420C56A7:0x3D943390 0x420C34D0:0x3D9433D0 0x420C3559:0x3D943400 0x420BCD49:0x3D943420 0x420BCF3A:0x3D943440 0x420AFEBD:0x3D943460 0x420AFF79:0x3D9434F0 0x420B06DD:0x3D943520 0x420B307E:0x3D943690 0x420B0F95:0x3D943740 0x420B0FB8:0x3D943760 0x420B0FDE:0x3D943780 0x420F0CC1:0x3D9437A0 0x420F05E1:0x3D9437C0 0x420F08FA:0x3D943840 0x420EF9E2:0x3D943870 0x4214019D:0x3D943890 0x420AD56A:0x3D9438B0 0x420AE2ED:0x3D9438D0 0x4200A085:0x3D9438F0 0x42011831:0x3D943A20 0x42011D77:0x3D943A50 0x42011EB2:0x3D943A70 0x42012BC1:0x3D943AA0 0x42011FAE:0x3D943AE0
E (547543) task_wdt: Print CPU 1 backtrace
Backtrace: 0x4037B0E9:0x3FC9C530 0x403772B1:0x3FC9C550 0x400559DD:0x3FCACA30 |<-CORRUPTED
E (552543) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (552543) task_wdt: - IDLE (CPU 0)
E (552543) task_wdt: Tasks currently running:
E (552543) task_wdt: CPU 0: http_stream_wri
E (552543) task_wdt: CPU 1: fetch_task
E (552543) task_wdt: Print CPU 0 (current core) backtrace
Backtrace: 0x420896AE:0x3FC9BF20 0x403772B1:0x3FC9BF50 0x420E6067:0x3D943370 0x420C56A7:0x3D943390 0x420C34D0:0x3D9433D0 0x420C3559:0x3D943400 0x420BCD49:0x3D943420 0x420BCF3A:0x3D943440 0x420AFEBD:0x3D943460 0x420AFF79:0x3D9434F0 0x420B06DD:0x3D943520 0x420B307E:0x3D943690 0x420B0F95:0x3D943740 0x420B0FB8:0x3D943760 0x420B0FDE:0x3D943780 0x420F0CC1:0x3D9437A0 0x420F05E1:0x3D9437C0 0x420F08FA:0x3D943840 0x420EF9E2:0x3D943870 0x4214019D:0x3D943890 0x420AD56A:0x3D9438B0 0x420AE2ED:0x3D9438D0 0x4200A085:0x3D9438F0 0x42011831:0x3D943A20 0x42011D77:0x3D943A50 0x42011EB2:0x3D943A70 0x42012BC1:0x3D943AA0 0x42011FAE:0x3D943AE0
E (552543) task_wdt: Print CPU 1 backtrace
Backtrace: 0x4037B0E9:0x3FC9C530 0x403772B1:0x3FC9C550 0x400559DD:0x3FCACA30 |<-CORRUPTED
WILLOW: failed to read HTTP POST response
Im getting this error "E (17:30:07.558) WILLOW: failed to read HTTP POST response" after i have flashed it and connected it. any one faced any similar issue and know how to resolve this?
Unable to find image 'willow:latest' locally
Following the instructions from https://github.com/toverainc/willow/blob/main/README.md when running ./utils.sh docker on macOS I get the following error:
Unable to find image 'willow:latest' locally
docker: Error response from daemon: pull access denied for willow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
WDT bootloop on first-time flash (S3 version 0.1)
Morning! Great looking project!
I went through build and flash for the first time just now, and hit WDT a bootloop :(
My ESP BOX is pretty old, not sure if that might be the issue; what version do you run with usually?
Any ideas?
Here's the serial output:
Rebooting...
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0xb (SPI_FAST_FLASH_BOOT)
Saved PC:0x40379e3c
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x1754
load:0x403c9700,len:0xbf8
load:0x403cc700,len:0x324c
entry 0x403c9964
I (24) boot: ESP-IDF v4.4.3-347-g9ee3c8337d-dirty 2nd stage bootloader
I (24) boot: compile time 20:49:43
I (24) boot: chip revision: 0
I (28) boot.esp32s3: Boot SPI Speed : 80MHz
I (32) boot.esp32s3: SPI Mode : DIO
I (37) boot.esp32s3: SPI Flash Size : 16MB
I (42) boot: Enabling RNG early entropy source...
I (47) boot: Partition Table:
I (51) boot: ## Label Usage Type ST Offset Length
I (58) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (66) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (73) boot: 2 factory factory app 00 00 00010000 00200000
I (81) boot: 3 model Unknown data 01 82 00210000 00500000
I (88) boot: 4 audio Unknown data 01 82 00710000 00080000
I (96) boot: End of partition table
I (100) esp_image: segment 0: paddr=00010020 vaddr=3c120020 size=43d30h (277808) map
I (158) esp_image: segment 1: paddr=00053d58 vaddr=3fc9f610 size=070e4h ( 28900) load
I (164) esp_image: segment 2: paddr=0005ae44 vaddr=40378000 size=051d4h ( 20948) load
I (169) esp_image: segment 3: paddr=00060020 vaddr=42000020 size=11ebech (1174508) map
I (383) esp_image: segment 4: paddr=0017ec14 vaddr=4037d1d4 size=12438h ( 74808) load
I (408) boot: Loaded app from partition at offset 0x10000
I (409) boot: Disabling RNG early entropy source...
I (420) opi psram: vendor id : 0x0d (AP)
I (420) opi psram: dev id : 0x02 (generation 3)
I (421) opi psram: density : 0x03 (64 Mbit)
I (424) opi psram: good-die : 0x01 (Pass)
I (429) opi psram: Latency : 0x01 (Fixed)
I (434) opi psram: VCC : 0x01 (3V)
I (438) opi psram: SRF : 0x01 (Fast Refresh)
I (444) opi psram: BurstType : 0x01 (Hybrid Wrap)
I (449) opi psram: BurstLen : 0x01 (32 Byte)
I (454) opi psram: Readlatency : 0x02 (10 cycles@Fixed)
I (460) opi psram: DriveStrength: 0x00 (1/1)
W (465) PSRAM: DO NOT USE FOR MASS PRODUCTION! Timing parameters will be updated in future IDF version.
I (476) spiram: Found 64MBit SPI RAM device
I (480) spiram: SPI RAM mode: sram 80m
I (484) spiram: PSRAM initialized, cache is in normal (1-core) mode.
I (491) cpu_start: Pro cpu up.
I (495) cpu_start: Starting app cpu, entry point is 0x403796f4
I (484) cpu_start: App cpu up.
I (787) spiram: SPI SRAM memory test OK
I (796) cpu_start: Pro cpu start user code
I (796) cpu_start: cpu freq: 240000000
I (796) cpu_start: Application information:
I (799) cpu_start: Project name: willow
I (803) cpu_start: App version: 1
I (808) cpu_start: Compile time: May 17 2023 06:46:49
I (814) cpu_start: ELF file SHA256: 62f4542a22a8af3e...
I (820) cpu_start: ESP-IDF: v4.4.3-347-g9ee3c8337d-dirty
I (827) heap_init: Initializing. RAM available for dynamic allocation:
I (834) heap_init: At 3FCAB218 len 0003E4F8 (249 KiB): D/IRAM
I (840) heap_init: At 3FCE9710 len 00005724 (21 KiB): STACK/DRAM
I (847) heap_init: At 600FE000 len 00002000 (8 KiB): RTCRAM
I (853) spiram: Adding pool of 8192K of external SPI memory to heap allocator
I (862) spi_flash: detected chip: gd
I (866) spi_flash: flash io: dio
I (870) sleep: Configure to isolate all GPIO pins in sleep state
I (876) sleep: Enable automatic switching of GPIO sleep configuration
I (883) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (904) spiram: Reserving pool of 16K of internal memory for DMA/internal allocations
I (00:11:59.350) gpio: GPIO[45]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
I (00:11:59.361) gpio: GPIO[4]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
I (00:11:59.370) gpio: GPIO[48]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
I (00:11:59.491) AUDIO_THREAD: The esp_periph task allocate stack on external memory
I (00:11:59.492) LVGL: Starting LVGL task
I (00:11:59.505) WILLOW: initializing SNTP client
I (19:11:59.506) WILLOW: Using configured SNTP server 'pool.ntp.org'
I (19:11:59.512) pp: pp rom version: e7ae62f
I (19:11:59.512) net80211: net80211 rom version: e7ae62f
I (1084) wifi:wifi driver task: 3fcb7144, prio:23, stack:6656, core=0
I (19:11:59.524) system_api: Base MAC address is not set
I (19:11:59.530) system_api: read default base MAC address from EFUSE
I (1094) wifi:wifi firmware version: b5b20ea
I (1104) wifi:wifi certification version: v7.0
I (1104) wifi:config NVS flash: disabled
I (1104) wifi:config nano formating: disabled
I (1114) wifi:Init data frame dynamic rx buffer num: 32
I (1114) wifi:Init management frame dynamic rx buffer num: 32
I (1124) wifi:Init management short buffer num: 32
I (1124) wifi:Init static tx buffer num: 16
I (1134) wifi:Init tx cache buffer num: 32
I (1134) wifi:Init static tx FG buffer num: 2
I (1134) wifi:Init static rx buffer size: 1600
I (1144) wifi:Init static rx buffer num: 10
I (1144) wifi:Init dynamic rx buffer num: 32
I (19:11:59.592) wifi_init: rx ba win: 6
I (19:11:59.596) wifi_init: tcpip mbox: 32
I (19:11:59.601) wifi_init: udp mbox: 6
I (19:11:59.606) wifi_init: tcp mbox: 6
I (19:11:59.610) wifi_init: tcp tx win: 5744
I (19:11:59.615) wifi_init: tcp rx win: 5744
I (19:11:59.620) wifi_init: tcp mss: 1440
I (19:11:59.625) wifi_init: WiFi/LWIP prefer SPIRAM
I (19:11:59.630) wifi_init: WiFi IRAM OP enabled
I (19:11:59.636) wifi_init: WiFi RX IRAM OP enabled
I (1204) wifi:Set ps type: 2
I (19:11:59.644) phy_init: phy_version 540,a5d905b,Oct 20 2022,19:36:11
I (1234) wifi:mode : sta (7c:df:a1:e1:ac:4c)
I (1234) wifi:enable tsf
I (1964) wifi:new:<7,0>, old:<1,0>, ap:<255,255>, sta:<7,0>, prof:1
I (2764) wifi:state: init -> auth (b0)
I (2764) wifi:state: auth -> assoc (0)
I (2774) wifi:state: assoc -> run (10)
I (2774) wifi:state: run -> init (2c0)
I (2774) wifi:new:<7,0>, old:<7,0>, ap:<255,255>, sta:<7,0>, prof:1
W (19:12:01.217) PERIPH_WIFI: Wi-Fi disconnected from SSID PiAccessPoint, auto-reconnect enabled, reconnect after 1000 ms
W (19:12:04.625) PERIPH_WIFI: Wi-Fi disconnected from SSID PiAccessPoint, auto-reconnect enabled, reconnect after 1000 ms
I (7184) wifi:new:<7,0>, old:<7,0>, ap:<255,255>, sta:<7,0>, prof:1
I (7184) wifi:state: init -> auth (b0)
I (7184) wifi:state: auth -> assoc (0)
I (7194) wifi:state: assoc -> run (10)
I (7204) wifi:connected with PiAccessPoint, aid = 3, channel 7, BW20, bssid = dc:a6:32:ee:db:75
I (7204) wifi:security: WPA2-PSK, phy: bgn, rssi: -42
I (7204) wifi:pm start, type: 2
I (7204) wifi:set rx beacon pti, rx_bcn_pti: 0, bcn_timeout: 0, mt_pti: 25000, mt_time: 10000
W (19:12:05.658) PERIPH_WIFI: WiFi Event cb, Unhandle event_base:WIFI_EVENT, event_id:4
I (7214) wifi:<ba-add>idx:0 (ifx:0, dc:a6:32:ee:db:75), tid:6, ssn:0, winSize:64
I (7294) wifi:AP's beacon interval = 102400 us, DTIM period = 2
I (19:12:06.652) esp_netif_handlers: sta ip: 192.168.4.15, mask: 255.255.255.0, gw: 192.168.4.1
I (19:12:06.652) PERIPH_WIFI: Got ip:192.168.4.15
I (8214) wifi:Set ps type: 0
I (19:12:06.664) DRV8311: ES8311 in Slave mode
I (19:12:06.676) gpio: GPIO[46]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
W (19:12:06.681) I2C_BUS: i2c_bus_create:58: I2C bus has been already created, [port:0]
I (19:12:06.687) ES7210: ES7210 in Slave mode
I (19:12:06.698) ES7210: Enable ES7210_INPUT_MIC1
I (19:12:06.702) ES7210: Enable ES7210_INPUT_MIC2
W (19:12:06.705) ES7210: Disable TDM mode. ES7210_SDP_INTERFACE2_REG12: 0
I (19:12:06.711) ES7210: config fmt 60
I (19:12:06.715) AUDIO_HAL: Codec mode is 3, Ctrl:1
I (19:12:06.723) WILLOW: audio_hal_ctrl_codec: ESP_OK
E (11074) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (11074) task_wdt: - IDLE (CPU 1)
E (11074) task_wdt: Tasks currently running:
E (11074) task_wdt: CPU 0: IDLE
E (11074) task_wdt: CPU 1: LVGL task
E (11074) task_wdt: Aborting.
abort() was called at PC 0x42086bf8 on core 0
Backtrace: 0x4037a1ba:0x3fca0110 0x40383c31:0x3fca0130 0x4038c14e:0x3fca0150 0x42086bf8:0x3fca01c0 0x4037b181:0x3fca01f0 0x4211a713:0x3fcb2860 0x42003bfd:0x3fcb2880 0x40385023:0x3fcb28a0
Tnx!
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.