The repo has implementing an esp32 standalone MQTT audio streamer. Is is desinged to work as a satellite for Rhasspy (https://rhasspy.readthedocs.io/en/latest/). It supports multiple devices
License: GNU General Public License v3.0
Hi,
i successfully installed on an Atom Echo, it seems to work (ping, webui reachable), but it keeps rebooting every few seconds in a loop when in local hotword mode.
`ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:8896
load:0x40080400,len:5828
entry 0x400806ac
BootiM5Atom initializing...Loading configuration
{
"mqtt_host": "10.123.1.2",
"mqtt_port": 1883,
"mqtt_user": "",
"mqtt_pass": "",
"mqtt_valid": true,
"mute_input": false,
"mute_output": false,
"amp_output": 0,
"brightness": 30,
"hotword_brightness": 100,
"hotword_detection": 0,
"volume": 100,
"gain": 5
}
Enter WifiDisconnected
Total heap: 270760
Free heap: 196232
Enter WifiConnected
Connected to Wifi with IP: 10.123.1.33
Enter MQTTDisconnected
Enter MQTTConnected
Connected as satellite
Enter Idle
Backtrace: 0x4008ca58:0x3ffbe170 0x4008cc89:0x3ffbe190 0x400ec4b3:0x3ffbe1b0 0x40084ab9:0x3ffbe1d0 0x4000bfed:0x3ffeb550 0x4008a181:0x3ffeb560 0x400896e5:0x3ffeb580 0x400d2e75:0x3ffeb5a0 0x40089015:0x3ffeb820
Rebooting...
ets Jun 8 2016 00:22:57
`
The Backtrace looks the same every time. Should "local hotword" even work yet...?
Any ideas?
regards,
/hp
Good evening,
i tried to reproduce the steps mentioned in the readme, but after installing the libraries Arduino is throwing the following compile errors:
/Users/norbert/Downloads/Matrix-Voice-ESP32-MQTT-Audio-Streamer-master/MatrixVoiceAudioServer/MatrixVoiceAudioServer.ino: In function 'void onMqttMessage(char*, char*, AsyncMqttClientMessageProperties, size_t, size_t, size_t)':
MatrixVoiceAudioServer:245:7: error: 'StaticJsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:245:29: error: 'jsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:247:16: error: 'ArduinoJson::JsonObject {aka class ArduinoJson670_0_0::ObjectRef}' has no member named 'success'
if (root.success()) {
^
MatrixVoiceAudioServer:278:7: error: 'StaticJsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:278:29: error: 'jsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:280:16: error: 'ArduinoJson::JsonObject {aka class ArduinoJson670_0_0::ObjectRef}' has no member named 'success'
if (root.success()) {
^
MatrixVoiceAudioServer:299:7: error: 'StaticJsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:299:29: error: 'jsonBuffer' was not declared in this scope
StaticJsonBuffer<300> jsonBuffer;
^
MatrixVoiceAudioServer:301:16: error: 'ArduinoJson::JsonObject {aka class ArduinoJson670_0_0::ObjectRef}' has no member named 'success'
if (root.success()) {
^
Mehrere Bibliotheken wurden für "WiFi.h" gefunden
Benutzt: /Users/norbert/Library/Arduino15/packages/esp32/hardware/esp32/1.0.0/libraries/WiFi
Nicht benutzt: /Applications/Arduino.app/Contents/Java/libraries/WiFi
exit status 1
'StaticJsonBuffer' was not declared in this scope
Do you have a hint how to fix this?
thanks in advance
Norbert
It would be really awesome if ESP32-Rhasspy-Satellite would support custom wake/sleep/error led patterns/animations, like Google or Alexa, possibly even with direction of arrival support lateron.
When playing around on a Matrix Voice I noticed that the LED color change is only in effect after the "wake" audio file has played. I tried to change this, since it seems unintuitive that, even though the wakeword has already been detected and the acknowledgment sound is already playing, the LEDs to not reflect this. Looking at the code this is due to the fact that the HotwordDetectedEvent is triggered after the sound file has been played. Is this part of the original hermes protocol? Or is this something that might be improved somehow?
The next step would be to allow LED animations, and then allow the acknowledgment sound and the LED animation to play in parallel, so they do not block each other.
Any hints whether this is achievable would be very much appreciated 🤓
Hey there, great project!
I just received my Matrix Voice and already tried it out by directly plugging it into a Raspberry Pi 3B+, which did work fine using the official guide.
However, I am not quite sure how I am supposed to configure the rhasspy master to make it communicate with the ESP32 satellite. Is there more documentation on this somewhere other than the Getting started guide?
As I understand it the Matrix is only used as an external microphone and speaker, constantly streaming microphone data via MQTT to rhasspy, is that correct? If so, how can I verify that? From what I can tell the documentation on debugging this whole setup is a little sparse at the moment and I would be very happy to contribute to it when I get this working :)
About my setup:
I have a rhasspy "master" running on a dedicated machine, without speaker or microphones attached. I want to use this master to do all of the work, like detecting the hotword, STT, TTS, dialogue management, etc.
I have successfully flashed the contents of this repo to the Matrix Voice device and it shows a constantly glowing blue ring, which from what I can tell by reading the code means either "WIFI connected" or "Idle", so I guess everything is fine.
On the rhasspy master I have altered my previously working configuration for the Raspberry Pi sattelite (which also ran a rhasspy instance), by enabling the "Audio Recording" component using the "Hermes MQTT" type, enabling the "Wake Word" using the "Porcupine" type as well as "Audio Playing" using the "Hermes MQTT" type. I have left the "Dialogue Management" on "Rhasspy" type, although I am unsure about that one.
Thx for you work!
EDIT:
I am not sure what changed, but it works! 🥳
I guess the question now becomes: Where would be the best place to document this?
The fact that there is a web interface at port 80 was also completely unknown to me before reading about in a comment from some other issue.
#31 seems to contain a basic overview of the correct configuration options.
After changing config in the web UI, the code will sometimes crash and reboot. It doesn't seem to be related to a specific parameter.
I noticed that while working in a fork but also confirmed this is the case with the "master" branch as well. I wasn't able to find the cause.
This bug is not critical as the config will be applied after the restart anyway and since configuration is rarely changed. But it might be related to something else or might have effect on a more "core" functionality in the project so maybe worth looking at.
Here are the crash dumps. I was able to witness two types:
Parameter mqtt_host, value 192.168.x
Mqtt host changed
Parameter mqtt_port, value 1883
Parameter mqtt_user, value x
Parameter mqtt_pass, value x
Parameter amp_output, value 1
Parameter volume, value 100
Parameter brightness, value 15
Parameter hw_brightness, value 15
Parameter hotword_detection, value 1
Parameter gain, value 5
Settings changed, saving configuration
Saving configuration
{
"mqtt_host": "192.168.x",
"mqtt_port": 1883,
"mqtt_user": "x",
"mqtt_pass": "x",
"mqtt_valid": true,
"mute_input": false,
"mute_output": false,
"amp_output": 1,
"brightness": 15,
"hotword_brightness": 15,
"hotword_detection": 1,
"volume": 100,
"gain": 5
}
Connect failed, retry
Enter MQTTDisconnected
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x400d95cb PS : 0x00060130 A0 : 0x800d979f A1 : 0x3ffbc020
A2 : 0x00000000 A3 : 0x3fff127c A4 : 0x00000000 A5 : 0x3fff0eee
A6 : 0x00000014 A7 : 0x3ffbb5c0 A8 : 0x800853a8 A9 : 0x3ffbc000
A10 : 0x3fff1474 A11 : 0x0000166f A12 : 0x5730a6b5 A13 : 0x00000b38
A14 : 0x00000000 A15 : 0x00000000 SAR : 0x00000010 EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff
Backtrace: 0x400d95cb:0x3ffbc020 0x400d979c:0x3ffbc040 0x400d97de:0x3ffbc060 0x4013b866:0x3ffbc090 0x40141eaf:0x3ffbc0b0 0x40147142:0x3ffbc0d0 0x4013327b:0x3ffbc0f0 0x40089015:0x3ffbc120
#0 0x400d95cb:0x3ffbc020 in AsyncClient::_lwip_fin(tcp_pcb*, signed char) at .pio\libdeps\esp32dev\AsyncTCP@src-7fb2940bccb78b8d2de6915ae328b7fc\src/AsyncTCP.cpp:1039
#1 0x400d979c:0x3ffbc040 in AsyncClient::_s_lwip_fin(void*, tcp_pcb*, signed char) at .pio\libdeps\esp32dev\AsyncTCP@src-7fb2940bccb78b8d2de6915ae328b7fc\src/AsyncTCP.cpp:1039
#2 0x400d97de:0x3ffbc060 in _tcp_recv(void*, tcp_pcb*, pbuf*, signed char) at .pio\libdeps\esp32dev\AsyncTCP@src-7fb2940bccb78b8d2de6915ae328b7fc\src/AsyncTCP.cpp:1039
#3 0x4013b866:0x3ffbc090 in tcp_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:475 (discriminator 1)
#4 0x40141eaf:0x3ffbc0b0 in ip4_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:746
#5 0x40147142:0x3ffbc0d0 in ethernet_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/netif/ethernet.c:184
#6 0x4013327b:0x3ffbc0f0 in tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:483
#7 0x40089015:0x3ffbc120 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)
Rebooting...
Parameter mqtt_host, value 192.168.x
Parameter mqtt_port, value 1883
Parameter mqtt_user, value x
Mqtt user changed
Parameter mqtt_pass, value x
Parameter amp_output, value 1
Parameter volume, value 100
Parameter brightness, value 15
Parameter hw_brightness, value 15
Parameter hotword_detection, value 1
Parameter gain, value 5
Settings changed, saving configuration
Saving configuration
{
"mqtt_host": "192.168.x",
"mqtt_port": 1883,
"mqtt_user": "x",
"mqtt_pass": "x
"mqtt_valid": true,
"mute_input": false,
"mute_output": false,
"amp_output": 1,
"brightness": 15,
"hotword_brightness": 15,
"hotword_detection": 1,
"volume": 100,
"gain": 5
}
Enter MQTTDisconnected
Backtrace: 0x4008ca58:0x3ffbe170 0x4008cc89:0x3ffbe190 0x400ec4eb:0x3ffbe1b0 0x40084ab9:0x3ffbe1d0 0x4000bfed:0x3ffeb140 0x4008a181:0x3ffeb150 0x40088e20:0x3ffeb170 0x40147242:0x3ffeb1b0 0x40135ecd:0x3ffeb1d0 0x400dabd9:0x3ffeb1f0 0x400daf2d:0x3ffeb220 0x400da668:0x3ffeb250 0x400da6e6:0x3ffeb290 0x400d23b5:0x3ffeb2c0 0x400d23eb:0x3ffeb360 0x400d24eb:0x3ffeb380 0x400d2f53:0x3ffeb3a0 0x40089015:0x3ffeb820
#0 0x4008ca58:0x3ffbe170 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#1 0x4008cc89:0x3ffbe190 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#2 0x400ec4eb:0x3ffbe1b0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
#3 0x40084ab9:0x3ffbe1d0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
#4 0x4000bfed:0x3ffeb140 in ?? ??:0
#5 0x4008a181:0x3ffeb150 in vTaskExitCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
#6 0x40088e20:0x3ffeb170 in xQueueGenericReceive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1996
#7 0x40147242:0x3ffeb1b0 in sys_mutex_lock at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/port/esp32/freertos/sys_arch.c:203
#8 0x40135ecd:0x3ffeb1d0 in lwip_ioctl_r at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/sockets.c:3532
#9 0x400dabd9:0x3ffeb1f0 in WiFiClientRxBuffer::r_available() at C:\Users\arik\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src/WiFiClient.cpp:564
#10 0x400daf2d:0x3ffeb220 in WiFiClientRxBuffer::available() at C:\Users\arik\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src/WiFiClient.cpp:564
(inlined by) WiFiClient::available() at C:\Users\arik\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src/WiFiClient.cpp:458
#11 0x400da668:0x3ffeb250 in PubSubClient::connect(char const*, char const*, char const*, char const*, unsigned char, bool, char const*, bool) at .pio\libdeps\esp32dev\PubSubClient\src/PubSubClient.cpp:706
#12 0x400da6e6:0x3ffeb290 in PubSubClient::connect(char const*, char const*, char const*) at .pio\libdeps\esp32dev\PubSubClient\src/PubSubClient.cpp:706
#13 0x400d23b5:0x3ffeb2c0 in MQTTDisconnected::entry() at src/StateMachine.hpp:19
#14 0x400d23eb:0x3ffeb360 in void tinyfsm::Fsm<StateMachine>::transit<MQTTDisconnected>() at src/StateMachine.hpp:19
#15 0x400d24eb:0x3ffeb380 in Idle::react(MQTTDisconnectedEvent const&) at src/StateMachine.hpp:19
#16 0x400d2f53:0x3ffeb3a0 in void tinyfsm::Fsm<StateMachine>::dispatch<MQTTDisconnectedEvent>(MQTTDisconnectedEvent const&) at src/StateMachine.hpp:19
(inlined by) void send_event<MQTTDisconnectedEvent>(MQTTDisconnectedEvent const&) at src/StateMachine.hpp:270
(inlined by) I2Stask(void*) at src/StateMachine.hpp:567
#17 0x40089015:0x3ffeb820 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)
Rebooting...
Hey there,
i used your streamer for quite some time with a version before you renamed it with a matrix voice module.
It worked quite well, but i decided to update so i can use some of your newer features (more satellites, OTA updates..).
I heavily used the possibility to change the LED colors on the fly (when my assistant is saying something, when i go to bed and want the led to turn off, and so on).
This seem to not work anymore and there is just an update to the leds if i trigger a voice command.
Is there any way to change this behaviour?
Also the "hotword" color was activated when rhasspy was "not listening".. so when it was speaking i also got another color then on idle. i want to get this procedere back to change colors, when rhasspy is speaking.
i could trigger all these color changes over MQTT, if there would be an option to change colors whenever a profile update is incoming.
I would love to help you test any code to improve this software! :)
Thanks for this awesome project and keep up the very good work!
Hi.
As you may know, there are cheap standalone ESP32 Audio Kits out there.
They have some buttons, microphones and speaker output etc.
Would an adapted MQTT-Streamer for Rhasspy also work on them?
Hi,
according to effafaf i think that animation is supported on Matrix Voice.
However when i send {"brightness":20,"idle":[240,210,17,0],"hotword":[173,17,240,0],"animation":3} or even just animation, there is no change in animation (brightness and colors works). It's always a Solid color.
hotword_brightness also has no effect.
Edit: seems like the animation is only active while the rhasspy satellite speaks something.
Only hotword_brightness is an issue. Also settings brightness to 0 and hotword_brightness to 50 will result in always dark leds. I changed it by setting {"brightness":50,"hotword_brightness":50, "animation":1,"idle":[0,0,0,0], "hotword":[173,17,240,0]} . This way the leds are only on after hotword.
I have flashed the Matrix Voice. Only parameters changed in platformio.ini are:
DWIFI_SSID
DWIFI_PASS
DMQTT_IP
DMQTT_HOST
Build was succesfull and so was the deploy. After powering up the Voice standalone nothing happens apart from blue led underneath. The voice does not connect to the wifi so I plugged it back into the Pi and ran:
sudo minicom -D /dev/ttyS0
With the result repeating:
I (12) boot: ESP-IDF v4.0-dev-459-gba1ff1692 2nd stage bootloader
I (12) boot: compile time 11:07:17
I (12) boot: Enabling RNG early entropy source...
I (17) boot: SPI Speed : 40MHz
I (21) boot: SPI Mode : DIO
I (25) boot: SPI Flash Size : 4MB
I (29) boot: Partition Table:
I (33) boot: ## Label Usage Type ST Offset Length
I (40) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (47) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (55) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (62) boot: 3 factory factory app 00 00 00010000 00100000
I (70) boot: 4 ota_0 OTA app 00 10 00110000 00100000
I (77) boot: 5 ota_1 OTA app 00 11 00210000 00100000
I (85) boot: End of partition table
I (89) boot: Defaulting to factory image
I (94) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x3449c (21p
I (178) esp_image: segment 1: paddr=0x000444c4 vaddr=0x3ffbdb60 size=0x035f8 ( d
I (183) esp_image: segment 2: paddr=0x00047ac4 vaddr=0x40080000 size=0x00400 ( d
I (185) esp_image: segment 3: paddr=0x00047ecc vaddr=0x40080400 size=0x08144 ( d
I (207) esp_image: segment 4: paddr=0x00050018 vaddr=0x400d0018 size=0xa8214 (6p
I (448) esp_image: segment 5: paddr=0x000f8234 vaddr=0x40088544 size=0x0bef0 ( d
I (481) boot: Loaded app from partition at offset 0x10000
I (481) boot: Disabling RNG early entropy source...
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandl.
Core 0 register dump:
PC : 0x401750af PS : 0x00060930 A0 : 0x80139eab A1 : 0x3
A2 : 0x00000000 A3 : 0x000000fe A4 : 0x00000001 A5 : 0x0
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x801399d1 A9 : 0x3
A10 : 0x00000020 A11 : 0x4013958c A12 : 0x3ffe3a78 A13 : 0x0
A14 : 0x00000000 A15 : 0x00000000 SAR : 0x0000001b EXCCAUSE: 0x0
EXCVADDR: 0x00000000 LBEG : 0x4000c46c LEND : 0x4000c477 LCOUNT : 0x0
Backtrace: 0x401750af:0x3ffe3a90 0x40139ea8:0x3ffe3ab0 0x400d492d:0x3ffe3ae0 0x0
Rebooting...
Disclaimer I'm a bit of a novice in the this area and I have not used the Voice for anything else yet apart from running everloop demo.
The Pi is running a fresh install of Raspbian Stretch followed by setup steps from here: https://www.hackster.io/matrix-labs/program-matrix-voice-esp32-with-vs-code-using-platformio-3dd498
Hi,
taking the topic here...
I want to use an ESP32-POE-ISO by Olimex. Is there a way to use this project with the Arduino IDE?
I feel a little stupid asking this question, because I have the feeling, I am missing something obvious...
Well, once I have it running I will try to add the missing features for the ESP32-POE-ISO board.
This is a continuation of discussion here.
I've managed to get Snips to recognise the wake-word, but only right after booting the Atom Echo.
Snips does recognise that there is audio input.
While in idle mode, Snips Watch indicates that audio is being heard.
[14:55:18] [VoiceActivity] Down on site atomecho
[14:55:21] [VoiceActivity] Up on site atomecho
[14:55:22] [VoiceActivity] Down on site atomecho
[14:55:35] [VoiceActivity] Up on site atomecho
[14:55:53] [VoiceActivity] Down on site atomecho
[14:55:55] [VoiceActivity] Up on site atomecho
[14:55:56] [VoiceActivity] Down on site atomecho
[14:56:00] [VoiceActivity] Up on site atomecho
If I press the button to start a session, a session is created, and the dialogue manager listens to the stream from the Atom Echo. But the voice input is not recognised as a voice command:
[14:56:10] [Dialogue] was asked to start a session on site atomecho
[14:56:10] [Asr] was asked to stop listening on site atomecho
[14:56:10] [Hotword] was asked to toggle itself 'off' on site atomecho
[14:56:10] [Dialogue] session with id 'fe685174-5053-4651-8756-8cb3b066003e' was started on site atomecho
[14:56:10] [Asr] was asked to listen on site atomecho
[14:56:11] [VoiceActivity] Down on site atomecho
[14:56:12] [VoiceActivity] Up on site atomecho
[14:56:15] [VoiceActivity] Down on site atomecho
[14:56:16] [VoiceActivity] Up on site atomecho
[14:56:18] [VoiceActivity] Down on site atomecho
[14:56:19] [VoiceActivity] Up on site atomecho
[14:56:26] [Dialogue] session with id 'fe685174-5053-4651-8756-8cb3b066003e' was ended on site atomecho. The session was ended because one of the component didn't respond in a timely manner
[14:56:26] [Asr] was asked to stop listening on site atomecho
[14:56:26] [Hotword] was asked to toggle itself 'on' on site atomecho
If I don't speak into the Raspberry Pi version, then things look a bit different.
[15:07:45] [VoiceActivity] Up on site azrxidia
[15:07:46] [Hotword] detected on site azrxidia, for model hey_snips
[15:07:46] [Asr] was asked to stop listening on site azrxidia
[15:07:46] [Hotword] was asked to toggle itself 'off' on site azrxidia
[15:07:46] [Dialogue] session with id '16427483-191a-4c39-9f0b-199dd4cb0e7e' was started on site azrxidia
[15:07:46] [Asr] was asked to listen on site azrxidia
[15:07:46] [VoiceActivity] Up on site atomecho
[15:07:47] [VoiceActivity] Down on site azrxidia
[15:07:50] [Asr] captured text "" in 4.0s
[15:07:50] [Asr] was asked to stop listening on site azrxidia
[15:07:50] [Dialogue] session with id '16427483-191a-4c39-9f0b-199dd4cb0e7e' was ended on site azrxidia. The session was ended because the platform didn't understand the user
[15:07:50] [Asr] was asked to stop listening on site azrxidia
So what would support the idea that some MQTT message is missing.
As an aside, I also noticed that the wave header is slightly different:
ESP32
RIFF,WAVEfmt ?>}datay?u?y?t?z?v?|?~?q?v?u??s?z?u?y?t?v?v?w?y?q?o?u?|?x?y?n?t?t?u?r?r?v?s?x?q?w?s?x?m?v?s?o?t?s?y?s?w?u?o?~?w?s?t?s?u?y?~?s?~?t?u?{Հ?x?u?u?u?}?xՁ?y?x?x?zՀ?s?u?s?v?z?z?z?p?n?n?w?r?v?z?p?t?r?q?w?|?x?u?o?q?|?y?y?t?t?o?~?}?y?w?p?w?}?~?t?v?v?z?{?{?{?z?u?xՌՋ?~?}?w?yՄՅՂ?{?{?yՂՈ?|Մ?v?~ՂՃ?Հ?|?wՁ?Ղ?z?|?|Յ?ՂՀ?|?}Ղ???|?~Հ?~Մ?y?zՀՈՃՁՂ?}ՆՋ?Ձ?zՄՂՈՁ?Մ?Ո?ՇՂՃՂ?}?}??|ՅՃՄՄՃ?|Ձ?~ՊՏ?}ՂՀՅՉՅՇՂ?~?yՊՄՄՁ?x?~Ձ??~?{?~?zՂ?{?y?~?}?{?|?|?y?z??~?{Ղ?qՀ?uՂ?t?w?|?z?~?x?{?
USB microphone on Raspberry Pi:
RIFF4WAVEfmt ?>}tim??wdata????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????#")%##0$*%%AAHNMVZPHNWALKGC))????????!
???????/!>@>$$2*8@1-?>GIKB:ECV=3/3<19C?=5H=80)&
) <@5/,?23JH;
I also check the output of the various commands in Mosquitto to find out which exact message was missing.
---SNIPS----
hermes/hotword/azrxidia/detected {"siteId":"azrxidia","modelId":"hey_snips","modelVersion":"workflow-hey_snips_subww_feedback_10seeds-2018_12_04T12_13_05_evaluated_model_0002","modelType":"universal","currentSensitivity":0.5,"detectionSignalMs":1614003432719,"endSignalMs":1614003432719}
hermes/asr/stopListening ä"siteId":"azrxidia","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e"å
hermes/hotword/toggleOff ä"siteId":"azrxidia","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e"å
hermes/dialogueManager/sessionStarted ä"sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e","customData":null,"siteId":"azrxidia","reactivatedFromSessionId":nullå
hermes/asr/startListening ä"siteId":"azrxidia","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e","startSignalMs":1614003432719å
hermes/audioServer/azrxidia/replayRequest {"requestId":"azrxidia-1614003432719","startAtMs":1614003432719,"siteId":"azrxidia"}
hermes/audioServer/azrxidia/replayResponse RIFF^WAVEfmt ?>}tim??wrpidazrxidia-1614003432719rprf
hermes/audioServer/azrxidia/replayResponse
hermes/audioServer/azrxidia/replayResponse
hermes/audioServer/azrxidia/replayResponse
hermes/asr/textCaptured ä"text":"what time is it","likelihood":1.0,"tokens":Ää"value":"what","confidence":1.0,"rangeStart":0,"rangeEnd":4,"time":ä"start":0.0,"end":1.05åå,ä"value":"time","confidence":1.0,"rangeStart":5,"rangeEnd":9,"time":ä"start":1.05,"end":1.17åå,ä"value":"is","confidence":1.0,"rangeStart":10,"rangeEnd":12,"time":ä"start":1.17,"end":1.3199999åå,ä"value":"it","confidence":1.0,"rangeStart":13,"rangeEnd":15,"time":ä"start":1.3199999,"end":2.1ååÅ,"seconds":2.0,"siteId":"azrxidia","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e"å
hermes/asr/stopListening ä"siteId":"azrxidia","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e"å
hermes/nlu/query ä"input":"what time is it","asrTokens":Ää"value":"what","confidence":1.0,"rangeStart":0,"rangeEnd":4,"time":ä"start":0.0,"end":1.05åå,ä"value":"time","confidence":1.0,"rangeStart":5,"rangeEnd":9,"time":ä"start":1.05,"end":1.17åå,ä"value":"is","confidence":1.0,"rangeStart":10,"rangeEnd":12,"time":ä"start":1.17,"end":1.3199999åå,ä"value":"it","confidence":1.0,"rangeStart":13,"rangeEnd":15,"time":ä"start":1.3199999,"end":2.1ååÅ,"intentFilter":Ä"createcandle:get_time","createcandle:set_value","createcandle:stop_timer","createcandle:set_timer","createcandle:get_value","createcandle:set_state","createcandle:get_boolean","createcandle:list_timers","createcandle:get_timer_count"Å,"id":"1eee4474-c274-4a09-a7a5-7a65229839fa","sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e"å
hermes/nlu/intentParsed ä"id":"1eee4474-c274-4a09-a7a5-7a65229839fa","input":"what time is it","intent":ä"intentName":"createcandle:get_time","confidenceScore":1.0å,"slots":ÄÅ,"sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e","alternatives":Ää"intentName":"createcandle:get_value","confidenceScore":0.06613055,"slots":ÄÅå,ä"intentName":"createcandle:list_timers","confidenceScore":0.048560027,"slots":ÄÅåÅå
hermes/intent/createcandle:get_time ä"sessionId":"f0b42455-fc95-4873-84ba-c6136b1dec3e","customData":null,"siteId":"azrxidia","input":"what time is it","asrTokens":ÄÄä"value":"what","confidence":1.0,"rangeStart":0,"rangeEnd":4,"time":ä"start":0.0,"end":1.05åå,ä"value":"time","confidence":1.0,"rangeStart":5,"rangeEnd":9,"time":ä"start":1.05,"end":1.17åå,ä"value":"is","confidence":1.0,"rangeStart":10,"rangeEnd":12,"time":ä"start":1.17,"end":1.3199999åå,ä"value":"it","confidence":1.0,"rangeStart":13,"rangeEnd":15,"time":ä"start":1.3199999,"end":2.1ååÅÅ,"asrConfidence":1.0,"intent":ä"intentName":"createcandle:get_time","confidenceScore":1.0å,"slots":ÄÅ,"alternatives":Ää"intentName":"createcandle:get_value","confidenceScore":0.06613055,"slots":ÄÅå,ä"intentName":"createcandle:list_timers","confidenceScore":0.048560027,"slots":ÄÅåÅå
also:
hermes/voiceActivity/azrxidia/vadDown {"siteId":"azrxidia","signalMs":1614003434199}
hermes/voiceActivity/azrxidia/vadUp ä"siteId":"azrxidia","signalMs":1614003432077å
----Atom Echo button----
hermes/dialogueManager/startSession æ"init":æ"type":"action","canBeEnqueued": falseå,"siteId":"atomecho"å
hermes/asr/stopListening æ"siteId":"atomecho","sessionId":"cd6118bf-a971-4921-a6b5-59aeb7967a3d"å
hermes/hotword/toggleOff æ"siteId":"atomecho","sessionId":"cd6118bf-a971-4921-a6b5-59aeb7967a3d"å
hermes/dialogueManager/sessionStarted æ"sessionId":"cd6118bf-a971-4921-a6b5-59aeb7967a3d","customData":null,"siteId":"atomecho","reactivatedFromSessionId":nullå
hermes/asr/startListening æ"siteId":"atomecho","sessionId":"cd6118bf-a971-4921-a6b5-59aeb7967a3d","startSignalMs":nullå
hermes/voco/atomecho/mute æ"mute": trueå
...
hermes/voiceActivity/atomecho/vadDown æ"siteId":"atomecho","signalMs":-386å
----Atom Echo hotword detected----
hermes/hotword/azrxidia/detected æ"siteId":"atomecho","modelId":"hey_snips","modelVersion":"workflow-hey_snips_subww_feedback_10seeds-2018_12_04T12_13_05_evaluated_model_0002","modelType":"universal","currentSensitivity":0.5,"detectionSignalMs":-70,"endSignalMs":-70å
hermes/voco/atomecho/play æ"sound_file": "start_of_input"å
hermes/audioServer/atomecho/audioFrame RIFF,WAVEfmt ?>ådata?????????ؽ??????????????????????????????????????????????ؽ????????غ????ؿؽ????غؿؾػؾؿ??ع??????ظ????????ؾ????ؾؾؾ????????ؽ??ؿ????ظ??????صطؽؼػغغ????????ؼ??ضؼؼشؾغ????ع??ؼ??ؼ??ظؼر??ػعػظ??ؾؼ??ذضظص??زؾشؼطؾظؽؾطؽخرخصؼذزشسزظخعػش??زطزػغطضضجزعج??ضرذ??غؼحذظ??ظرش??ؼذطصثزضص??سشش??رشص??عسصح??عظخؼجؽزػرخرشصشذظصحطظشغغصشسسسذسصغزضطشعغص??ؽظؿظرظطؼصؿ??عصغؼؽطػص???
hermes/asr/stopListening æ"siteId":"atomecho","sessionId":"a9e2c85d-cdfd-40ce-9ad6-ad30c9ed2868"å
hermes/audioServer/atomecho/audioFrame RIFF,WAVEfmt ?>ådata???غ??????ظؿؾ????ؼؾظغطغ??ظؿع??ط??ؾع??ططظعشظغ??ظؽعظؼطضسطضؼشرطسعسطظغؽؿ????صعصغ??غؿؼظط????????ػؾؽؼ????ؼ??صػظػظؾظػػععؼػ??ؾ??سؿشغػششرؽخغشػؽزؼضزسض??ضصض??سظػػرؽعؼػعظص??صغسؼؼػظؼظغؼؼظ????ظػ??ؿعغعظظ??ػع??ظحؽش??ضظعػطصذ??ظؿؽظطصؽصؽشغسسط??صؾؿغعغؽؽضعغ??ػػؽؽ??ؼص??ظدضعضصضغظخصػطؽضػتػشظرعصصدذشطزطخخظظذزظضصؾد??حرخ?
hermes/hotword/toggleOff æ"siteId":"atomecho","sessionId":"a9e2c85d-cdfd-40ce-9ad6-ad30c9ed2868"å
hermes/dialogueManager/sessionStarted æ"sessionId":"a9e2c85d-cdfd-40ce-9ad6-ad30c9ed2868","customData":null,"siteId":"atomecho","reactivatedFromSessionId":nullå
hermes/asr/startListening æ"siteId":"atomecho","sessionId":"a9e2c85d-cdfd-40ce-9ad6-ad30c9ed2868","startSignalMs":-70å
hermes/voco/atomecho/mute æ"mute": trueå
hermes/voiceActivity/atomecho/vadDown {"siteId":"atomecho","signalMs":-386}
Hello Rom,
Thanks for sharing such an amazing work. I'm new in programming, and have a silly question on your code. ( line 331-334&337 of MatrixVoiceAudioServer.ino)
for (int i=0;i<message_count;i++) {
for (uint32_t s = CHUNK*i; s < CHUNK * (i+1); s++) {
voicebuffer[s-(CHUNK*i)] = mics.Beam(s);
}
As far as I understanding, voicebuffer only has 256 elements, and the first voicebuffer ( when i =0 ) will be overlapped after running this part of code.
memcpy(voicemapped,voicebuffer,CHUNK*WIDTH)
Thus how to copy the voicebuffer to voicemapped from 0-511?
Thank you!
Hi,
is it possible to connect some LED Stripe (like WS2812 or APA102) to ESP 32 Audio Kit?
If so, can you explain how to do this?
Big Thanks!
Hi,
when Matrix Voice boots up, the leds change blue/red all the time and in mosquito log I see:
1608233147: New connection from 192.168.132.27 on port 1883. 1608233147: Socket error on client <unknown>, disconnecting.
any hints whats going on there?
Just a quick question, are there any particular reasons, why I couldn't use an esp8266 as a rhasspy satellite?
And when reading through the closed issues I saw someone writing something about using a button to start the recording. So i wanted to know does the device constantly stream audio to a Rhasspy base station and the wakeword and intent detection is done there or do I have to trigger the stream by pressing or holding a button?
I'm exploring if it's possible to create a privacy friendly smart doorbell / security camera based on an ESP32. Since I'm a fan of Rhasspy, I thought perhaps going the MQTT route isn't such a bad idea. In that case ESP32-Rhasspy-Satellite could perhaps form a basis for my experiment.
Do you think that once in a while streaming an additional JPG over MQTT back to the main server could work? Or would that just be too much for the ESP32? Is there any processing power or memory left?
I've currently been working with a TTGO camera with mic for my experiments, which I would then have to add a small audio amplifier board to.
Issuing "make" within the "OTABuilder" folder fails with
Makefile:6:` /make/project.mk: No such file or directory make: *** No rule to make target '/make/project.mk'. Stop.
It seems like, the makefile is not configured at all. Please add make-instructions to the Makefile so one can simply compile it via "make"
Thanks a lot for an amazing work!
I'm trying to install it on my MATRIX devices and thought it would be amazing if it could be used together with
ESPHome, which provides a super neat way to build firmware for ESP based on desired modules.
Readme mentions a known issue with sample rate. Instructs to use sample rate of 22K or lower.
I am getting tons of "Buffer underflow" and I assume this issue affects me. However it was not clear where to change this sample rate.
Is this the sample rate that is set in this code? Is this the sample rate that is set by the main Rhasspy site? Is this the sample rate of the microphone?
Can you elaborate here or in the documentation what is needed to be changed?
I can build the latest version but the folder structure seems to have changed quite a bit. There is no lon ger an ota folder and no ./install.sh ?
Basically I got no clue on how to install the new build.
Hello,
i´m new to rhasspy and to your esp32-satellite and i´ve setup rhasspy on a raspberry pi 4 and your software on a esp32-audio-kit 2.2.
I have managed to see the satellite on my mqtt-explorer producing audio and connected my server-rhasspy to it too.
But I failed in geting output or wakeword detection on the server.
Here is my config file :
{ "dialogue": { "satellite_site_ids": "satellite-01", "system": "hermes", "volume": "0.2" }, "handle": { "system": "command" }, "intent": { "satellite_site_ids": "satellite-01", "system": "fsticuffs" }, "microphone": { "pyaudio": { "device": "1" }, "system": "hermes" }, "mqtt": { "enabled": "true", "host": "192.168.188.23", "password": "test", "site_id": "server", "username": "test" }, "sounds": { "aplay": { "device": "sysdefault:CARD=Headphones", "volume": "0.6" }, "remote": { "url": "http://192.168.188.26:12101/api/play-wav" }, "system": "hermes" }, "speech_to_text": { "satellite_site_ids": "satellite-01", "system": "kaldi" }, "text_to_speech": { "satellite_site_ids": "satellite-01", "system": "nanotts" }, "wake": { "porcupine": { "keyword_path": "computer_raspberry-pi.ppn" }, "raven": { "keywords": { "Computer": { "enabled": true } } }, "satellite_site_ids": "satellite-01", "system": "porcupine" } }
How has the server to be configured to work with the satellite ?
Is there a documentation or a wiki ?
I can´t program code, but I am willing to contribute in other ways if I can !?
Can the wakeword-detection in the satellite be set to "Local"
Thank you for your work
Greetings from Aachen, Germany
As discussed in #23 (comment), i imagine the following scenarios:
Any other ideas?
Copy and configure the settings.ini.example to settings.ini and configure the proper parameters.
I can find no such file:
Hi,
I am new to this but the instructions seems pretty straightforward.
When i build however, i get an error of a missing library - ESPAsyncWebServer.h - output below
I have checked my sums but i wonder if i have missed a step or whether one of the dependency repositories has secretly changed.....
I have also had a quick search on github and only find the file here:-
luqmanhakimpens/esp32_sht31_oled
lib/ESPAsyncWebServer/src/ESPAsyncWebServer.h
afonsus1997/Pixels-Camp-PCB-Badge
CODE/ESP/ESPAsyncWebServer.h
Thanks.!
error details:-
Building in release mode
Compiling .pio\build\esp32dev\src\MatrixVoiceAudioServer.cpp.o
Compiling .pio\build\esp32dev\lib2b1\WiFi\WiFiMulti.cpp.o
Compiling .pio\build\esp32dev\lib2b1\WiFi\WiFiSTA.cpp.o
Compiling .pio\build\esp32dev\lib2b1\WiFi\WiFiScan.cpp.o
Compiling .pio\build\esp32dev\lib2b1\WiFi\WiFiServer.cpp.o
Compiling .pio\build\esp32dev\lib2b1\WiFi\WiFiUdp.cpp.o
src\MatrixVoiceAudioServer.cpp:104:31: fatal error: ESPAsyncWebServer.h: No such file or directory
compilation terminated.
Compiling .pio\build\esp32dev\lib2c2\ESPmDNS\ESPmDNS.cpp.o
*** [.pio\build\esp32dev\src\MatrixVoiceAudioServer.cpp.o] Error 1
My settings.ini General section is as follows
[General]
hostname=satellite
deployhost=192.168.43.24
siteId=satellite
;supported: M5ATOMECHO=0, MATRIXVOICE=1, AUDIOKIT=2, INMP441=3, INMP441MAX98357A=4
device_type=0
;network_type: 0: WiFi, 1: Ethernet
network_type=0
When I subscribe to hermes/# with mosquito_sub to see what the mic is sending I see a lot of:
hermes/audioServer/satellite/audioFrame RIFF,WAVEfmt >}data))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))
When I play something from rhasspy I see the proper logging being displayed and in the serial interface, yet not sound is played:
Samplerate: 16000, Channels: 1, Format: 1, Bits per Sample: 16
I wonder if I'm missing something?
Hi,
as I understand from following up the discussion from #30 , at the moment a local hotword detection is not implemented so remote hotword detection is now the only option. However, there is already local activation via a key press supported.
I propose to enable this as a choice, i.e. offer two settings for activation of Rhasspy voice recognition: Key Press Activation Only and Remote Detection + Key Press.
Changes to existing code are very limited, it uses the existing HW_REMOTE/HW_LOCAL concept. I have the code working locally.
Shall I submit a pull request or is this out of scope?
I am currently trying to setup this streamer on my matrix voice, unfortunately it seems like the device is not able to connect to my wifi, the LED is red at the beginning, turns to blue after a few seconds but I cannot find the device in the clients-list of my firewall and there is no connection to my MQTT broker.
This is my current config, flashing it works like without any problem.
Of course the ssid/mqtt settings are changed, but i triple-checked all params and passwords.
Has anyone an idea what I am doing wrong?
[Matrix]
hostname=10.10.10.30
siteId=matrixlivingroom
[Wifi]
ssid=myWifi
password=my.super!password?
[OTA]
password=OTA
port=3232
[MQTT]
ip=10.10.10.11
hostname=homeassistant
port=1883
username=myUser
password=mySuperMqttPassword
maxPacketSize=2000
I alluded to this issue in #53 and think I may have found the issue for this but cannot confirm the comment. The following code is as mentioned in the previous example from issue directly from the example however on the bits per sample there is a comment saying even though its set to 16 bit it's fixed at 12 bit.
i2s_config_t i2s_config = {
.mode = (i2s_mode_t)(I2S_MODE_MASTER),
.sample_rate = 16000,
.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT, // is fixed at 12bit, stereo, MSB
.channel_format = I2S_CHANNEL_FMT_ALL_RIGHT,
.communication_format = I2S_COMM_FORMAT_I2S,
.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
.dma_buf_count = 6,
.dma_buf_len = 60,
};
It doesn't really sense for the mic to be stereo in this case and from what I can tell from the amplifier datasheet it also doesn't make much sense so is likely either a copied comment from something else whenever they made the example or a software limitation of the SDK.
Going on the idea it was 12 bit and not 16 bit I tried to multiply the audio by various powers of 2 initially just to increase the loudness but was all overly loud and led to clipping. Trying to multiply by anything except a power of 2 leads to a crash that I cannot pin point.
I sadly lack the audio knowledge to figure out if its actually taking the PDM audio and making it 12 bit or 16 bit but a gain function would be nice. However due to the relative lack of power in the esp32 I am unsure if its smarter here or at rhasspy.
Hi,
Is there any way to debug a code running onESP32 on matix board ? How do you debug that code? Is there a possibility do to it by OTA ?
Thanks
I was able to set up everything properly working with Rhasspy at least if I set the global siteId of my Rhasspy instance to the same the matrix has.
My original idea was to use one Rhasspy instance and add multiple matrix voice as satellites, unfortunately, if i configure it this way, the LED is flashing green for a short time on wake word, turns back to blue and then to green again and the commands are mostly not recognised (1 out of 10 is recognised or so).
I am not sure, this might be of course an issue with Rhasspy itself, but maybe you have an idea on how to setup satellites the proper way?
This configuration is working great:
This configuration is not working:
Currently must pass the IP address of an mqtt broker in settings.ini or web config. It would be more flexible and in-line with best practices to allow to specify a hostname too.
Is it limited to an IP intentionally? It seems the mqtt libraries used by the project will accept hostnames as server address parameters.
Hi, I tried modify existing devices for INMP441 but without success.
Below the code with my esp32cam. I see raw data on Mqtt Explorer, but remote wakeword is not recognised by Rhasspy. Any ideas?
#include <driver/i2s.h>
// I2S pins
#define IIS_SCLK 14
#define IIS_WSLC 2
#define IIS_DSIN 15
#define IIS_DSOU -1 //not used, only for speaker
#define CONFIG_I2S_BCK_PIN IIS_SCLK
#define CONFIG_I2S_LRCK_PIN IIS_WSLC
#define CONFIG_I2S_DATA_PIN IIS_DSIN
#define CONFIG_I2S_DATA_IN_PIN IIS_DSOU
#define I2S_PORT I2S_NUM_0
class Inmp441 : public Device
{
public:
Inmp441();
void init();
bool readAudio(uint8_t *data, size_t size);
};
Inmp441::Inmp441() {};
void Inmp441::init() {
Serial.printf("Connect to Inmp441... ");
esp_err_t err = ESP_OK;
i2s_config_t i2s_config = {
.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
.sample_rate = 16000,
.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
.communication_format = i2s_comm_format_t(I2S_COMM_FORMAT_I2S),
.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
.dma_buf_count = 4,
.dma_buf_len = 64,
.use_apll = false,
.tx_desc_auto_clear = false,
.fixed_mclk = 0
};
err += i2s_driver_install(I2S_PORT, &i2s_config, 0, NULL);
i2s_pin_config_t tx_pin_config;
tx_pin_config.bck_io_num = CONFIG_I2S_BCK_PIN;
tx_pin_config.ws_io_num = CONFIG_I2S_LRCK_PIN;
tx_pin_config.data_out_num = CONFIG_I2S_DATA_PIN;
tx_pin_config.data_in_num = CONFIG_I2S_DATA_IN_PIN;
err += i2s_set_pin(I2S_PORT, &tx_pin_config);
err += i2s_set_clk(I2S_PORT, 16000, I2S_BITS_PER_SAMPLE_16BIT, I2S_CHANNEL_MONO);
return;
}
bool Inmp441::readAudio(uint8_t *data, size_t size) {
size_t byte_read;
i2s_read(I2S_PORT, data, size, &byte_read, (100 / portTICK_RATE_MS));
return true;
}
Hi @Romkabouter,
Already asked it at a Matrix forum. Just curios if you had a chance to take a look at WebSockets support?
I have a vosk-api installed on RPi. And there's also a simple WS server which accepts an audio stream and returns JSON transcript.
From a laptop I'm using a nodejs WS client which streams chunks captured from the integrated microphone via rec tool:
const RECORDER_OPTIONS = {
program: 'rec',
bits: 16,
channels: 1,
encoding: 'signed-integer',
rate: 8000,
type: 'wav',
silence: 0
}
So now I want to extend your library with WS. I found some EPS32 WS sample.
As far as I can see from your code, the main audio processing happens in
void Audiostream(void *p), where you send chunks via
audioServer.publish(audioFrameTopic.c_str(), (uint8_t *)payload, sizeof(payload));
Can you point me to where I can adjust audio settings to meet the above requirements (16b/8k/1ch/wav) before actual sending? As there's a lot of "magic" for me in your code yet. I just want to replace mqtt call with WS to test it with Kaldi as the initial step. Would be appreciated any help.
I am running a MATRIX Voice and I do hear the beeps when it hears the wake word & such, but I have not been able to figure out how the hub can play the audio that it generates on the satellite.
I have tried setting Hermes MQTT hoping that somehow it might be routed to the satellite that sent the request, but the audio is not played.
I have also tried to POST the audio to the satellite via HTTP post but I don't think there is an API running as part of this project.
Does anyone have a working solution? I plan to have a bunch of satellites around the house so my goal is to route the resulting audio back to the device that was spoken to for playback.
Hi all,
I have flashed my Matrix Voice ES32 with the newest version of the audio streamer. The MQTT based connection with snips works well, also the hotword detection and the intent detection.
Unfortunately, I cannot hear any reply (e. g. current time), except some noise.
The snips server is installed with default settings in an Ubuntu VM running on a FreeNAS.
Changing the sample rate and channels (e. g. to "32000" and "2" or "44100" and "2") did not result in working speech output.
Do you have any ideas what I can do to get the audio output via Matrix working?
Many thanks,
Roland
I saw a sdk from espressif https://github.com/espressif/esp-skainet/tree/master/examples/wake_word_detection with an example of Wake word detection feature.
Can you look that and tell me if you can implemented in your great project?
Thanks
https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/blob/master/matrixvoice.md
Copy and configure the settings.ini.example to settings.ini and configure the proper parameters.
I can find no such file:
As part of #88 I looked a bit into the usage of TinyFSM in the code. TinyFSM by itself does not care about multitasking / multicore execution. We are calling send_event from main task and I2STask. send_event will execute the respective event reat code which includes state transitions and also running entry and exit functions in the context of two different tasks. The state´s run function always executes on the main task loop.
If task switch would always happen when outside code run on behalf of the StateMachine this would not be a problem. But as far as I understand how Arduino on the ESP32 runs, this is not guaranteed.
Not yet sure if there is currently a real impact of this as the code runs pretty stable but this kind of problem is very difficult to track down when it appears...
One way to solve this issue would be to make sure that all TinyFSM code is executed in the main task by not directly using send_event in its original form in the I2STask. Would need to send an event to the main loop which in turn then calls send_event when it gets the control back.
Hi,
I followed your tutorial to the letter, I think.
The ESP flashed and all looks good. However the last part doesn't work for me:
"Make a change (or not) and do a Sketch -> Upload. The leds will turn WHITE
Sometimes uploading fails, just retry until it succeeds."
I thought at first it would be the IP address, but that checked out ok, I believe.
I tried a dozen times, but keep getting the "No response from the ESP" error from Arduino, after I enter the correct password. Also started over numerous times.
The leds do look white though and the device connected to the wifi on its own just fine. (I pinged it and checked the router)
Can someone help me with this issue?
Hi @Romkabouter,
I am just getting started with this and was wondering if it´s possible to also use your program on the Non-ESP32 version of matrix voice?
Best regards,
Thomas
I am slightly confused by the following code:
// Convert 4 byte little-endian to a long,
#define longword(bfr, ofs) (bfr[ofs + 3] << 24 | bfr[ofs + 2] << 16 | bfr[ofs + 1] << 8 | bfr[ofs + 0])
#define shortword(bfr, ofs) (bfr[ofs + 1] << 8 | bfr[ofs + 0])according to the comment this converts from little endian to long. Since the endianness of long is not stated I assumed, that it means the host endianness. However since the esp32 is little endian aswell this does not make much sense. It would probably help to use a naming convention similar to e.g. Linux https://linux.die.net/man/3/endian
Hello Romkabouter, congratulations for your work.
I'm a newbie, first time I use IDE Arduino…
I do step by step the Readme guide.
When I compile, I obtain this error:
Arduino : 1.8.9 (Windows Store 1.8.21.0) (Windows 10), Carte : "ESP32 Dev Module, Disabled, Default 4MB with spiffs (1.2MB APP/1.5MB SPIFFS), 240MHz (WiFi/BT), QIO, 80MHz, 4MB (32Mb), 115200, None"
MatrixVoiceAudioServer:49:26: error: wishbone_bus.h: No such file or directory
Plusieurs bibliothèque trouvées pour "WiFi.h"
compilation terminated.
Utilisé : C:\Users\leten\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.2\libraries\WiFi
Non utilisé : C:\Program Files\WindowsApps\ArduinoLLC.ArduinoIDE_1.8.21.0_x86__mdqgnx93n4wtt\libraries\WiFi
exit status 1
wishbone_bus.h: No such file or directory
Ce rapport pourrait être plus détaillé avec
l'option "Afficher les résultats détaillés de la compilation"
activée dans Fichier -> Préférences.
"
Please, could you help me to find an issue ?
Thank you in advance.
Hi, first of all, thanks for your great project. I'm trying to build using rewrite branch but i had encountered some issues which i don't know what is going on here.
I'm using Windows 10 and the tool is PlatformIO on VSCode. The settings.ini value is same as the settings.ini.example.
The reason to use rewrite branch because it support atom echo.
Some output logs:
...
Found 40 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <MATRIXVoiceESP32HAL> 0.77.0+sha.320c897
|-- <RingBuffer> 1.0.3+sha.06e8bf1
|-- <AsyncMqttClient> 0.8.2+sha.7f1ba48
| |-- <AsyncTCP> 1.1.1
|-- <AsyncTCP> 1.1.1+sha.ca8ac5f
|-- <PubSubClient> 2.8.0+sha.2d228f2
|-- <ArduinoJson> 6.17.2+sha.1360b6a
|-- <ESP Async WebServer> 1.2.3
| |-- <AsyncTCP> 1.1.1
| |-- <FS> 1.0
| |-- <WiFi> 1.0
| |-- <ArduinoJson> 6.17.2+sha.1360b6a
|-- <M5Atom> 0.0.1
| |-- <FastLED> 3.4.0
| | |-- <SPI> 1.0
| |-- <Wire> 1.0.1
|-- <FastLED> 3.4.0
| |-- <SPI> 1.0
|-- <AC101> 0.0.1
| |-- <Wire> 1.0.1
|-- <SPIFFS> 1.0
| |-- <FS> 1.0
|-- <WiFi> 1.0
|-- <ArduinoOTA> 1.0
| |-- <Update> 1.0
| |-- <WiFi> 1.0
| |-- <ESPmDNS> 1.0
| | |-- <WiFi> 1.0
|-- <libspeex>
Building in release mode
Compiling .pio\build\esp32dev\libefb\FastLED\FastLED.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\bitswap.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\colorpalettes.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\colorutils.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\hsv2rgb.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\lib8tion.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\noise.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\platforms.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\platforms\esp\32\clockless_rmt_esp32.cpp.o
.pio\libdeps\esp32dev\FastLED\src\platforms\esp\32\clockless_rmt_esp32.cpp: In static member function 'static void ESP32RMTController::interruptHandler(void*)':
.pio\libdeps\esp32dev\FastLED\src\platforms\esp\32\clockless_rmt_esp32.cpp:318:10: warning: unused variable 'stuff_to_do' [-Wunused-variable]
bool stuff_to_do = false;
^
Compiling .pio\build\esp32dev\libefb\FastLED\power_mgt.cpp.o
Compiling .pio\build\esp32dev\libefb\FastLED\wiring.cpp.o
Compiling .pio\build\esp32dev\src\Satellite.cpp.o
Generating partitions .pio\build\esp32dev\partitions.bin
Compiling .pio\build\esp32dev\liba6c\MATRIXVoiceESP32HAL\everloop.cpp.o
Compiling .pio\build\esp32dev\liba6c\MATRIXVoiceESP32HAL\matrix_driver.cpp.o
Compiling .pio\build\esp32dev\liba6c\MATRIXVoiceESP32HAL\microphone_array.cpp.o
Compiling .pio\build\esp32dev\liba6c\MATRIXVoiceESP32HAL\microphone_core.cpp.o
Compiling .pio\build\esp32dev\liba6c\MATRIXVoiceESP32HAL\wishbone_bus.cpp.o
Archiving .pio\build\esp32dev\lib21b\libRingBuffer.a
Compiling .pio\build\esp32dev\libe65\AsyncTCP\AsyncTCP.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\ConnAckPacket.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PingRespPacket.cpp.o
In file included from src\Satellite.cpp:97:0:
src/General.hpp:23:3: error: stray '\' in program
IPAddress mqtt_host = MQTT_IP;
^
src/General.hpp:23:3: error: stray '\' in program
In file included from .pio\libdeps\esp32dev\MATRIXVoiceESP32HAL\components\hal\microphone_core.cpp:31:0:
.pio\libdeps\esp32dev\MATRIXVoiceESP32HAL\components\hal\./microphone_array_location.h:33:14: warning: 'matrix_hal::micarray_location' defined but not used [-Wunused-variable]
static float micarray_location[8][2] = {
^
In file included from .pio/libdeps/esp32dev/M5Atom/src/M5Atom.h:52:0,
from src\devices/M5AtomEcho.hpp:1,
from src\Satellite.cpp:87:
.pio/libdeps/esp32dev/FastLED/src/FastLED.h:14:21: note: #pragma message: FastLED version 3.004.000
# pragma message "FastLED version 3.004.000"
^
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PubAckPacket.cpp.o
In file included from .pio/libdeps/esp32dev/FastLED/src/FastLED.h:65:0,
from .pio/libdeps/esp32dev/M5Atom/src/M5Atom.h:52,
from src\devices/M5AtomEcho.hpp:1,
from src\Satellite.cpp:87:
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PubCompPacket.cpp.o
.pio/libdeps/esp32dev/FastLED/src/fastspi.h:135:23: note: #pragma message: No hardware SPI pins defined. All SPI access will default to bitbanged output
# pragma message "No hardware SPI pins defined. All SPI access will default to bitbanged output"
^
Archiving .pio\build\esp32dev\liba6c\libMATRIXVoiceESP32HAL.a
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PubRecPacket.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PubRelPacket.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\PublishPacket.cpp.o
Archiving .pio\build\esp32dev\libe65\libAsyncTCP.a
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\SubAckPacket.cpp.o
Compiling .pio\build\esp32dev\lib18a\AsyncMqttClient\AsyncMqttClient\Packets\UnsubAckPacket.cpp.o
Compiling .pio\build\esp32dev\lib349\AsyncTCP@src-7fb2940bccb78b8d2de6915ae328b7fc\AsyncTCP.cpp.o
*** [.pio\build\esp32dev\src\Satellite.cpp.o] Error 1
============================================ [FAILED] Took 25.91 seconds ============================================
...
As the title of issue already indicates: I am not quite sure how to set / test the hotword.
Any tips in that regards?
Thanks a lot!
I've tried to create an Arduino version of this project. That may allow users of the WebThings Gateway to create new satellites by simply plugging an Atom Echo into their Raspberry Pi and click on a "make a new satellite" button.
https://github.com/createcandle/voco-mini-satellite
The goal is to make it compatible with Voco. Voco still relies on Snips for now, so I'm curious: would it be hard to make the code compatible with Snips again? Have there been big changes in this regard?
Is this correct for the Atom Echo?
{
"mqtt_host": "192.168.2.167",
"mqtt_port": 1883,
"mqtt_user": "",
"mqtt_pass": "",
"mqtt_valid": true,
"mute_input": false,
"mute_output": false,
"amp_output": 0,
"brightness": 30,
"hotword_brightness": 100,
"hotword_detection": 1,
"volume": 100,
"gain": 5
}
Also, I seem to run into the same error where is reboots every few seconds. I've added some extra debug outputs in the log below to try and figure out at what point things to wrong.
/config.json
{
"mqtt_host": "192.168.2.167",
"mqtt_port": 1883,
"mqtt_user": "",
"mqtt_pass": "",
"mqtt_valid": true,
"mute_input": false,
"mute_output": false,
"amp_output": 0,
"brightness": 30,
"hotword_brightness": 100,
"hotword_detection": 1,
"volume": 100,
"gain": 5
}
config file loaded
no audiogroup so xEventGroupCreate
Enter WifiDisconnected
Total heap: 276128
Free heap: 203396
Enter WifiConnected
Connected to Wifi with IP: 192.168.2.137
end of setupEnter MQTTDisconnected
MQTT was not initialized.
past setClientId
192.168.2.167
1883
clientID = atomechoAudio
past asyncclient.connectend of mqttdisconnectedEntry complete
Enter MQTTConnected
Connected as atomecho
Enter Idle
Still Idle
start streaming
past stream
end of idle
Enter MQTTDisconnected
Connect failed, retry
Enter MQTTDisconnected
past setClientIdpast setClientId
192.192168.168..2.167
2.167
1883
1883
clientID = atomechoAudio
clientID = past asyncclient.connectatomechoAudio
past asyncclient.connect
Backtrace: 0x4008c728:0x3ffbe170 0x4008c959:0x3ffbe190 0x400e8aaf:0x3ffbe1b0 0x40084789:0x3ffbe1d0 0x40081a7b:0x3ffdecc0 0x400826bf:0x3ffdece0 0x4008126f:0x3ffded00 0x400ddfe1:0x3ffded20 0x400de05a:0x3ffded60 0x400d2082:0x3ffded90 0x400d20c3:0x3ffdee20 0x400d21cf:0x3ffdee40 0x400d2a50:0x3ffdee60 0x40088ce5:0x3ffdf2e0
Rebooting...
It's almost as if it's running twice.
I have a Matrix Voice ESP32 and everything I attempt to play over mqtt just comes out of hisses and crackling or is extremely quiet.
I have tried with the Rhasspy directly and even the beeps don't come through just a barely audible hiss
One of my responses from Rhasspy is reading from a list and some words are audible but quite and others are just hissing
Then I tried playing audio using python streaming to mqtt however all I got is crackling.
Changing volume or using headphones seem to have no effect.
Hello,
I have downloaded and compiled the latest version.
Cleared the flash and flashed the new version.
Using rhasspy 2.5.11 with internal mqtt.
If I run the matrix voice with "headphones" output there are no problems.
However if I switch to speakers, interesting things start to happen.
Volume is on 100% never touched since flashing.
At the first and sometimes the second activation the volume is loud.
However on every subsequent audio played (I notice this on the wake wav, recored wav and error wav) the volume gradually lowers up to around 20% maybe...
The web interface still shows 100%, if I switch to headphones at this time the volume is low.
If I switch to headphones and restart... no issues for as many activations as I want ... If I switch to speakers same issue starts....
I followed the instructions on the README but I'm stuck trying to run deploy.sh but getting an error. I've checked the steps and I don't know what I'm doing wrong.
When I try to run deploy.sh this is the output
$ ./deploy.sh
cp: cannot stat '../OTABuilder/build/bootloader/bootloader.bin': No such file or directory
Loading firmware: .pio/build/esp32dev/firmware.bin
-----------------------------------
esptool.py wrapper for MATRIX Voice
-----------------------------------
usage: esptool write_flash [-h] [--erase-all]
[--flash_freq {keep,40m,26m,20m,80m}]
[--flash_mode {keep,qio,qout,dio,dout}]
[--flash_size FLASH_SIZE]
[--spi-connection SPI_CONNECTION] [--no-progress]
[--verify] [--encrypt]
[--ignore-flash-encryption-efuse-setting]
[--compress | --no-compress]
<address> <filename> [<address> <filename> ...]
esptool write_flash: error: argument <address> <filename>: [Errno 2] No such file or directory: '/tmp/bootloader.bin'
done
[SUCCESS] Please disconnect your MatrixVoice from the RaspberryPi and reconnect it alone for future OTA updates.
Also tried to to run make on the OTABuilder folder, but it also failed:
$ make
Makefile:6: /make/project.mk: No such file or directory
make: *** No rule to make target '/make/project.mk'. Stop.
What could I be doing wrong?
Hello again,
I have this working quite well now, but have an issue with the audio playback.
issue:-
The audio playback is hissy/crackly and of extremely poor quality. (nothing approaching the 'Alexa demo' video i have seen of yours from a while back).
Sometimes i just get a hiss when something is attempted to be spoken.
It sounds like there is an issue with the volume or the frequency, but i understand your software corrects the formatting/ freq etc.
Mostly i hear the correct response, it just sounds awful.
I have tried:-
I have turned the volume right down and i understand that your software changes the sound to the correct format.
I wondered if it was latency, so i connected my wifi router directly to the 1gb switch which has the rhasppy / Homeassistant pi's.
its exactly the same if i use the 3.5mm jack or the 2 speaker connectors.
I have tried a couple of different speakers and also a headset. all the same.
I have tried a couple of different power sources.
setup:-
I think my setup is pretty much as you have given in examples in the past.
Responses are called to Rhasspy by homeassistant using the rest api http://server.local:12101/api/text-to-speech?siteId={{ site }} which works fine for my pi0 satellites.
I am using Mary TTS / Prudence voice.
I have one pi4 which has Mary TTS and Rhasspy.
Another pi4 with Homeassistant using the integrated MQTT.
I very much appreciate the assistance you have given so far.
Thanks again.
Is it possible to define a fixed volume level in code or otherwise configurable over for example MQTT?
I would like to use a speaker connected to the jack output that has no volume control capabilities, but this would also be useful for speakers connected to the speaker output on the matrix Voice.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.