Speech-to-text, text-to-speech, and speaker recongition using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift
Home Page: https://k2-fsa.github.io/sherpa/onnx/index.html
License: Apache License 2.0
k2-fsa/icefall#1108 adds support to export the latest zipformer to ONNX (both streaming and non-streaming are supported).
We need to support the exported models in sherpa-onnx.
We also need to update the doc:
Help to wrap the C++ implementation of online websocket server to Python is appreciated.
HI. I have using this script export wenet model:
python export_onnx.py --bpe_model weights/icefall_asr_wenetspeech_pruned_transducer_stateless2/data/lang_char/ --pretrained_model weights/icefall_asr_wenetspeech_pruned_transducer_stateless2/exp/pretrained_epoch_10_avg_2.pt
this weighs using pytorch inference result OK.
but when using it inference on onnx, I get result:
wav: ../../../data/test_data/16k16bit.wav search: greedy
loading ../../../weights/onnx/encoder.onnx
loading ../../../weights/onnx/decoder.onnx
loading ../../../weights/onnx/joiner.onnx
loading ../../../weights/onnx/joiner_encoder_proj.onnx
loading ../../../weights/onnx/joiner_decoder_proj.onnx
../../../data/test_data/16k16bit.wav
reading ../../../data/test_data/16k16bit.wav
wav shape: (1,112400)
Elapsed: 0.332 seconds
rtf: 0.0474286
Hyps:
3938-2261-519-3938-376-657-657-4250-4745-637-1449-1449-59-3978-376-249-5294-2480-1449-657-3786-519-249-519-249-519-249-519-249-519-249-1449-2480-1872-657-519-1713-3201-4186-3938-3938-4745-519-3938-376-519-249-3669-3938-4877-5525-3938-4745-519-3938-376-657-657-519-1713-3938-2261-1449-59-3978-376-249-5294-4745-1449-1449-3655-1449-3710-1449-1449-59-3978-376-249-5294-2480-1449-657-3786-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-1449-2480-3281-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-519-249-1449-2480-637-1449-1449-3978-858-1852-1574-249-2368-2480-1923-1449-2169-361-2480-1449-657-3786-519-249-519-249-519-249-519-249-1449-2480-1501-657-3786-3669-3938-4877-657-3938-4745-519-3938-376-|
诋犹估诋亲谅谅惬颔扑殊殊就孵亲务邗竖殊谅椎估务估务估务估务估务殊竖瞭谅估顿尬蜕诋诋颔估诋亲估务殓诋蹰蚡诋颔估诋亲谅谅估顿诋犹殊就孵亲务邗颔殊殊挎殊跺殊殊就孵亲务邗竖殊谅椎估务估务估务估务估务估务估务估务估务估务估务估务殊竖撇估务估务估务估务估务估务估务估务估务估务殊竖扑殊殊孵螃园径务咽竖脖殊液立竖殊谅椎估务估务估务估务殊竖迈谅椎殓诋蹰谅诋颔估诋亲
I checked the tokens seems normal.
What's could be missed?
您的项目我已经尝试，是非常方便部署和运行的ASR优秀模型！我是一名研究生，希望学习向您了解，写出这样优质的项目（或者说至少努力接近的程度）的基本知识有哪些？ 我已经具有python的深度学习经验。 关于C++、onnx，我还需要学习哪些知识呢？
感谢您的回答！
Hello,
Are there any pre-trained models like icefall-asr-multidataset-pruned_transducer_stateless7-2023-05-04 (Englis) which are trained on GigaSpeech + LibriSpeech + Common Voice 13.0 for Streaming ASR
Concurrently, which is the best pre-trained model which you have for Streaming ASR
I changed GetSessionOptionsImpl() in session.cc to use OpenVINO.
Ort::SessionOptions sess_opts;
sess_opts.SetIntraOpNumThreads(num_threads);
sess_opts.SetInterOpNumThreads(num_threads);
sess_opts.SetGraphOptimizationLevel(ORT_DISABLE_ALL);
OrtOpenVINOProviderOptions options;
options.device_type = "CPU_FP32";
sess_opts.AppendExecutionProvider_OpenVINO(options);
return sess_opts;After that, I ran into errors like
2023-05-16 15:37:00.162490913 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may o
r may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-16 15:37:00.162554167 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignmen
ts.
2023-05-16 15:37:00.573335241 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may o
r may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-16 15:37:00.573379616 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignmen
ts.
terminate called after throwing an instance of 'Ort::Exception'
what(): Encountered unknown exception in Run()
I looked at the OnlineZipformerTransducerModel class:
Ort::SessionOptions sess_opts_;
Ort::AllocatorWithDefaultOptions allocator_;While sess_opts_ is initialized using sess_opts_(GetSessionOptions(config)), allocator_ is probably not using the updated session options. That may be the root cause of the above issue.
I appreciate your help on this issue. Thanks a lot!
it is nice work,i real like it.
but, when i run this project,i have some questions.
i try this web 1 and 3 method to install this project.
if i install this with method 1
it work nice ,the words can recognize and can split when the talk have some time no voice.
but when i install with method 3, the words can recognize but can not split when the talk have long time no voice.it will output one line
can you tell me how to solve it.
Thank you
See https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html
In k2-sherpa, during warm-up, the encoder is initialized to the model initial states.
void WarmUp(torch::Tensor features, torch::Tensor features_length) {
torch::IValue states = GetEncoderInitStates();
states = StackStates({states});
However, I don't see similar warm-up process with sherpa-onnx. This may cause a bit worse performance at the beginning of a stream. It may be tricky to save the model initial states during ONNX export I guess.
Appreciate any suggestions.
(base) C:\Users\K2\sherpa-onnx\build>cmake -DCMAKE_BUILD_TYPE=Release ..
-- Building for: NMake Makefiles
CMake Error at CMakeLists.txt:2 (project):
Running
'nmake' '-?'
failed with:
The system cannot find the file specified
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "C:/Users/K2/sherpa-onnx/build/CMakeFiles/CMakeOutput.log".
The following file
https://github.com/k2-fsa/sherpa-onnx/blob/master/python-api-examples/decode-file.py
uses a fixed set of parameters. It is nice to change it to accept arguments from the commandline.
I've seen that for Icefall, the 2 ways to export models are using either ONNX (this package) or NCNN.
Has there been any benchmarking done for the 2 methods? I'm wondering which one would be faster.
I did find that there's this page k2-fsa/sherpa-ncnn#44 which includes some NCNN run times.
We have wrapped OnlineRecognizer to Python,
It is nice if someone could help wrap OfflineRecognizer to Python.
The offline flow has no warning. However, when I ran the streaming flow, I got the following warnings:
2023-05-25 00:21:23.008301845 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-25 00:21:23.008326867 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
I turned on ONNX verbose output and found these messages:
2023-05-25 00:21:20.004017871 [V:onnxruntime:, session_state.cc:1116 VerifyEachNodeIsAssignedToAnEp] Node placements
2023-05-25 00:21:20.004031728 [V:onnxruntime:, session_state.cc:1122 VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [CPUExecutionProvider]. Number of nodes: 875
......
2023-05-25 00:21:20.006623532 [V:onnxruntime:, session_state.cc:1122 VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [CUDAExecutionProvider]. Number of nodes: 2426
......
So, the warnings are correct as some nodes are assigned to CPU while the others are assigned to GPU. ONNX runtime gave the following explanation:
2023-05-25 00:21:19.287699117 [I:onnxruntime:, fallback_cpu_capability.cc:157 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: Unsqueeze_5018 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
I wonder if we need to do anything to these warnings and appreciate your feedback. Thanks.
I checked out the latest master branch and ran ./scripts/check_style_cpplint.sh 2. I got the following errors:
......
/workspace/sherpa-onnx/sherpa-onnx/csharp-api/offline-api.h:122: At least two spaces is best between code and comments [whitespace/comments] [2]
/workspace/sherpa-onnx/sherpa-onnx/csharp-api/offline-api.h:122: Could not find a newline character at the end of the file. [whitespace/ending_newline] [5]
Done processing /Users/jou2019/workspace/sherpa-onnx/sherpa-onnx/csharp-api/offline-api.h
Total errors found: 92
[FAILED] /workspace/sherpa-onnx/sherpa-onnx/csharp-api/offline-api.h
Follow
to support arm32 and aarch64.
Hi, when will Chinese onnx support and when will all-in-one inference will be ready?
Really hoping for a workable Chinese ASR ONNX support here.
In Sherpa, features are extracted with Kaldifeat and it is dependent on Torch. How can I calculate fbanks without Kaldifeat? Should I use kaldi scripts? To give fbanks to onnx model, I should obtain inputs as std::vector<float> like
size_t feature_size = 1 * 600 * 80; // TODO: dynamic shape
std::vector<float> features(feature_size);
std::vector<int64_t> feature_dims = {1, 600, 80};
size_t feature_length_size = 1;
std::vector<int64_t> features_lengths = {600}; // TODO: dynamic shape
std::vector<int64_t> feature_length_dims = {1};
std::vector<const char*> output_node_names = {"encoder_out"};
// create input tensor object from data values
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value feature_tensor = Ort::Value::CreateTensor<float>(memory_info, features.data(), feature_size, feature_dims.data(), 3);
Ort::Value feature_tensor_length = Ort::Value::CreateTensor<int64_t>(memory_info, features_lengths.data(), feature_length_size, feature_length_dims.data(), 1);
Help to wrap the C++ implementation of offline websocket server to Python is appreciated.
Following : k2-fsa/icefall#1119
See https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html
and
https://learn.microsoft.com/en-us/windows/ai/directml/dml-intro
and
https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.DirectML
It would be nice to also support DirectML
From the export, I didn't find how to export the projection onnx as well?
I tried export lstm-transducer, fails.
Hi @csukuangfj,
Would you mind porting your onnx related c++ code to https://github.com/k2-fsa/sherpa-onnx
I haven't written c++ code yet, my fork includes search functions in Python. c++ code will be done probably in a few days.
See
sherpa-onnx/sherpa-onnx/csrc/offline-recognizer-transducer-impl.h
Lines 56 to 62 in 6707ec4
Hi, I tried export onnx by myself, but the result are wrong (Chinese version):
./build/bin/sherpa-onnx ../../weights/onnx/tokens.txt ../../weights/onnx/encoder.onnx ../../weights/onnx/decoder.onnx ../../weights/onnx/joiner.onnx ../../weights/onnx/joiner_encoder_proj.onnx ../../weights/onnx/joiner_decoder_proj.onnx ../../data/test_data/16k16bit.wav
wav filename: ../../data/test_data/16k16bit.wav
wav duration (s): 7.025
Recognition result for ../../data/test_data/16k16bit.wav
珲珲珲仪韭韭熏呸鞘熏足绽Z韭珲猢仪梆鸥柜懈爷绽懈珲懈爷铍梆韭珲爷殡韭熏呸爷冥寓桐积珲懈爷铍岑霜坷熏鸥严肱苜珲懈爷铍岑霜坷熏鸥严芦斐涿苜珲懈爷铍岑霜坷熏鸥严肱苜珲懈爷绽懈懈恰韭熏呸览珲缪流苜鸳捆执绽圄懈恰韭熏呸鞘螺熏桐积珲懈爷铍岑霜懈熏恰韭熏呸苜熏桐积珲懈爷铍岑霜坷熏武绽韭韭熏呸鞘熏桐积韭珲猢仪梆乒柜苜苜蜕狰诽铉熏岑苜蜕坷桐懈狩殃武绽韭韭熏呸鞘柜坷蜕
this is how I export:
python export_onnx.py --pretrained_model weights/icefall_asr_wenetspeech_pruned_transducer_stateless2/exp/pretrained_epoch_10_avg_2.pt --bpe_model weights/icefall_asr_wenetspeech_pruned_transducer_stateless2/data/lang_char/
what's the bias here caused unable to get right result?
Hi,
I am trying to use Kaldi Native IO to read wav, but I got this error in knf::OnlineFbank fbank(opts);
start
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
ı wrote a basic main.cpp:
#include <iostream>
#include "kaldi-native-fbank/csrc/online-feature.h"
int main(int argc, char* argv[]) {
std::cout << "start" << std::endl;
knf::FbankOptions opts;
opts.frame_opts.dither = 0;
opts.frame_opts.frame_shift_ms = 10.0f;
opts.frame_opts.frame_length_ms = 25.0f;
opts.mel_opts.num_bins = 80;
knf::OnlineFbank fbank(opts);
}
It works fine.
I cloned and build from source. After that, I added Cmake file (which is taken from Sherpa) to sherpa-onnx/cmake
if(DEFINED ENV{KALDI_NATIVE_IO_INSTALL_PREFIX})
message(STATUS "Using environment variable KALDI_NATIVE_IO_INSTALL_PREFIX: $ENV{KALDI_NATIVE_IO_INSTALL_PREFIX}")
set(KALDI_NATIVE_IO_CMAKE_PREFIX_PATH $ENV{KALDI_NATIVE_IO_INSTALL_PREFIX})
else()
# PYTHON_EXECUTABLE is set by cmake/pybind11.cmake
message(STATUS "Python executable: ${PYTHON_EXECUTABLE}")
execute_process(
COMMAND "${PYTHON_EXECUTABLE}" -c "import kaldi_native_io; print(kaldi_native_io.cmake_prefix_path)"
OUTPUT_STRIP_TRAILING_WHITESPACE
OUTPUT_VARIABLE KALDI_NATIVE_IO_CMAKE_PREFIX_PATH
)
endif()
message(STATUS "KALDI_NATIVE_IO_CMAKE_PREFIX_PATH: ${KALDI_NATIVE_IO_CMAKE_PREFIX_PATH}")
list(APPEND CMAKE_PREFIX_PATH "${KALDI_NATIVE_IO_CMAKE_PREFIX_PATH}")
find_package(kaldi_native_io REQUIRED)
message(STATUS "KALDI_NATIVE_IO_FOUND: ${KALDI_NATIVE_IO_FOUND}")
message(STATUS "KALDI_NATIVE_IO_VERSION: ${KALDI_NATIVE_IO_VERSION}")
message(STATUS "KALDI_NATIVE_IO_INCLUDE_DIRS: ${KALDI_NATIVE_IO_INCLUDE_DIRS}")
message(STATUS "KALDI_NATIVE_IO_CXX_FLAGS: ${KALDI_NATIVE_IO_CXX_FLAGS}")
message(STATUS "KALDI_NATIVE_IO_LIBRARIES: ${KALDI_NATIVE_IO_LIBRARIES}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${KALDI_NATIVE_IO_CXX_FLAGS}")
message(STATUS "CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
Then
include(cmake/kaldi_native_io.cmake) is added to sherpa-onnx/CMakeLists.txt.target_link_libraries(capi_test onnxruntime kaldi-native-fbank-core kaldi_native_io_core) is added to sherpa-onnx/sherpa-onnx/csrc/CMakeLists.txt.After these steps, I can include files (#include "kaldi_native_io/csrc/kaldi-io.h", #include "kaldi_native_io/csrc/wave-reader.h"). I can also read a wav file. However, knf::OnlineFbank fbank(opts); line gives 'std::bad_alloc'.
How can I solve this? I think the issue is in cmake file, but I cannot find the reason.
When attempting to build four ios using Xcode 14.2 I receive the following error:
fatal error: 'coreml_provider_factory.h' file not found
There are no tests covering files in
https://github.com/k2-fsa/sherpa-onnx/tree/master/sherpa-onnx/c-api
Bugs like the following one
sherpa-onnx/sherpa-onnx/c-api/c-api.cc
Lines 97 to 99 in cd3c58c
It is nice if someone can help integrate https://github.com/google/sanitizers/wiki/AddressSanitizer into shepra-onnx.
It should help detect memory leak issues.
cmake/onnxruntime.cmake
is getting larger as we are adding support for more onnxruntime execution providers.
We need to split cmake/onnxruntime.cmake
into smaller files, where each file handles only one operating system.
Streaming vesion of conformer?
目前，sherpa-onnx ,websocket server 还没有支持endpoint，timestamp，tokens。希望能够支持
when i run python example in windows x64 with vscode
the terminal output is not nornal work
i think sherpa_onnx.Display have some bug
i add the output with print
'
print("print:： "+ last_result)
display.print(-1, result)
'
it is normal work
the python code is download from
the package is from pip install sherpa-onnx
print:： 你
浣
print:： 你好
浣犲ソ
print:： 你好啊
浣犲ソ鍟
print:： 你好啊今天天
浣犲ソ鍟婁粖澶╁ぉ
print:： 你好啊今天天气
浣犲ソ鍟婁粖澶╁ぉ姘
print:： 你好啊今天天气真不错
浣犲ソ鍟婁粖澶╁ぉ姘旂湡涓嶉敊
print:： 你好啊今天天气真不错 HELLO
浣犲ソ鍟婁粖澶╁ぉ姘旂湡涓嶉敊 HELLO
print:： 你好啊今天天气真不错 HELLO WHATS YOUR
浣犲ソ鍟婁粖澶╁ぉ姘旂湡涓嶉敊 HELLO WHATS YOUR
print:： 你好啊今天天气真不错 HELLO WHATS YOUR NAME
浣犲ソ鍟婁粖澶╁ぉ姘旂湡涓嶉敊 HELLO WHATS YOUR NAME
Hello,w hen will stream Chinese support/?
#151 adds CoreML support for macOS. We also need to support iOS.
The pre-built libs can be downloaded from
https://onnxruntimepackages.z14.web.core.windows.net/pod-archive-onnxruntime-c-1.14.0.zip
After unzipping, you will find the following files:
.
├── Headers
│ ├── coreml_provider_factory.h
│ ├── cpu_provider_factory.h
│ ├── onnxruntime_c_api.h
│ ├── onnxruntime_cxx_api.h
│ └── onnxruntime_cxx_inline.h
├── LICENSE
├── a.txt
└── onnxruntime.xcframework
├── Info.plist
├── ios-arm64
│ └── onnxruntime.framework
│ ├── Headers
│ │ ├── coreml_provider_factory.h
│ │ ├── cpu_provider_factory.h
│ │ ├── onnxruntime_c_api.h
│ │ ├── onnxruntime_cxx_api.h
│ │ └── onnxruntime_cxx_inline.h
│ ├── Info.plist
│ └── onnxruntime
└── ios-arm64_x86_64-simulator
└── onnxruntime.framework
├── Headers
│ ├── coreml_provider_factory.h
│ ├── cpu_provider_factory.h
│ ├── onnxruntime_c_api.h
│ ├── onnxruntime_cxx_api.h
│ └── onnxruntime_cxx_inline.h
├── Info.plist
└── onnxruntime
8 directories, 22 files
provider, which is added by #151Look like that sherpa-onnx only supports LSTM and ZipFormer models. I wonder if anyone has code to support conformer models. Thanks!
When people use pip install sherpa-onnx to install sherpa-onnx from pre-compiled wheels, we should also
install all binaries from sherpa-onnx, such as the following binaries in the screenshot.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.