ERROR: Unknown file read mode when reading VectorFst about openfst

[SamLam0206]

I also tried to make the HCLG.fst on Windows. The script ran smooth and HCLGa.fst could be composed by OpenFst until some Kaldi's internal FST utility was run:

'C:\Users\kaldi\kaldiwin_vs2017_OPENBLAS\x64\Debug\add-self-loops.exe' --self-loop-scale=1.0 --reorder=true tdnn_7q_mxr_1/final.mdl tdnn_7q_mxr_1/graph_ctf/HCLGa.fst
ERROR: VectorFst::Read: Read failed: standard input
ERROR: VectorFst::Write: Write failed: standard output
ERROR (add-self-loops.exe[5.5-win]:main():c:\kaldi\src\bin\add-self-loops.cc:109) add-self-loops: error writing FST to standard output

ERROR (add-self-loops.exe[5.5-win]:main():c:\kaldi\src\bin\add-self-loops.cc:109) add-self-loops: error writing FST to standard output

I guess there are some mismatch of Kaldi's VectorFst type or IO process which I still not figure out.

[kkm000]

Thanks. This is interesting. Can you please come up with a small piece of code (or a binary command line) that reproduces the problem? On 10 Dec 2018 20:23, SamLam0206 <[email protected]> wrote: I also tried to make the HCLG.fst on Windows. The script ran smooth and HCLGa.fst could be composed by OpenFst until some Kaldi's internal FST utility was run: 'C:\Users\kaldi\kaldiwin_vs2017_OPENBLAS\x64\Debug\add-self-loops.exe' --self-loop-scale=1.0 --reorder=true tdnn_7q_mxr_1/final.mdl tdnn_7q_mxr_1/graph_ctf/HCLGa.fst ERROR: VectorFst::Read: Read failed: standard input ERROR: VectorFst::Write: Write failed: standard output ERROR (add-self-loops.exe[5.5-win]:main():c:\kaldi\src\bin\add-self-loops.cc:109) add-self-loops: error writing FST to standard output ERROR (add-self-loops.exe[5.5-win]:main():c:\kaldi\src\bin\add-self-loops.cc:109) add-self-loops: error writing FST to standard output I guess there are some mismatch of Kaldi's VectorFst type or IO process which I still not figure out. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH2Nhd34L5JvTbBvyn8KqDSGr0lFAVXdks5u3zMjgaJpZM4ZK_uv>.

[SamLam0206]

Sorry I couldn't find a simple enough way to reproduce the problem and I also suspect that most of the problem lies in Kaldi's window build.

But I've discovered some problem when reading VectorFst.
When reading the header, it crashed when retrieving the arc type (which in this line I think it should be hdr->ArcType()?).

When I commented out those involving Arc::Type(), the header could be read. But it crashed again when adding state. Maybe that my code missed something in using the Fst template?

[kkm000]

It really looks like something, somewhere (sorry, it's hard to be more specific) is writing binary files in text mode. There is no difference on Linux, but on Windows this may turn the \n (a binary byte with the hex value 0A) into the sequence \r\n, 0C 0A. Since you are seeing the problem in header, find a port of hexdump (cygwin has one; there is an independent Windows exe) or just open the file in VS (dropdown on the Open button. Open with, Binary editor) if the FST file is not extra large, and look for this sequence in header.

If you are passing an istream to Kaldi or FST functions, do not forget ios::binary. I am pretty sure that all Kaldi functions that create streams internally do that, but if passing streams to them, be aware of this, ehm, feature! From experience, I've never had a problem exchanging FST files between Windows and Linux. This is not to say there is not a possible bug--of course it's always a possibility; just to let you know this generally works flawlessly.

[SamLam0206]

I checked and it seems that the header does not contain 0C 0A. It turns out that the header could be read.

The problem is that when I read the VectorFst using VectorFst<StdArc> *fst = VectorFst<StdArc>::Read(ki.Stream(), ropts);, there is some problem with the implementation constructor at
std::unique_ptr<VectorFstImpl<S>> impl(new VectorFstImpl());
The StdArc class or the VectorState constructed from it somehow could not be passed to the template, such that when retrieving Arc::Type() or Weight::Type() of the new impl it would crash.

As the new impl object is independent of the content of my fst, I think there maybe other issues. I'm still finding ways to work around this. Thanks if you could offer any suggestions.

[SamLam0206]

I found that it's my own mistakes again. Thanks for all your previous advice. This windows port of OpenFst is magnificent and crucial for my work. Many thanks again!