stdout newline conversion of binary data about openfst

[jtrmal]

I think Kyril is busy but will definitely want the PR :) y.

…

On Mon, Jul 8, 2019 at 3:02 AM David Zurow ***@***.***> wrote: FYI, I fixed this issue, at least for my use cases. The changes are small, but spread out. @kkm000 <https://github.com/kkm000> Do you want a PR, or is it not important enough? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=ACUKYX46U4CVCNSIMYOCW4LP6LRADA5CNFSM4H22MBKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZMFEYI#issuecomment-509104737>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACUKYX27VNIPIBURRSOF2R3P6LRADANCNFSM4H22MBKA> .

[kkm000]

It's supposed to be more portable, although it's not required by the standard and left to the implementation as to which mode can or cannot be changed. But IIRC, it worked with MSVC. I'll check and let you know.

[daanzu]

FYI, I fixed this issue, at least for my use cases. The changes are small, but spread out. @kkm000 Do you want a PR, or is it not important enough?

[brucelin]

FYI, I fixed this issue, at least for my use cases. The changes are small, but spread out. @kkm000 Do you want a PR, or is it not important enough?

I may meet the same problem when I use kaldi under the cygwin enviroment, can u show your solution. Thank you very much. Below is my case

utils/make_lexicon_fst.pl --pron-probs $tmpdir/lexiconp.txt $sil_prob $silphone | \
    fstcompile --isymbols=$dir/phones.txt --osymbols=$dir/words.txt \
    --keep_isymbols=false --keep_osymbols=false | \
     fstarcsort --sort_type=olabel > $dir/L.fst || exit 1;

and the error information

ERROR: GenericRegister::GetEntry: No such file or directory
ERROR: MutableFst::Read: Unknown FST type "vector" (arc type = "standard"): standard input

[daanzu]

Oops, I've been lax in sending this in. I'll try to post it today.

[daanzu]

@brucelin Should be fixed in #23. Also, some binaries with this fix are available if you download and unpack (with zip) the wheels for https://github.com/daanzu/kaldi-active-grammar.

[brucelin]

@daanzu Thank you very much. I will try it.

[kkm000]

Thanks for discovering the issue and the patch, and I'm sorry--was really busy last couple months.

I remember there was a way to deal with the text/binary streams using std::freopen, do you think it would be a better approach if it works?

If filename is a null pointer, then the function attempts to reopen the file that is already associated with stream (it is implementation defined which mode changes are allowed in this case)

So it should be like std::freopen(nullptr, "wb", stdout);. I recall it did the trick.

[daanzu]

Neat, I don't recall knowing about the std::freopen approach. It's been awhile since I wrote the patch, so I can't remember exactly why I chose this approach. I can say that it appears to have been working perfectly for me for the past 7 months, however.

[kkm000]

Blocked-By: #30

[kkm000]

Repro:

$ printf '0 1 10 10\n1\n' | fstcompile | xxd -u -s64

Linux output (ok):

00000040: 0000 0000 807F 0100 0000 0000 0000 0A00
00000050: 0000 0A00 0000 0000 0000 0100 0000 0000
00000060: 0000 0000 0000 0000 0000

Windows output (bug):

00000040: 0000 0000 807F 0100 0000 0000 0000 0D0A
00000050: 0000 000D 0A00 0000 0000 0000 0100 0000
00000060: 0000 0000 0000 0000 0000 0000

[kkm000]

Repro 2. On a Windows machine, save the binary generated by fstcompile:

$  printf '0 1 10 10\n1\n'
0 1 10 10
1
$ printf '0 1 10 10\n1\n' | fstcompile | gzip | base64 -w0                                                             
H4sIAAAAAAAAA7v2d1MdGwMDQ1lqckl+EQeQVVySmJeSWJTCxAABzAyNUVkMCMDEgA4a6hmhLF4uGAECjGgqAUmO7J5sAAAA

Switch to Linux terminal, and do the reverse, unencode, ungzip, fstprint:

$ echo 'H4sIAAAAAAAAA7v2d1MdGwMDQ1lqckl+EQeQVVySmJeSWJTCxAABzAyNUVkMCMDEgA4a6hmhLF4uGAECjGgqAUmO7J5sAAAA' | base64 -d | gunzip | ./fstprint
0       65536   2573    658688
1

The result is nonsense.

[kkm000]

Fixed in #32