New optimized write performance seems to break glamor about reflow

[kiyonlin]

Copy from #10 (comment)

Hey @zavislak I don't know why you use bad = append(bad, []byte(string(c))[0])

You can see this demo instead. https://play.golang.org/p/fHX2TW2-ORG

[kiyonlin]

Do you have more detail examples?

[zavislak]

This should illustrate the problem better: https://play.golang.org/p/RLOMbGDu3RV

If you convert a single byte of the a UTF-8 sequence to a string, go will mangle it. This is in effect what happens when the new ansi.Writer outputs the sequences one byte at a time, when downstream code tries to reinterpret these single bytes as strings (as padding.go does).

I found this by building glow at head. Let me know if glow properly shows UTF-8 sequences for you.

[kiyonlin]

I believe you can build glow at head and see the broken UTF-8 sequences, which led me to try track this down to happening between the indent and padding pipes in glamor. If glow properly handles utf-8 for you at head, please let me know.

We can discuss here. One thing you should notice is all changes are not released yet.

[zavislak]

in that case, another way to see this is to modify glamour's helloworld example to include a UTF-8 multi-character sequence.

[kiyonlin]

If you convert a single byte of the a UTF-8 sequence to a string, go will mangle it.

I don't think this gonna happen. ansi.Writer is just a wrapper which doesn't care about the original non-ansi bytes.

[kiyonlin]

You can view the test case.

Btw it is best if there is code that can reproduce the 'bug'.

[zavislak]

The problem is not ansi.Writer in isolation, but that this change now causes single bytes to be written instead of full UTF-8 runes. Downstream consumers like padding.go's Write method cast these single byte writes into strings, misinterpreting them.

You can just modify https://github.com/charmbracelet/glamour/blob/master/examples/helloworld/main.go to include a UTF-8 character sequence, which will reproduce the problem.

[kiyonlin]

Once upgrade to master version, I saw the broken string. Will do some investigations.

Thanks for reporting!

[zavislak]

Cheers! Was going to just say go get github.com/muesli/reflow@aa57fa9 first.

[kiyonlin]

@zavislak Fixed by iterating in string. Wait @muesli to handle the PR.

Or you can get the latest commit from my repo.

Any feedbacks are welcome!