acli / trn Goto Github PK

View Code? Open in Web Editor NEW

5.0 4.0 1.0 1.16 MB

trn + UTF-8 = ♥ // Unicode might look like trash on trn 4.0 test77, but if it’s open-source it can be fixed

Home Page: https://github.com/acli/trn/wiki

License: Other

Shell 14.37% C 74.12% Roff 6.09% C++ 0.22% Yacc 1.17% Perl 0.36% Tcl 3.56% Makefile 0.11%

usenet-client terminal-based old-stuff unofficial unicode-support old-school work-in-progress

trn's Introduction

Unicode-patched Threaded Read News (trn) 4.0 test77

This is a version of trn-4.0-test77 that has been patched to display UTF-8 reasonably correctly. However, the original “character set” conversions are currently disabled. Bugs that have nothing to do with UTF-8 support are also being worked on, as this is the newsreader I’m actually using (yes, I’ve tried tin and nn, and no, tin is not easier and nn is not better).

Posting is half-working, with Content-Transfer-Encoding declared as 8bit. A more proper fix is the next step.

Further along, the original conversions need to be put back in, but in a way that wouldn’t corrupt UTF-8.

The original README from 4.0-test77 is in README.

trn's People

Contributors

Stargazers

Watchers

Forkers

eli-the-bearded

trn's Issues

“Connecting” message not preceded by newline in rn mode

The “connecting to” NNTP server message is not preceded by a newline in rn-compatible mode. This is very un-rn-like.

filter_html() (in mime.c) segfaults while reading inline HTML

The line was

bcopy(cp, cp + fudge, t - cp);

Articles sometimes displayed with wrong encoding

Test case: “MrChrisV is the only one marching in step” in news.software.readers

Article detected as Latin1 and displayed as mojibake, but actually tagged as utf-8. “v” causes it to be detected as utf-8, but ^R causes it to be re-misdetected as Latin1

(The post itself is mojibake, but it should still honour content-type; it’s currently not.)

Subjects with MIME continuation lines have spurious spaces inserted

The header decoder is incorrectly treating continuation markers (=\n ) as regular spaces

/usr/sbin/sendmail: No such file or directory

I’m pretty sure I’ve configured it to find sendmail elsewhere. Need to investigate.

Unicode support is broken

trn knows it should decode quoted-printable, but the decoded values are all wrong (mojibake).

Intermittent missed tags

Sometimes trn misses lots of tags, to the point the entire post can come out blank. The cause might be buffer size or tags being cut in half by the buffer boundary, as trn seems to never see the missed tags at all.

Need to support at least a few other character sets such as ISO-8859-15

Otherwise some real posts, e.g. on fr.lettres.langue.francaise, will come up as mojibake

Compressed name not properly terminated

Leading to incorrect output like “School o t A I o CInstitute of” which overflows the “From” column. I probably did something wrong in 447ecd0

Sometimes v followed by ^R causes interpreted and raw headers to be both displayed

Does not seem to affect HTML posts. Need to check if this is my bug or a bug in the original 4.0 test77. xueyan’s plain text posts can be used as test cases.

Names can get “shortened” to mojibake

If a name started with CJK followed by ASCII, the name shortener can corrupt the CJK part

This is happening obviously because it has no notion of UTF-8 or even DBCS.

MIME auto-view doesn't work when reading from local spool

When trn is reading from local spool, MIME articles display as blank. When using an NNTP server, MIME auto-view is the default.

Are all unit tests passing?

I merged your changes to my fork and they applied cleanly to my master (which is trn4.0test77 from sourceforge). Then I rebased my current work in my cmake branch on top of your changes. In the process of integrating your unit tests into the ones that I had added, I am getting the following failures in gtest:

[ RUN      ] TerminateStringAtVisualIndexTest.iso8859_1
C:\Code\trn\trn\tests\trn\test_utf.cpp(405): error: Expected equality of these values:
  m_after
    Which is: "\xC3\xA1\xC3\xAD\xC3\xBA\xC3\xA9\xC3\xB3"
    As Text: "├í├¡├║├⌐├│"
  m_buffer
    Which is: "\xC3\xA1\xC3\xAD\xC3\xBA"
    As Text: "├í├¡├║"
terminate_string_at_visual_index(5)
[  FAILED  ] TerminateStringAtVisualIndexTest.iso8859_1 (11 ms)
[ RUN      ] TerminateStringAtVisualIndexTest.cjk_basic
C:\Code\trn\trn\tests\trn\test_utf.cpp(416): error: Expected equality of these values:
  m_after
    Which is: "\xAF\xA7\xE5\x8C\x96\xE9\xA3\x9B\xE7\x81\xB0"
  m_buffer
    Which is: "\xAF\xA7\xE5\x8C\x96\xE9\xA3\x9B\xE7\x81\xB0\xE4"
terminate_string_at_visual_index(8)
[  FAILED  ] TerminateStringAtVisualIndexTest.cjk_basic (2977 ms)
[ RUN      ] TerminateStringAtVisualIndexTest.cjk_basic_at_wrong_boundary
C:\Code\trn\trn\tests\trn\test_utf.cpp(427): error: Expected equality of these values:
  m_after
    Which is: "\xE5\xAF\xA7\xE5\x8C\x96\xE9\xA3\x9B\xE7\x81\xB0 "
    As Text: "σ»ºσîûΘú¢τü░ "
  m_buffer
    Which is: "\xE5\xAF\xA7\xE5\x8C\x96\xE9\xA3\x9B\xE7\x81\xB0\xE4"
terminate_string_at_visual_index(9)
[  FAILED  ] TerminateStringAtVisualIndexTest.cjk_basic_at_wrong_boundary (3 ms)

My assumption is that I missed something when I was rebasing my changes on top of yours, but I wanted to check to see if you get a clean test run on your tree. I am developing on Windows, so I don't have ready access to a unix machine right now. I'm going to see what I can do under WSL, but that's a whole other journey....