How are characters like tab and new line handled? about tsv-utils

[jondegenhardt]

By default, TAB and newline characters are replaced a space. A different replacement string can be specified using the --r|replacement option.

These details can be found in the csv2tsv reference, or by using the --help-verbose option.

[dclong]

But doesn't this change the original data? Tabs and newlines might special means in the original CSV files.

Generally speaking, are there field delimiters and line delimiters that are safe to use in most situations? That is people seldom use them in text and no escape is required for them.

[jondegenhardt]

TSV file format uses TABs and newlines as delimiters precisely because these characters are not part of the data in the vast majority of large data sets. It's certainly true of most data mining scenarios. The reason csv2tsv defaults to spaces as the replacement character is that it tends to be a reasonable choice when working with NLP data sets.

Of course, TAB and newline may carry meaning in some data sets. Most software tools support alternate field delimiters, so something other than TAB can be chosen. Very few tools support alternate record (line) delimiters. If newlines need to be preserved in the data set being used, then it's better to use a file format that supports full character escaping.

Using TSV is a tradeoff. The lack of escapes results in much simpler and less error prone processing, and can run faster as well. However, it is not the best format for every use case.

To your more general question: ASCII does define control characters intended for these purposes, CTRL-~ and CTRL-_ (decimal 30 and 31, Hex 1E and 1F) for record separator and field separator. However, these never got much support. However, you use one of these in place of TAB for field delimiter.

There is related discussion in the Comparing TSV and CSV formats page in this repository.

The wikipedia article on Tab-separated values has more info on TSV file format.

Update: In Release 2.1.0 csv2tsv supports options to specify separate replacement strings for TABs and newlines found in CSV data fields. This allows selection of characters that can used to preserve the meaning of TAB and newline characters. There are several unicode characters that can be used for this purpose, including U+2028 (Line Separator), U+2029 (Paragraph Separator), U+2424 (␤, Visual Newline), U+2409 (␉, Visual TAB). Whether this is approach is applicable depends on the specific task. See PR #303 for details.

[jondegenhardt]

Been 13 days, assuming the answer is sufficient and closing. Please add addition comments if further discussion would benefit.