Remove byte-order markers from CSV files? about d3-dsv

[mbostock]

Can you attach an example file I can use for testing purposes?

[robinhouston]

Sure! GitHub won’t let me attach a .csv file, so I’ve zipped it.
Workbook1.csv.zip

[mbostock]

As TXT: Workbook1.txt

[mbostock]

Interestingly if you use FileReader.readAsText, it automatically strips the BOM bytes for you, per the Encoding specification.

[mbostock]

Seems like XMLHttpRequest and Fetch also automatically strip the BOM. Here’s a CORS-accessible URL I tested:

https://rawgit.com/mbostock/3fe6055309cff87cba4103837d914fee/raw/48cec3b15411fe2a9d9f678c5988d03b3988f498/test.csv

So my question is how are you getting a string with the BOM still in it? It seems like the BOM stripping should happen earlier, before it gets to d3-dsv.

[robinhouston]

Sorry, I should have included a complete repro. I’m getting this in node, by fs.readFile(filename, "utf8", …). It looks as though the node developers have decided against stripping BOMs by default.

It’s okay if I should handle this in the app: I just thought I should flag it.

[mbostock]

Okay. I’m going to close this issue. If you want to submit a pull request with an edit to the README suggesting that Node users use strip-bom that would be 💯 .

[robinhouston]

Great, will do!

[aubergene]

Gah this just got me too. Could we consider adding it directly to d3-dsv? I think the code is considerably shorter than the comment in the README, plus I wasted a good ten minutes, thanks Excel!