ben-strasser / fast-cpp-csv-parser Goto Github PK
View Code? Open in Web Editor NEWfast-cpp-csv-parser
License: BSD 3-Clause "New" or "Revised" License
fast-cpp-csv-parser
License: BSD 3-Clause "New" or "Revised" License
This is a nice little library, but "header-only" should not mean "one header only", and 1200 lines, while not terrible, is still a lot for just a CSV parser. There are clear cutoff points for separating out several header files; certainly LineReader
and CSVReader
can have files of their own.
... and then you tie it all together with a csv.h
that includes the different header files.
Rather than
io::CSVReader<3> in("ram.csv");
change it to
io::CSVReader<> in(3, "ram.csv");
I did the change already, it work as expected.
could you change the code so that you allow CSVReader to be defined with the number of rows specified by a variable rather than a constant, and then allow read_header to use a vector array of strings for the header names?
When the LineReader
destructor is called, it frees the data buffer. However, there can be an in-flight read call in the AsynchronousReader
worker thread, which will try to put stuff into the freed data buffer. This leads to a segfault. To reproduce, read a large file and throw an exception while it is reading.
The solution is simple: don't deallocate the buffer until the reader is joined. I have a PR I can send over, which basically replaces the char* buffer
with a std::vector<char> buffer
and puts that before the AsynchronousReader reader
in the class members so that it gets destructed after the reader is destroyed.
I have a line with, say 15 columns, but I'm interested only in the first 5.
However, I don't have a header so I use set_header()
instead of read_header()
. But set_header()
does not have an ignore_policy.
How can I parse just the first 5 columns, without a bunch of dummies?
I've tried compiling the example code from the readme using CMake but I keep getting the following error:
libc++abi.dylib: terminating with uncaught exception of type io::error::can_not_open_file: Can not open file "ram.csv" because "No such file or directory".
I've place a dummy CSV file named ram.csv
in the same path as main.cpp
and the included csv.h
. The file also contains the appropriate columns and dummy data to match the example. My CMakeLists.txt is simply:
cmake_minimum_required(VERSION 3.3)
project(csv_example)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
set(SOURCE_FILES main.cpp)
add_executable(csv_example ${SOURCE_FILES})
Any ideas on what I'm doing wrong? Thanks.
I ran into an issue attempting to parse the CSV file here: https://github.com/donohoe/nyt-crossword. The file has CR line endings, but this parser only supports LF and CR-LF. I had to run mac2unix
on the file to get it to parse. It would be nice to have extra line ending support, or have it be configurable.
Thanks for this library and it works great under macOS/LLVM. I do have a need to run this library under Win32 and for reasons I won't go into here, I'm using a free C++11 compiler from Embarcadero (Borland C++ in my youth). bcc32c reports the following errors:
include\fast-cpp-csv-parser/csv.h:236:52: error: use of overloaded operator '!=' is ambiguous
(with operand types 'const std::unique_ptr<ByteSourceBase>' and 'int')
return byte_source != 0;
include\fast-cpp-csv-parser/csv.h:262:48: error: use of overloaded operator '!=' is ambiguous
(with operand types 'std::unique_ptr<ByteSourceBase>' and 'int')
if(byte_source != 0){
include\fast-cpp-csv-parser/csv.h:296:52: error: use of overloaded operator '!=' is ambiguous
(with operand types 'const std::unique_ptr<ByteSourceBase>' and 'int')
return byte_source != 0;
Any thoughts or insights?
How can i compíle? I am using VC++ 2008
when I use in.read_row(x,y,z,a,b,c),compiler said that "error C2338: too many columns specified"
Example:
struct extra_column_in_header :
base,
with_file_name,
with_column_name{
void format_error_message()const{
std::snprintf(error_message_buffer, sizeof(error_message_buffer),
"Extra column \"%s\" in header of file \"%s\"."
, column_name, file_name);
}
};
struct missing_column_in_header :
base,
with_file_name,
with_column_name{
void format_error_message()const{
std::snprintf(error_message_buffer, sizeof(error_message_buffer),
"Missing column \"%s\" in header of file \"%s\"."
, column_name, file_name);
}
};
Essentially the same code but a different string. Considering how exceptions are, well, exceptional, I think it should not be impossible to get to something like:
using extra_column_in_header = exception<file_name, column_name>;
const char* exception_message_traits<extra_column_in_header>::format_string =
"Extra column \"%s\" in header of file \"%s\".";
using missing_column_in_header = exception<file_name, column_name>;
const char* exception_message_traits<missing_column_in_header >::format_string =
"Missing column \"%s\" in header of file \"%s\".";
or whatever.
is not supported when compiling with /clr or /clr:pure
Your code works well while creating normal exe file. But I want to create a setup file for my project so I added Visual Studio Installer --> Setup Project. Upon trying to build a setup file it came up with these warnings.
You may refer my cpp program here.
I get a "too many columns" error when trying to parse a string with comma in it. However, I can't escape this string in the CSV because it is a valid WKT string (and read by other tools reading this CSV file).
My CSV looks like this:
id, wkt
0,"POLYGON ((1 0,-1 0,1 0))"
When a cell in .csv file that has many lines, It throws exception "Too few columns in line ... ".
I can not read it as char *, please teach me how to solve it.
Thanks
Hi Ben,
This is not really an issue, just a question. I've read your code (async part) and I noticed that you used 3 buffers: every time you complete the first buffer you copy the second to the beginning, wait for the last third to complete and then you copy it as well . What's the reason for this design?
Why not using just double-buffer technique: parse from the first part while asynchronously read into the second half, wait for it to complete and then to parse the second part, asynchronously reading from the first part. It seems simpler so I am guessing there is a catch.
https://github.com/ben-strasser/fast-cpp-csv-parser/blob/master/csv.h#L900
This is a bitwise and instead of a logical and. This causes parsing to be incorrect for some cases, for example when ignore_policy = 2 and ignore_missing_column = 3. The same thing is true for line 909.
To fix it, replace the & with an &&.
a,b,c
1,"23,3",44
23,545,56
I get libc++abi.dylib: terminating with uncaught exception of type io::error::too_many_columns: Too many columns in line 2 in file "/Users/talha/Coding/ClionProjects/pricerightnlp/test.csv".
I want to use this library on embedded device which has limited processing power and memory.
The size of CSV file to be imported is around 200KB - 500KB.
Regards
What is the best way to write the code to handle the situation where you get a file with the correct format but the header line is missing?
We get the same data from multiple sources and some sources provide a header and some do not.
very nice. but I needed to read only some lines of a file, and only some parameters from these lines. this is why I decided unwillingly to do something worse myself... (https://github.com/pgallir/easy-cpp-subfile-parser).
by the way, this could be an useful features for many. as far as I am concerned, you should introduce this feature in your lib.
Hi,
when trying to pass in a istream& in here, clang barfs because it tries to call a copy ctor.
The proper fix is to make the argument take a universal ref...
explicit CSVReader(Args&&...args):in(std::forward<Args>(args)...){
Cheers
Jiri
Say I have a CSV file with 30 columns, an unreliable header (but order is guaranteed), and I only care about 4 columns. So, I've got to call set_header with 30 names for the columns. Then in my read call I need to pass 30 variables. Is that correct? Seems cumbersome, even if I needed all 30 columns. It would be nice to have an option where the row elements are returned in std::vector and then I do the data type conversions myself for the columns I need.
I know it's probably a bad design but I'm trying to find a way to reuse an existing CSVReader instance for reading another file so that I can treat a lot of csv files as a single merged file, any ideas? (I can't iterate on a list of files either as there's no offset setting on the library to return to last read row when I have to stop the iteration midway)
Hi,
I was wondering if you had any test code/benchmarking code ?
I add some CMake logic to easily add bunches of gtest tests...
I would like to add an istream interface to LineReader, but before starting any coding,
it would nice if there are some tests already I could add.
Cheers
Jiri
Hi, firstly thanks for writing this CSV library!
I was wondering if it is possible to read a CSV file whose structure is only known at run-time. In particular, I have a vector of column headers (std::vector<std::string>
) to read from the CSV which I would ideally like to pass to the parser directly – instead of a series of separate parameters which must be known at compile-time. The size of the vector is also only known at run-time.
I am sure I could use the LineReader class for this, however, if the changes required in the CSVReader class are only minor I might try and adapt it.
Do you think the class is apt for this purpose?
in README.md
unsigned get_file_line(unsigned)const;
-> unsigned get_file_line()const;
Is there any plan on supporting loading some format of compressed csv files. It is very common people compress csv files to save space on disk and reduce I/O.
Consider following input:
"Id","Title","Body","Tags"^M
"1","How to check if an uploaded file is an image without mime type?","<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem i s that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>
<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>
","php image-processing file-upload upload mime-types"^M
Here the Body column consists of HTML text with \n as a separator.The newline is denoted with \r\n or ^M. The attached file highlights occurance of all newline characters in the text. I can mail you the sample file in case there is any confusion.Unable to format it correctly using markdown. The data is actually stackoverflow tag prediction data from kaggle which is available openly.
The parser throws following error:
what(): Escaped string was not closed in line 2 in file "../data/pristine/Train_100.txt".
Aborted (core dumped)
I temporarily fixed this issue by a small hack in next_line() function line number 272.
while(buffer[line_end] != '\r' && line_end != data _end){
++line_end;
}
++line_end;//on assumption that \r is followed by \n
I am searching for \r instead of \n. This works for my case but can we have something more robust that handles such input data.
PS: I am new to github as well and this is my first issue
Could you please move the instructions from google code here local in github?
https://code.google.com/p/fast-cpp-csv-parser/issues/detail?id=6
For within some regions, we could not access services from Google =_=!
It seems OK after deleting constexpr and replacing std::snprintf() with fprintf()
How would you recommend reading/writing encrypted CSV files with this parser?
Assume the user has a encrypt/decrypt functions. string encrypt(string); and string decrypt(string)
I could extend your code.. but also any guidance would be great
I have an issue linking the library to our project on Windows,
you can see the original report and Appveyor output at: numenta/nupic.core-legacy#890 (comment)
In the followup we tried adding pthread
as suggested, but didn't help, so we resolved to a workaround with disabled multithreading on Win (-DCSV_IO_NO_THREAD
).
It's not critical but would be nice if the Win version worked the same as on UNIXes.
Hello Ben Strasser,
I have CSV files which I want to import. Unfortunately I do not know numbers of columns before(either at compile time or before opening header in CSV file and counting header columns.)
I'm trying to parse a GTFS file, with a certain number of columns. Is this code conceptually able to parse columns in a random order as long as they are mentioned in the header? Using this code:
io::CSVReader<3> in("/mnt/volatile/skinkie/calgary/stops.txt");
in.read_header(io::ignore_extra_column, "stop_id", "stop_lat", "stop_lon");
std::string quaycoderef; float latitude; double longitude;
while(in.read_row(quaycoderef, latitude, longitude)){
I get the following error:
terminate called after throwing an instance of 'io::error::too_few_columns'
what(): Too few columns in line 2 in file "/mnt/volatile/skinkie/calgary/stops.txt".
With the data:
stop_id,stop_code,stop_name,stop_desc,stop_lat,stop_lon,zone_id,stop_url,location_type
112112,112112,"Crescent Heights High School",51.060931,-114.065158,,,0
I have also tried:
io::CSVReader<3, io::trim_chars<' '>, io::double_quote_escape<',','\"'>> in("/mnt/volatile/skinkie/calgary/stops.txt");
and...
io::CSVReader<9, io::trim_chars<' '>, io::double_quote_escape<',','\"'>> in("/mnt/volatile/skinkie/calgary/stops.txt");
in.read_header(io::ignore_extra_column, "stop_id", "stop_code", "stop_name", "stop_desc", "stop_lat", "stop_lon", "zone_id", "stop_url", "location_type");
std::string quaycoderef; std::string stop_code; std::string stop_name; std::string stop_desc; float latitude; double longitude; std::string zone_id; std::string stop_url; std::string location_type;
while(in.read_row(quaycoderef, stop_code, stop_name, stop_desc, latitude, longitude, zone_id, stop_url, location_type)){
I have a .csv
file with data that looks like the following:
a,b,c
1,2,3
1,,3
1,2,3
I'd like to know that there's no b
value for the second line (rather than interpreting it as 0
). Is it possible to do this?
I have CSV with pipe (|) separator and I cant figure out howto to change separator character.
So I'm trying to parse a .CSV file that has a data member that is distinguished by quotes, but contains line feed in it.
Example line with 3 elements:
Colors,3,"Red
Blue
Green"
Is there a way to parse this with your tool? It seems to read: Colors,3,"Red, and then throw an escaped string not closed error, or a not enough columns error if i dont use escape strings. Any solutions?
Can this class be used to parse a single CSV file without a header that may contain a variable number of columns per line?
For example, each line's first column may contain a type. Some types may be associated with more or less columns than others.
The goal would be to have a way of iterating through each line, peeking at the type in the first column, and parsing other fields as needed based on the type.
Hello,
I have a NFS file which is an append-only CSV. Another process keeps adding more lines to the end and my application keeps checking file size of it: when size increases it picks up new lines by calling pread() and uses fast-cpp-csv-parser to parse each line.
But sometimes I only got partial lines, and I believe it's because of this piece of code in next_line() method:"
if(buffer[line_end] == '\n'){
buffer[line_end] = '\0';
}else{
// some files are missing the newline at the end of the
// last line
++data_end;
buffer[line_end] = '\0';
}
"
Since the NFS file is a moving target, there might be partial lines when the other process each time writes/saves, but it doesn't mean it's the end of file missing the new line.
Thanks!
Sometimes I don't want to read all values.
Is there a way to ignore columns e.g.:
int i;
in.read_row(i, io::ignore, io::ignore, io::ignore);
This would save me a lot of lines.
Hi,
Instead of explicitly typing out variables into which you'd parse the CSV file like this:
std::string id; std::string speed; std::string value; _reader->read_row(id, speed, value);
Is it possible to instead accept a vector containing columns with specific types to do this? Something like the following:
auto cols = new Columns{ ... three string cols ... };
_reader->read_row(cols);
will then populate cols[0], cols[1] and cols[2] as needed. Any suggestions on how this can be done?
Hi,
is there a way to skip arbitrary N
rows of the header? Our CSV files look like this:
colName1, colName2, ...
meta1,meta2
comment1, comment2
data1-1,data2-1
data1-2,data2-2
....
Is there a way to parse this? I'd call set_header("colName1", "colName2")
; then something like: skipToRow(4)
, ...read data
If not, would you consider implementing that?
Thank you,
I get a lot of warnings for unused parameters. Is there any way to fix or suppress them?
This is with gcc (i686-posix-dwarf-rev1, Built by MinGW-W64 project) 4.9.2
.
I use it like this
CSVReader<2, trim_chars<>, no_quote_escape<'\t'>> reader(filename.toStdString());
reader.read_header(ignore_no_column, "time", "painlevel");
And the warnings look like this
In file included from ..\PainDetection\csvreader.h:7:0,
from ..\PainDetection\controller.cpp:2:
..\PainDetection\csv.h:519:51: warning: unused parameter 'line' [-Wunused-parameter]
static bool is_comment(const char*line){
^
..\PainDetection\csv.h: In instantiation of 'static constexpr bool io::trim_chars<trim_char_list>::is_trim_char(char) [with char ...trim_char_list = {}]':
..\PainDetection\csv.h:509:73: required from 'static void io::trim_chars<trim_char_list>::trim(char*&, char*&) [with char ...trim_char_list = {}]'
..\PainDetection\csv.h:707:69: required from 'void io::detail::parse_header_line(char*, std::vector<int>&, const string*, io::ignore_column) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; std::string = std::basic_string<char>; io::ignore_column = unsigned int]'
..\PainDetection\csv.h:964:86: required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_header(io::ignore_column, ColNames ...) [with ColNames = {const char*, const char*}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; io::ignore_column = unsigned int]'
..\PainDetection\controller.cpp:20:61: required from here
..\PainDetection\csv.h:498:57: warning: unused parameter 'c' [-Wunused-parameter]
constexpr static bool is_trim_char(char c){
^
..\PainDetection\csv.h: In instantiation of 'static void io::no_quote_escape<sep>::unescape(char*&, char*&) [with char sep = '\011']':
..\PainDetection\csv.h:708:74: required from 'void io::detail::parse_header_line(char*, std::vector<int>&, const string*, io::ignore_column) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; std::string = std::basic_string<char>; io::ignore_column = unsigned int]'
..\PainDetection\csv.h:964:86: required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_header(io::ignore_column, ColNames ...) [with ColNames = {const char*, const char*}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; io::ignore_column = unsigned int]'
..\PainDetection\controller.cpp:20:61: required from here
..\PainDetection\csv.h:571:44: warning: unused parameter 'col_begin' [-Wunused-parameter]
static void unescape(char*&col_begin, char*&col_end){
^
..\PainDetection\csv.h:571:61: warning: unused parameter 'col_end' [-Wunused-parameter]
static void unescape(char*&col_begin, char*&col_end){
^
..\PainDetection\csv.h: In instantiation of 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t) [with unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]':
..\PainDetection\csv.h:1029:50: recursively required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t, T&, ColType& ...) [with T = int; ColType = {}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]'
..\PainDetection\csv.h:1029:50: required from 'void io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::parse_helper(std::size_t, T&, ColType& ...) [with T = double; ColType = {int}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment; std::size_t = unsigned int]'
..\PainDetection\csv.h:1053:64: required from 'bool io::CSVReader<column_count, trim_policy, quote_policy, overflow_policy, comment_policy>::read_row(ColType& ...) [with ColType = {double, int}; unsigned int column_count = 2u; trim_policy = io::trim_chars<>; quote_policy = io::no_quote_escape<'\011'>; overflow_policy = io::throw_on_overflow; comment_policy = io::no_comment]'
..\PainDetection\controller.cpp:24:39: required from here
..\PainDetection\csv.h:1012:47: warning: unused parameter 'r' [-Wunused-parameter]
void parse_helper(std::size_t r){}
^
Fails to parse following input:
"Id","Title","Body","Tags"^M
"1","How to check if an uploaded file is an image without mime type?","<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt=""""></p>
","php image-processing file-upload upload mime-types"^M
The output for Body column should be
<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt=""></p>
But the ouput is:
<p><img src="http://i.stack.imgur.com/bA7Tz.jpg" alt="></p>
I get this error when there is a comma enclosed with quotes. Is there a way to ignore commas inside quotes?
Hi! Many thanks for a beautiful library.
I noticed that when we use in.read_header(io::ignore_extra_column, "vendor", "size", "speed");
if the csv file's last column is not "vendor", "size" or "speed" a too_many_columns
error is thrown.
Would it make sense to not throw that error if we used the ignore_extra_column
policy on the reader?
I seem to have trouble compiling this code if the number of columns is greater than 10. For example here is code with 11 columns specified.
Invoking.. $ gcc -g -Wall -std=c++11 -I . -c test1.cpp
where test1.cpp is...
#include "src/include/fast-cpp-csv-parser/csv.h"
int main(int argc, char* argv[] ) {
io::CSVReader<11> * in = new io::CSVReader<11>("testdata.csv");
in->set_header("a","b","c","d","e","f","g","h","i","j",'k');//,"l");
typedef char * valtype;
valtype sa=0,sb=0,sc=0,sd=0,se=0,sf=0,sg=0,sh=0,si=0,sj=0,sk=0;//,sl=0;
in->read_row(sa,sb,sc,sd,se,sf,sg,sh,si,sj,sk);//,sl);
}
The error message include:
./src/include/fast-cpp-csv-parser/csv.h:1104:41: error: no matching function for call to ‘io::CSVReader<11u>::set_column_names(char)’
set_column_names(std::forward<ColNames>(cols)...);
^
The 3,6,9,10 column versions compile fine. I've tried this with gcc ver 4.9.2 and ver 5.4.0
Hello,
My current application need to parse a CSV stream produced by a WebService.
Presently, i have to write it down to disk to use your parser, maybe it would be useful to create an interface for (string)stream ?
So I have a csv file which contains 480 rows of 640 comma-separated numbers (so 640 columns of 480 values each) - representing a 480x640 matrix that was stored in the csv file. Is it possible to read this file into a cv::Mat or a standard float[][] matrix in C++ in one go? i.e. instead of copying 640 columns and figuring out the mapping to a Mat, can this copy all this data into a matrix? I don't have any headers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.