stride - invalid command about pg_bulkload

[mikecaat]

Hi,

Though I took a fast look, the "stride" seems to be supported for only binary not csv.

http://ossc-db.github.io/pg_bulkload/pg_bulkload.html

Binary input format
STRIDE = n
Length of one row. Use if you want to truncate the end of row. The default is whole of the row, which means the total of COLs.

[mybigman]

@mikecaat - looks like your spot on, thanks.

I am trying to move from mariadb to postgres and their "copy" command is not very flexible compared to "mysqlimport".

We digest huge csv files daily and the way the csv files are they have an additional separator at the end if its an empty field or not.

a^b^c^d^^
a^b^c^d^e^

The files are just to large to run "sed" over them and haven't found a clean way of handling this :/

[mikecaat]

Hmm, is the following your expected result?

$ cat /tmp/test.csv
a^b^c^d^^
a^b^c^d^e^

psql=# CREATE TABLE test (a text, b text, c text, d text, e text, f text);
CREATE TABLE
psql=# COPY test FROM '/tmp/test.csv' WITH (FORMAT CSV, DELIMITER '^');
COPY 2
psql=# SELECT * FROM test;
 a | b | c | d |   e    |   f
---+---+---+---+--------+--------
 a | b | c | d | (null) | (null)
 a | b | c | d | e      | (null)
(2 rows)

[mybigman]

Whoops sorry my mistake...

I comes likes this. It always adds one extra regardless :/

a^b^c^d^^^
a^b^c^d^e^^

The other hassle is that when they decide to add new fields to the end of the csv file we have to race and update the table with the new fields.

With mysqlimport any additional field is completely ignored and still loads the table, this gives us time to add extra fields at our leisure.

[mikecaat]

Oh, that’s hassle.

Though there are some workaround including using the pg_bulkload’s Fillter function, I agree that mysqlimport’s way is better for your case.
https://stackoverflow.com/questions/12618232/copy-a-few-of-the-columns-of-a-csv-file-into-a-table

[mybigman]

Thanks mate for the info... I will see what I can come up with that's some what reliable, otherwise we will be sticking with mariadb :)