In Pg_bulkload, following syntax is used for SQL FILTER.

CREATE FUNCTION sample_filter (integer, text, text, real DEFAULT 0.05) RETURNS record
    AS $$ SELECT $ 1 * $ 4, upper ($ 3) $$
    LANGUAGE SQL;

I can't find any sample syntax for C programming.As per documentation, C filter is much faster than SQL filter.

Anybody have used C programming Filter? Kindly provide some sample syntax?

Thanks

Hi,

I got SEGV error with large input file that includes parse error data when MULTI_PROCES enabled. Input file is 13MB 500000 lines csv file and contains parse error data like "" at 153249 line. And the control file is written like follows,

OUTPUT = wikipedia_title                   # [<schema_name>.]table_name
#INPUT = /path/to/test.csv  # Input data location (absolute path)
TYPE = CSV                            # Input file type
QUOTE = "\""                          # Quoting character
ESCAPE = \                            # Escape character for Quoting
DELIMITER = ","                       # Delimiter
WRITER = BUFFERED
VERBOSE = YES
MULTI_PROCESS = YES

Also error message I got is following.

WARNING:  Parse error Record 1: Input Record 153249: Rejected - column 92. unterminated CSV quoted field
WARNING:  Maximum parse error count exceeded - 1 error(s) found in input file
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: LOG:  server process (PID 88994) was terminated by signal 11: Segmentation fault

I think the main cause of this problem is that AsyncSource could not allocate new memory in appropriate memory context when expanding read buffer. That memory context AsyncSource allocates for the read buffer could be switched by main thread and the current pg_bulkload doesn't take care about it. Because pg_bulkload resets the used memory context when parse error occurred and then clean up the read buffer, pg_bulkload could try to free the memory that is already reseted by MemoryContextReset().

I used the DROP EXTENSION command to drop the pg_bulkload extension from my database a few weeks ago, and then removed the RPMs from the server.

Today I noticed that the public.pg_bulkload function was still there in the database (when pg_upgrade --check pointed out that $libdir/pg_bulkload was missing). Should this function have been manually dropped after dropping the extension? Shouldn't dropping the extension drop the function?

This was installed by my predecessors so I can't say how it was installed.

Looks like the last version released to PGXN was 3.1.9. Any chance of an update?

pg_bulkload sometimes makes requests with unreasonable size to palloc, causing errors shown in the subject. What's worse, any previous error messages are overwritten, see below:

I tried to load a CSV file containing 500,000 lines with some parse errors, where it showed the error message correctly (no invalid memory alloc requests here)

$ pg_bulkload /tmp/p.ctl
NOTICE: BULK LOAD START
WARNING:  Parse error Record 1: Input Record 1: Rejected - column 2. unterminated CSV quoted field
WARNING:  Maximum parse error count exceeded - 1 error(s) found in input file
NOTICE: BULK LOAD END
	0 Rows skipped.
	0 Rows successfully loaded.
	1 Rows not loaded due to parse errors.
	0 Rows not loaded due to duplicate errors.
	0 Rows replaced with new rows.
WARNING: some rows were not loaded due to errors.

But with 10,000,000 lines, things turned to this:

$ pg_bulkload /tmp/p.ctl
NOTICE: BULK LOAD START
WARNING:  Parse error Record 1: Input Record 1: Rejected - column 2. invalid memory alloc request size 1074789376
WARNING:  Maximum parse error count exceeded - 1 error(s) found in input file
NOTICE: BULK LOAD END
	0 Rows skipped.
	0 Rows successfully loaded.
	1 Rows not loaded due to parse errors.
	0 Rows not loaded due to duplicate errors.
	0 Rows replaced with new rows.

pg_bulkload shouldn't make palloc requests of unreasonable size. :)

type = function
input = generate_series(1,1000)
error: function generate_series(integer, integer) is not unique

Some data, which could not be inserted by pg_bulkload, could be inserted by another tool called pg_loader.
For instance, I use sql.txt to build up a table ,and insert it with the data in instance.txt.
However, it failed by using pg_bulkloader, and succeeded by using pg_loader,

The cmd I use with pg_loader is below:
pgloader --type csv --field length,city,interface_type,xdr_id,rat,imsi,imei,msisdn,machine_ip_address_type,sgw_ggsn_ip_address,enb_sgsn_ip_address,sgw_ggsn_port,enb_sgsn_port,enb_sgsn_gtp_teid,sgw_ggsn_gtp_teid,tac,cell_id,apn,app_type_code,procedure_start_time,procedure_end_time,protocol_type,app_type,app_sub_type,app_content,app_status,user_ipv4,user_ipv6,user_port,l4_protocal,app_server_ip_ipv4,app_server_ip_ipv6,app_server_port,ul_data,dl_data,ul_ip_packets,dl_ip_packets,ul_disorder_ip_packets,dl_disorder_ip_packets,ul_retrans_ip_packets,dl_retrans_ip_packets,tcp_response_time,tcp_ack_time,ul_ip_frag_packets,dl_ip_frag_packets,first_req_time,first_response_time,window_sec,mss,tcp_syn_num,tcp_status,session_end,http_version,message_type,message_status,first_http_response_time,last_content_packet_time,last_ack_time,host,uri,x_online_host,user_agent,http_content_type,refer_uri,cookie,content_length,target_behaviour,wtp_abort_type,wtp_abort_reason,title,key_word,service_behavior_flag,service_comp_flag,service_time,ie,portal,reserve1,reserve2,reserve3,reserve4 --with workers=16 --with concurrency=4 --with "fields terminated by '|'" --with "fields not enclosed" /software/bulkloader10test/instance.txt postgresql://postgres:postgres@:5432/posteva?tablename=raw_s1u_http_test

sql.txt

instance.txt

I have more than 100 columns from input file. So i used more than 100 arguments in Postgres function. But Postgres doesn't support 100 arguments in function.

Then i tried using composite type with simple example;

Create Type;

CREATE TYPE test_123 AS (
name text,
supplier_id integer,
price numeric
);

Then called Type inside the Postgres function like below;

CREATE OR REPLACE FUNCTION test123(test_123)
RETURNS record AS
$BODY$ SELECT $1.name, $1.price $BODY$
LANGUAGE sql VOLATILE
COST 100;

Created Table:

create table test123
(
name text,
price numeric(5,0)
)

Input file;
raja|0|100

But the table was not loaded; Got the below error;

Parse error Record 1: Input Record 1: Rejected - column 3. extra data after last expected column
Maximum parse error count exceeded - 1 error(s) found in input file

If input file has data something like this, it works.

(raja,0,100)

Did the conversion using UNIX SED command from raja|0|100 to (raja,0,100). It's taking 5 minutes for 16 million data. But it's not ideal solution.

Can you please help me, how this case can be handled? Our data was supposed to be delimited without braces. Definitely, someone would have faced this issue before.

Thanks

It seems we have a problem.

A failure during trying to load into a composite type field (a record value) causes TupleDesc leak (I see that leaking TupleDesc is for the record's type)

$ psql
create type foot as (a int, b char);
create table foo (a foot);

$ cat /tmp/foo.csv
"(1,i)"
"(i,1)"

$ pg_bulkload -O foo -i /tmp/foo.csv
NOTICE: BULK LOAD START
WARNING:  TupleDesc reference leak: TupleDesc 0x7fc3f8304ea8 (28062,-1) still referenced
NOTICE: BULK LOAD END
	0 Rows skipped.
	1 Rows successfully loaded.
	1 Rows not loaded due to parse errors.
	0 Rows not loaded due to duplicate errors.
	0 Rows replaced with new rows.
WARNING: some rows were not loaded due to errors.

The error occurs for the second record:

Parse error Record 1: Input Record 2: Rejected - column 1. invalid input syntax for integer: "i"

We should have some way of preventing that leak: Maybe, PG_CATCH() block in ReaderNext() is a place to fix such an issue. Not yet sure how. The error occurs deep within PostgreSQL guts (in this case, when integer type input function is called to parse the line's field to populate a column).

Is it possible to use pg_bulkload with foreign tables (e.g. cstore_fdw)? I tried it, but I got the following error message:

ERROR: query failed: ERROR:  cannot load to non-table relation "videos"

Hi to everybody,

I must ask for apologize due to this is not an issue. I just want to consult expert people in the best place.

I'm developing a ruby gem that use pg_bulkload to load a very huge CSV file, but it's possible that the proccess could be interrupted. How could I stop a running pg_bulkload in a safe way? I mean, the CSV file must be loaded normally just until the moment of interruption. Do I must sent a especial kill code ?(something else to kill -9)
Please any suggest are welcome.

Thanks in advance.

I'm having trouble installing pg_bulkload.
The server originally hosted postgresql 9.0 with pg_bulkload 3.0 working fine but I have since removed V9.0 and upgraded to V9.3 but I can't get pg_bulkload 3.1 to re-install.
Can you see anything obvious as to why it won't install for me from the log below.
Something to do with -ledit so I'm missing a package of some sort but can't work out what.

Any help much appreciated.

Paul

paul@re-srv-2ek:/pg_bulkload_3.1$ make
make[1]: Entering directory /home/paul/pg_bulkload_3.1/bin' gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.5 -c -o pg_bulkload.o pg_bulkload.c gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.5 -c -o recovery.o recovery.c gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.5 -c -o pgut/pgut.o pgut/pgut.c gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.5 -c -o pgut/pgut-fe.o pgut/pgut-fe.c gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.5 -c -o pgut/pgut-list.o pgut/pgut-list.c gcc -g -O2 -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o -L/usr/lib -lpq -L/usr/lib -Wl,-Bsymbolic-functions -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib//mit-krb5 -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib//mit-krb5 -Wl,--as-needed -lpgport -lpgcommon -lpam -lssl -lcrypto -lkrb5 -lcom_err -lgssapi_krb5 -lz -ledit -lcrypt -ldl -lm -o pg_bulkload /usr/bin/ld: cannot find -ledit collect2: ld returned 1 exit status make[1]: *** [pg_bulkload] Error 1 make[1]: Leaving directory/home/paul/pg_bulkload_3.1/bin'
make: *** [all] Error 2
paul@re-srv-2ek:/pg_bulkload_3.1$

This doesn't seem to handle Unicode, formatted as UTF-8 for input data.

We have following scenario

Server-1(DB)
Server-2(ETL)-Etl jobs will write the data to a file; These files will be loaded to table which was present in server1 using pg_bulkload.

Planning to install only pg_client .Will it work without postgres server?

Thanks

Hello,

I have two problems and I hope to get your help.

1 How to load part of the column from csv file to a table?

A cvs file contains following column:

value_a|value_b|value_c|value_d|value_e|value_f

The table only contains three of them:

column_a,
column_b,
column_c

so,what should I do to load value_a to column_a, value_b tocolumn_b, value_c to column_c?

Is there any functions which can provide arguments to let us load columns which we wanted to table?

2 Load indntity type value.

When load indntity type value, when the value is NULL, can it automatically be assigned a non-NULL value? How to do?

best regards, thank you.

Hi to everybody,
When I use pg_bulkload with -P, -u and -l flags, permissions for files generated are set to 600 ( -rw- --- --- ). How could I modify that behavior in order to set 644 for generated files automatically. I can't use sudo to execute a chmod command.

Welcome any suggest. Thanks a lot!.

Hello,

Can pg_loader add or change function to import specified column from a CSV file to postgresSQL?

Waiting for your reply.

Best regards, thank you.

Is there any develop plan?

Hi all, I noticed this problem in my log, can you give me some hint about it ?
It seems a pg_bulkload crash, how can I find more details about this ?
TIA

*** buffer overflow detected ***: postgres: postgres DatabaseName [local] SELECT terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7feb01837597]
/lib64/libc.so.6(+0x10c750)[0x7feb01835750]
/lib64/libc.so.6(+0x10e507)[0x7feb01837507]
/usr/pgsql-9.6/lib/pg_bulkload.so(+0x15ef6)[0x7feaedc86ef6]
/usr/pgsql-9.6/lib/pg_bulkload.so(+0x12b8b)[0x7feaedc83b8b]
/usr/pgsql-9.6/lib/pg_bulkload.so(pg_bulkload+0x615)[0x7feaedc7f1a5]
< 2016-12-22 14:48:22.035 CET > LOG: server process (PID 6849) was terminated by signal 11: Segmentation fault
< 2016-12-22 14:48:22.035 CET > DETAIL: Failed process was running: SELECT * FROM pg_bulkload(ARRAY['TYPE=TUPLE','INPUT=' || $1,'WRITER=DIRECT','OUTPUT=' || $2,'ON_DUPLICATE_KEEP=' || $3,'DUPLICATE_ERRORS=' || $4,'DUPLICATE_BADFILE=' || $5,'LOGFILE=' || $6,'VERBOSE=' || $7,'TRUNCATE=' || $8])
< 2016-12-22 14:48:22.035 CET > LOG: terminating any other active server processes
< 2016-12-22 14:48:22.035 CET > WARNING: terminating connection because of crash of another server process
< 2016-12-22 14:48:22.035 CET > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
< 2016-12-22 14:48:22.035 CET > HINT: In a moment you should be able to reconnect to the database and repeat your command.
< 2016-12-22 14:48:22.035 CET > WARNING: terminating connection because of crash of another server process
< 2016-12-22 14:48:22.035 CET > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
< 2016-12-22 14:48:22.035 CET > HINT: In a moment you should be able to reconnect to the database and repeat your command.
< 2016-12-22 14:48:22.035 CET > WARNING: terminating connection because of crash of another server process
< 2016-12-22 14:48:22.035 CET > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
< 2016-12-22 14:48:22.035 CET > HINT: In a moment you should be able to reconnect to the database and repeat your command.
< 2016-12-22 14:48:22.036 CET > WARNING: terminating connection because of crash of another server process
< 2016-12-22 14:48:22.036 CET > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
< 2016-12-22 14:48:22.036 CET > HINT: In a moment you should be able to reconnect to the database and repeat your command.
< 2016-12-22 14:48:22.040 CET > WARNING: terminating connection because of crash of another server process
< 2016-12-22 14:48:22.040 CET > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
< 2016-12-22 14:48:22.040 CET > HINT: In a moment you should be able to reconnect to the database and repeat your command.
< 2016-12-22 14:48:22.061 CET > LOG: all server processes terminated; reinitializing
< 2016-12-22 14:48:22.128 CET > LOG: database system was interrupted; last known up at 2016-12-22 14:46:52 CET
< 2016-12-22 14:48:22.237 CET > LOG: database system was not properly shut down; automatic recovery in progress
< 2016-12-22 14:48:22.240 CET > LOG: redo starts at 95/85C84CA0
< 2016-12-22 14:48:22.797 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.801 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.806 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.839 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.872 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.876 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.878 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.882 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.889 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.904 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.910 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.915 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.928 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.932 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.937 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.943 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.948 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.950 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:22.958 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:23.155 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:23.179 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:23.322 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:25.852 CET > LOG: unexpected pageaddr 95/5C432000 in log segment 00000001000000950000008E, offset 4399104
< 2016-12-22 14:48:25.852 CET > LOG: redo done at 95/8E431ED0
< 2016-12-22 14:48:25.852 CET > LOG: last completed transaction was at log time 2016-12-22 14:48:15.167961+01
< 2016-12-22 14:48:27.573 CET > FATAL: the database system is in recovery mode
< 2016-12-22 14:48:28.090 CET > LOG: MultiXact member wraparound protections are now enabled
< 2016-12-22 14:48:28.107 CET > LOG: database system is ready to accept connections
< 2016-12-22 14:48:28.112 CET > LOG: autovacuum launcher started

Checksum error raised when pg_bulkload loads to an existing table if the table size becomes more than 1GB.

Reproduce.
PostgreSQL 9.4.3, pg_bulkload 3.1.8

initdb with "--data-checksums".
$ pgbench -i -s 100
$ psql -c "COPY (SELECT * FROM pgbench_accounts WHERE aid <= 7000000) TO '/tmp/test1.csv' WITH CSV;"
$ psql -c "COPY (SELECT * FROM pgbench_accounts WHERE aid > 7000000) TO '/tmp/test2.csv' WITH CSV;"
$ psql -c "CREATE TABLE testtbl (LIKE pgbench_accounts);"
$ cat >/tmp/test1.ctl
OUTPUT = testtbl
INPUT = /tmp/test1.csv
TYPE = CSV
^d
$ pg_bulkload /tmp/test1.ctl
$ psql -c "ALTER TABLE testtbl ADD CONSTRAINT testtbl_pkey PRIMARY KEY (aid);"
$ psql -c "ALTER TABLE testtbl DROP CONSTRAINT testtbl_pkey;"
$ cat >/tmp/test2.ctl
OUTPUT = testtbl
INPUT = /tmp/test2.csv
TYPE = CSV
^d
$ pg_bulkload /tmp/test2.ctl
$ psql -c "ALTER TABLE testtbl ADD CONSTRAINT testtbl_pkey PRIMARY KEY (aid);"

WARNING:  page verification failed, calculated checksum 48076 but expected 0
ERROR:  invalid page in block 130568 of relation base/12146/16431

While generating build for this tool I am facing below traceback.

`$ make

make[1]: Entering directory /pg_bulkload/bin'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -fno-omit-frame-pointer pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o -L/usr/lib/x86_64-linux-gnu -lpq -L/usr/lib/x86_64-linux-gnu -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib/x86_64-linux-gnu/mit-krb5 -Wl,--as-needed -lpgcommon -lpgport -lselinux -lpam -lssl -lcrypto -lgssapi_krb5 -lz -ledit -lrt -lcrypt -ldl -lm -o pg_bulkload
/usr/bin/ld: cannot find -lselinux
/usr/bin/ld: cannot find -lpam
/usr/bin/ld: cannot find -ledit
collect2: error: ld returned 1 exit status
/usr/lib/postgresql/9.5/lib/pgxs/src/makefiles/pgxs.mk:296: recipe for target 'pg_bulkload' failed
make[1]: *** [pg_bulkload] Error 1
make[1]: Leaving directory '/pg_bulkload/bin'
Makefile:27: recipe for target 'all' failed
make: *** [all] Error 2
`

Operating system: Ubuntu 16.04, 64bit

Hi, i try to import csv ( ~100K rows) with some multiline columns. Allways recieve same error

NOTICE: BULK LOAD START
NOTICE: BULK LOAD END
0 Rows skipped.
0 Rows successfully loaded.
1 Rows not loaded due to parse errors.
0 Rows not loaded due to duplicate errors.
0 Rows replaced with new rows.
WARNING: some rows were not loaded due to errors.

Hi,

Is it possible to change the format of PARSE_BAD_FILE and DUPLICATE_BADFILE ?
My input file looks like:
31/12/2016|00000001|RUIZ GOMES CARLOS ALBERTO|00023232||09/12/1998|15/05/1959|1|S96|1|1|E|1|1|0 |03-04|07/10/2004

I need that PARSE_BADFILE y DUPLICATE_BADFILE to have the same structure. Instead the contents looks like this:
00000001,"RUIZ GOMES CARLOS ALBERTO",00023232,,1998-12-09,1959-05-15,t,S96,1,1,E,1,1,0,"2016-12-31 00:00:00-04","2004-10-07 00:00:00-04",03-04

Any suggest?.

Hello,
I'm searching for "import tool" as standalone application to be able to perform import to remote database. As I've read in the documentation it's needed to have postgresql-server installed before. Is there any possibility to have standalone installation without need of postgresql-server? Or what are the steps if I only need pg_bulkload as a client to import data to remote db?

Many thanks

Having trouble building from source--make can't find libpq.so.5, even though it is right where pg_config says it is.

Any ideas?

Thanks,
Jeff

[root@717311-app1 pg_bulkload-3.1.8]# which pg_config
/usr/pgsql-9.4/bin/pg_config

[root@717311-app1 pg_bulkload-3.1.8]# pg_config
BINDIR = /usr/pgsql-9.4/bin
DOCDIR = /usr/share/doc/postgresql
HTMLDIR = /usr/share/doc/postgresql
INCLUDEDIR = /usr/pgsql-9.4/include
PKGINCLUDEDIR = /usr/pgsql-9.4/include
INCLUDEDIR-SERVER = /usr/pgsql-9.4/include/server
LIBDIR = /usr/pgsql-9.4/lib
PKGLIBDIR = /usr/pgsql-9.4/lib
LOCALEDIR = /usr/pgsql-9.4/share/locale
MANDIR = /usr/pgsql-9.4/share/man
SHAREDIR = /usr/pgsql-9.4/share
SYSCONFDIR = /etc/sysconfig/pgsql
PGXS = /usr/pgsql-9.4/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--disable-rpath' '--prefix=/usr/pgsql-9.4' '--includedir=/usr/pgsql-9.4/include' '--mandir=/usr/pgsql-9.4/share/man' '--datadir=/usr/pgsql-9.4/share' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib64' '--with-gssapi' '--with-openssl' '--with-pam' '--with-includes=/usr/include' '--with-libraries=/usr/lib64' '--enable-nls' '--with-uuid=e2fs' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--docdir=/usr/share/doc' '--with-zlib'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2
CFLAGS_SL = -fpic
LDFLAGS = -L../../../src/common -L/usr/lib64 -Wl,--as-needed
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -lcrypt -ldl -lm
VERSION = PostgreSQL 9.4.4

[root@717311-app1 pg_bulkload-3.1.8]# ls -al /usr/pgsql-9.4/lib/libpq.so.5
lrwxrwxrwx 1 root root 12 Aug 20 17:39 /usr/pgsql-9.4/lib/libpq.so.5 -> libpq.so.5.7

[root@717311-app1 pg_bulkload-3.1.8]# make clean
make[1]: Entering directory `/root/pg_bulkload-3.1.8/bin'
rm -f sql/load_function.sql sql/load_filter.sql
rm -f sql/init.sql sql/init-{8.3,8.4,9.0,9.1,9.2,9.3,9.4}.sql
rm -f pg_bulkload
rm -f pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o
rm -rf results/ regression.diffs regression.out tmp_check/ log/
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/bin'
make[1]: Entering directory `/root/pg_bulkload-3.1.8/lib'
rm -f nbtsort.c
rm -f pg_bulkload.so   libpg_bulkload.a  libpg_bulkload.pc
rm -f pg_bulkload.sql
rm -f binary.o logger.o parser_binary.o parser_csv.o parser_function.o parser_tuple.o pg_btree.o pg_bulkload.o pg_strutil.o reader.o source.o writer.o writer_binary.o writer_buffered.o writer_direct.o writer_parallel.o pgut/pgut-be.o pgut/pgut-ipc.o
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/lib'
make[1]: Entering directory `/root/pg_bulkload-3.1.8/util'
rm -f pg_timestamp.so   libpg_timestamp.a  libpg_timestamp.pc
rm -f pg_timestamp.sql
rm -f pg_timestamp.o
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/util'

[root@717311-app1 pg_bulkload-3.1.8]# make
make[1]: Entering directory `/root/pg_bulkload-3.1.8/bin'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_bulkload.o pg_bulkload.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o recovery.o recovery.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgut/pgut.o pgut/pgut.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgut/pgut-fe.o pgut/pgut-fe.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgut/pgut-list.o pgut/pgut-list.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o -L/usr/pgsql-9.4/lib -lpq -L/usr/pgsql-9.4/lib -L/usr/lib64 -Wl,--as-needed  -lpgcommon -lpgport -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -lcrypt -ldl -lm -o pg_bulkload
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/bin'
make[1]: Entering directory `/root/pg_bulkload-3.1.8/lib'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o binary.o binary.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o logger.o logger.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o parser_binary.o parser_binary.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o parser_csv.o parser_csv.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o parser_function.o parser_function.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o parser_tuple.o parser_tuple.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_btree.o pg_btree.c
nbtree/nbtsort-9.4.c:178: warning: ‘unused_bt_spooldestroy’ defined but not used
nbtree/nbtsort-9.4.c:188: warning: ‘unused_bt_spool’ defined but not used
nbtree/nbtsort-9.4.c:198: warning: ‘unused_bt_leafbuild’ defined but not used
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_bulkload.o pg_bulkload.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_strutil.o pg_strutil.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o reader.o reader.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o source.o source.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o writer.o writer.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o writer_binary.o writer_binary.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o writer_buffered.o writer_buffered.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o writer_direct.o writer_direct.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o writer_parallel.o writer_parallel.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgut/pgut-be.o pgut/pgut-be.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I/usr/pgsql-9.4/include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgut/pgut-ipc.o pgut/pgut-ipc.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -shared -o pg_bulkload.so binary.o logger.o parser_binary.o parser_csv.o parser_function.o parser_tuple.o pg_btree.o pg_bulkload.o pg_strutil.o reader.o source.o writer.o writer_binary.o writer_buffered.o writer_direct.o writer_parallel.o pgut/pgut-be.o pgut/pgut-ipc.o -L/usr/pgsql-9.4/lib -L/usr/lib64 -Wl,--as-needed  -L/usr/pgsql-9.4/lib -lpq
sed 's,MODULE_PATHNAME,$libdir/pg_bulkload,g' pg_bulkload.sql.in >pg_bulkload.sql
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/lib'
make[1]: Entering directory `/root/pg_bulkload-3.1.8/util'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -I../include -I. -I./ -I/usr/pgsql-9.4/include/server -I/usr/pgsql-9.4/include/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pg_timestamp.o pg_timestamp.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -fpic -shared -o pg_timestamp.so pg_timestamp.o -L/usr/pgsql-9.4/lib -L/usr/lib64 -Wl,--as-needed
sed 's,MODULE_PATHNAME,$libdir/pg_timestamp,g' pg_timestamp.sql.in >pg_timestamp.sql
make[1]: Leaving directory `/root/pg_bulkload-3.1.8/util'

[root@717311-app1 pg_bulkload-3.1.8]# ldd bin/pg_bulkload
    linux-vdso.so.1 =>  (0x00007ffcf5db2000)
    libpq.so.5 => not found
    libc.so.6 => /lib64/libc.so.6 (0x00000030a5a00000)
    /lib64/ld-linux-x86-64.so.2 (0x00000030a5200000)
[root@717311-app1 pg_bulkload-3.1.8]#

[root@717311-app1 pg_bulkload-3.1.8]# bin/pg_bulkload
bin/pg_bulkload: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory
[root@717311-app1 pg_bulkload-3.1.8]#

PG 10 has partitioned tables. We cannot target the partitioned tables directly because they do not have their own storage. But we can load data directly into partitions, assuming the data being loaded satisfies the partition constraint. PostgreSQL will apply the partition constraint if data is directly inserted into a partition.

I think pg_bulkload should do the same. CheckerInit() should also initialize the partition constraint, in addition to traditional constraints.

Hi,

I am getting these while building 3.1.8 against PostgreSQL 9.5:

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -DLINUX_OOM_ADJ=0 -fpic -I../include -I/usr/pgsql-9.5/include -I. -I./ -I/usr/pgsql-9.5/include/server -I/usr/pgsql-9.5/include/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o source.o source.c
pg_btree.c:47:2: error: #error unsupported PostgreSQL version
#error unsupported PostgreSQL version
^
pg_btree.c:100:42: error: unknown type name 'BTWriteState'
static void _bt_mergeload(Spooler _self, BTWriteState *wstate, BTSpool *btspool,
^
pg_btree.c: In function 'SpoolerOpen':
pg_btree.c:132:2: error: too few arguments to function 'ExecOpenIndices'
ExecOpenIndices(self->relinfo);
^
In file included from pg_btree.c:19:0:
/usr/pgsql-9.5/include/server/executor/executor.h:366:13: note: declared here
extern void ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative);
^
pg_btree.c: In function 'IndexSpoolBegin':
pg_btree.c:211:13: error: dereferencing pointer to incomplete type 'BTSpool {aka struct BTSpool}'
spools[i]->isunique = indices[i]->rd_index->indisunique;
^
pg_btree.c: In function 'IndexSpoolEnd':
pg_btree.c:247:4: error: too few arguments to function 'reindex_index'
reindex_index(indexOid, false);
^
In file included from pg_btree.c:18:0:
/usr/pgsql-9.5/include/server/catalog/index.h:117:13: note: declared here
extern void reindex_index(Oid indexId, bool skip_constraint_checks,
^
pg_btree.c: In function 'IndexSpoolInsert':
pg_btree.c:343:14: warning: passing argument 1 of '_bt_spool' from incompatible pointer type [-Wincompatible-pointer-types]
_bt_spool(itup, spools[i]);
^
In file included from pg_btree.c:15:0:
/usr/pgsql-9.5/include/server/access/nbtree.h:748:13: note: expected 'BTSpool * {aka struct BTSpool *}' but argument is of type 'IndexTuple {aka struct IndexTupleData *}'
extern void _bt_spool(BTSpool *btspool, ItemPointer self,
^
pg_btree.c:343:20: warning: passing argument 2 of '_bt_spool' from incompatible pointer type [-Wincompatible-pointer-types]
_bt_spool(itup, spools[i]);
^
In file included from pg_btree.c:15:0:
/usr/pgsql-9.5/include/server/access/nbtree.h:748:13: note: expected 'ItemPointer {aka struct ItemPointerData *}' but argument is of type 'BTSpool * {aka struct BTSpool *}'
extern void _bt_spool(BTSpool *btspool, ItemPointer self,
^
pg_btree.c:343:4: error: too few arguments to function '_bt_spool'
_bt_spool(itup, spools[i]);
^
In file included from pg_btree.c:15:0:
/usr/pgsql-9.5/include/server/access/nbtree.h:748:13: note: declared here
extern void _bt_spool(BTSpool *btspool, ItemPointer self,
^
pg_btree.c: In function 'bt_mergebuild':
pg_btree.c:359:2: error: unknown type name 'BTWriteState'
BTWriteState wstate;
^
pg_btree.c:367:8: error: request for member 'index' in something not a structure or union
wstate.index = btspool->index;
^
pg_btree.c:375:8: error: request for member 'btws_use_wal' in something not a structure or union
wstate.btws_use_wal = self->use_wal &&
^
pg_btree.c:376:3: warning: implicit declaration of function 'XLogIsNeeded' [-Wimplicit-function-declaration]
XLogIsNeeded() && !RELATION_IS_LOCAL(wstate.index);
^
In file included from /usr/pgsql-9.5/include/server/storage/lmgr.h:20:0,
from pg_btree.c:21:
pg_btree.c:376:46: error: request for member 'index' in something not a structure or union
XLogIsNeeded() && !RELATION_IS_LOCAL(wstate.index);
^
/usr/pgsql-9.5/include/server/utils/rel.h:453:4: note: in definition of macro 'RELATION_IS_LOCAL'
((relation)->rd_islocaltemp ||
^
pg_btree.c:376:46: error: request for member 'index' in something not a structure or union
XLogIsNeeded() && !RELATION_IS_LOCAL(wstate.index);
^
/usr/pgsql-9.5/include/server/utils/rel.h:454:4: note: in definition of macro 'RELATION_IS_LOCAL'
(relation)->rd_createSubid != InvalidSubTransactionId)
^
pg_btree.c:383:8: error: request for member 'btws_pages_alloced' in something not a structure or union
wstate.btws_pages_alloced = BTREE_METAPAGE + 1;
^
pg_btree.c:384:8: error: request for member 'btws_pages_written' in something not a structure or union
wstate.btws_pages_written = 0;
^
pg_btree.c:385:8: error: request for member 'btws_zeropage' in something not a structure or union
wstate.btws_zeropage = NULL; / until needed */
^
pg_btree.c:392:21: error: request for member 'index' in something not a structure or union
LockRelation(wstate.index, AccessExclusiveLock);
^
pg_btree.c:393:29: error: request for member 'index' in something not a structure or union
FlushRelationBuffers(wstate.index);
^
pg_btree.c:396:38: error: request for member 'index' in something not a structure or union
merge = BTReaderInit(&reader, wstate.index);
^
In file included from /usr/pgsql-9.5/include/server/postgres.h:48:0,
from ../include/pg_bulkload.h:9,
from pg_btree.c:11:
pg_btree.c:399:33: error: request for member 'index' in something not a structure or union
RelationGetRelationName(wstate.index),
^
/usr/pgsql-9.5/include/server/utils/elog.h:212:23: note: in definition of macro 'elog'
elog_finish(elevel, VA_ARGS);
^
/usr/pgsql-9.5/include/server/utils/rel.h:366:3: note: in expansion of macro 'NameStr'
(NameStr((relation)->rd_rel->relname))
^
pg_btree.c:399:3: note: in expansion of macro 'RelationGetRelationName'
RelationGetRelationName(wstate.index),
^
pg_btree.c:401:9: error: request for member 'btws_use_wal' in something not a structure or union
wstate.btws_use_wal ? "with" : "without");
^
/usr/pgsql-9.5/include/server/utils/elog.h:212:23: note: in definition of macro 'elog'
elog_finish(elevel, VA_ARGS);
^
pg_btree.c:404:34: error: request for member 'index' in something not a structure or union
RelationSetNewRelfilenode(wstate.index, InvalidTransactionId);
^
pg_btree.c:404:2: error: too few arguments to function 'RelationSetNewRelfilenode'
RelationSetNewRelfilenode(wstate.index, InvalidTransactionId);
^
In file included from /usr/pgsql-9.5/include/server/access/genam.h:21:0,
from pg_btree.c:13:
/usr/pgsql-9.5/include/server/utils/relcache.h:98:13: note: declared here
extern void RelationSetNewRelfilenode(Relation relation, char persistence,
^
pg_btree.c:410:3: warning: implicit declaration of function '_bt_mergeload' [-Wimplicit-function-declaration]
_bt_mergeload(self, &wstate, btspool, &reader, heapRel);
^
pg_btree.c:417:3: warning: implicit declaration of function '_bt_load' [-Wimplicit-function-declaration]
_bt_load(&wstate, btspool, NULL);
^
pg_btree.c: At top level:
pg_btree.c:428:30: error: unknown type name 'BTWriteState'
_bt_mergeload(Spooler self, BTWriteState *wstate, BTSpool *btspool, BTReader *btspool2, Relation heapRel)
^
pg_btree.c:38:13: warning: 'unused_bt_spooldestroy' declared 'static' but never defined [-Wunused-function]
static void unused_bt_spooldestroy(BTSpool *);
^
pg_btree.c:39:13: warning: 'unused_bt_spool' declared 'static' but never defined [-Wunused-function]
static void unused_bt_spool(IndexTuple, BTSpool *);
^
pg_btree.c:40:13: warning: 'unused_bt_leafbuild' declared 'static' but never defined [-Wunused-function]
static void unused_bt_leafbuild(BTSpool *, BTSpool *);
^
pg_btree.c:626:1: warning: 'BTSpoolGetNextItem' defined but not used [-Wunused-function]
BTSpoolGetNextItem(BTSpool *spool, IndexTuple itup, bool *should_free)
^
pg_btree.c:794:1: warning: 'BTReaderGetNextItem' defined but not used [-Wunused-function]
BTReaderGetNextItem(BTReader *reader)
^
pg_btree.c:846:1: warning: 'compare_indextuple' defined but not used [-Wunused-function]
compare_indextuple(const IndexTuple itup1, const IndexTuple itup2,
^
pg_btree.c:912:1: warning: 'heap_is_visible' defined but not used [-Wunused-function]
heap_is_visible(Relation heapRel, ItemPointer htid)
^
pg_btree.c:926:1: warning: 'remove_duplicate' defined but not used [-Wunused-function]
remove_duplicate(Spooler *self, Relation heap, IndexTuple itup, const char *relname)
^
: recipe for target 'pg_btree.o' failed
make[2]: ** [pg_btree.o] Error 1
make[2]: *** Waiting for unfinished jobs....
reader.c: In function 'CheckerInit':
reader.c:433:16: error: 'RangeTblEntry {aka struct RangeTblEntry}' has no member named 'modifiedCols'
rte->modifiedCols = bms_add_member(rte->modifiedCols, i);
^
reader.c:433:51: error: 'RangeTblEntry {aka struct RangeTblEntry}' has no member named 'modifiedCols'
rte->modifiedCols = bms_add_member(rte->modifiedCols, i);
^
: recipe for target 'reader.o' failed
make[2]: *** [reader.o] Error 1
make[2]: Leaving directory '/home/devrim/Documents/Devrim/Projects/repo/pgrpms/rpm/redhat/9.5/pg_bulkload/F-22/pg_bulkload-VERSION3_1_8/lib'
Makefile:27: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/home/devrim/Documents/Devrim/Projects/repo/pgrpms/rpm/redhat/9.5/pg_bulkload/F-22/pg_bulkload-VERSION3_1_8'
error: Bad exit status from /var/tmp/rpm-tmp.9l7FIc (%build)

Hi,

I have few questions to ask.Can you help me with that?

-->Pg_bulkload is more like Sql loader in Oracle right? What if the table has indexes. Will it be rebuild after loading data as like oracle? Not sure, how this process works.
-->In pg_bulkloads, records will be inserted not updated. right?
Came across, few records got replaced. Is it because of ON_DUPLICATE_KEEP = NEW. Will that be be updated, or record will be removed and re-inserted. Kindly clarify that.

Thanks

what's new features going to add pg_bulkload?

Hi,

Getting this against PostgreSQL 9.6, which will be released on Thursday. Can we please get a new version that includes the fix soon?

Thanks!

+ make USE_PGXS=1 -j3
pg_btree.c:52:2: error: #error unsupported PostgreSQL version
 #error unsupported PostgreSQL version
  ^
pg_btree.c:107:42: error: unknown type name 'BTWriteState'
 static void _bt_mergeload(Spooler *self, BTWriteState *wstate, BTSpool *btspool,
                                          ^
pg_btree.c: In function 'IndexSpoolBegin':
pg_btree.c:207:33: error: 'BTREE_AM_OID' undeclared (first use in this function)
    indices[i]->rd_rel->relam == BTREE_AM_OID)
                                 ^
pg_btree.c:207:33: note: each undeclared identifier is reported only once for each function it appears in
pg_btree.c:222:13: error: dereferencing pointer to incomplete type
    spools[i]->isunique = indices[i]->rd_index->indisunique;
             ^
pg_btree.c: In function 'IndexSpoolInsert':
pg_btree.c:290:12: warning: variable 'heapRelation' set but not used [-Wunused-but-set-variable]
  Relation  heapRelation;
            ^
pg_btree.c: In function '_bt_mergebuild':
pg_btree.c:371:2: error: unknown type name 'BTWriteState'
  BTWriteState wstate;
  ^
pg_btree.c:377:31: error: dereferencing pointer to incomplete type
  tuplesort_performsort(btspool->sortstate);
                               ^
pg_btree.c:379:8: error: request for member 'index' in something not a structure or union
  wstate.index = btspool->index;
        ^
pg_btree.c:379:24: error: dereferencing pointer to incomplete type
  wstate.index = btspool->index;
                        ^
pg_btree.c:387:8: error: request for member 'btws_use_wal' in something not a structure or union
  wstate.btws_use_wal = self->use_wal &&
        ^
In file included from /usr/pgsql-9.6/include/server/storage/lmgr.h:20:0,
                 from pg_btree.c:21:
pg_btree.c:388:46: error: request for member 'index' in something not a structure or union
   XLogIsNeeded() && !RELATION_IS_LOCAL(wstate.index);
                                              ^
/usr/pgsql-9.6/include/server/utils/rel.h:479:4: note: in definition of macro 'RELATION_IS_LOCAL'
  ((relation)->rd_islocaltemp || \
    ^
pg_btree.c:388:46: error: request for member 'index' in something not a structure or union
   XLogIsNeeded() && !RELATION_IS_LOCAL(wstate.index);
                                              ^
/usr/pgsql-9.6/include/server/utils/rel.h:480:4: note: in definition of macro 'RELATION_IS_LOCAL'
   (relation)->rd_createSubid != InvalidSubTransactionId)
    ^
pg_btree.c:395:8: error: request for member 'btws_pages_alloced' in something not a structure or union
  wstate.btws_pages_alloced = BTREE_METAPAGE + 1;
        ^
pg_btree.c:396:8: error: request for member 'btws_pages_written' in something not a structure or union
  wstate.btws_pages_written = 0;
        ^
pg_btree.c:397:8: error: request for member 'btws_zeropage' in something not a structure or union
  wstate.btws_zeropage = NULL; /* until needed */
        ^
pg_btree.c:404:21: error: request for member 'index' in something not a structure or union
  LockRelation(wstate.index, AccessExclusiveLock);
                     ^
pg_btree.c:405:29: error: request for member 'index' in something not a structure or union
  FlushRelationBuffers(wstate.index);
                             ^
pg_btree.c:408:38: error: request for member 'index' in something not a structure or union
  merge = BTReaderInit(&reader, wstate.index);
                                      ^
In file included from /usr/pgsql-9.6/include/server/postgres.h:48:0,
                 from ../include/pg_bulkload.h:9,
                 from pg_btree.c:11:
pg_btree.c:411:33: error: request for member 'index' in something not a structure or union
   RelationGetRelationName(wstate.index),
                                 ^
/usr/pgsql-9.6/include/server/utils/elog.h:202:23: note: in definition of macro 'elog'
   elog_finish(elevel, __VA_ARGS__); \
/usr/pgsql-9.6/include/server/utils/rel.h:392:3: note: in expansion of macro  NameStr'
  (NameStr((relation)->rd_rel->relname))
   ^
pg_btree.c:411:3: note: in expansion of macro 'RelationGetRelationName'
   RelationGetRelationName(wstate.index),
   ^
pg_btree.c:413:9: error: request for member 'btws_use_wal' in something not a structure or union
   wstate.btws_use_wal ? "with" : "without");
         ^
/usr/pgsql-9.6/include/server/utils/elog.h:202:23: note: in definition of macro 'elog'
   elog_finish(elevel, __VA_ARGS__); \
                       ^
pg_btree.c:416:34: error: request for member 'index' in something not a structure or union
  RelationSetNewRelfilenode(wstate.index, InvalidTransactionId);
                                  ^
pg_btree.c:416:2: error: too few arguments to function 'RelationSetNewRelfilenode'
  RelationSetNewRelfilenode(wstate.index, InvalidTransactionId);
  ^
In file included from /usr/pgsql-9.6/include/server/access/genam.h:21:0,
                 from pg_btree.c:13:
/usr/pgsql-9.6/include/server/utils/relcache.h:99:13: note: declared here
 extern void RelationSetNewRelfilenode(Relation relation, char persistence,
             ^
pg_btree.c:418:23: error: dereferencing pointer to incomplete type
  if (merge || (btspool->isunique && self->max_dup_errors > 0))
                       ^
pg_btree.c:422:3: warning: implicit declaration of function '_bt_mergeload' [-Wimplicit-function-declaration]
   _bt_mergeload(self, &wstate, btspool, &reader, heapRel);
   ^
pg_btree.c:429:3: warning: implicit declaration of function '_bt_load' [-Wimplicit-function-declaration]
   _bt_load(&wstate, btspool, NULL);
   ^
pg_btree.c: At top level:
pg_btree.c:440:30: error: unknown type name 'BTWriteState'
 _bt_mergeload(Spooler *self, BTWriteState *wstate, BTSpool *btspool, BTReader *btspool2, Relation heapRel)
                              ^
pg_btree.c: In function 'BTSpoolGetNextItem':
pg_btree.c:642:38: error: dereferencing pointer to incomplete type
  return tuplesort_getindextuple(spool->sortstate, true, should_free);
                                      ^
pg_btree.c: At top level:
pg_btree.c:38:13: warning: 'unused_bt_spooldestroy' declared 'static' but never defined [-Wunused-function]
 static void unused_bt_spooldestroy(BTSpool *);
             ^
pg_btree.c:40:13: warning: 'unused_bt_spool' declared 'static' but never defined [-Wunused-function]
 static void unused_bt_spool(BTSpool *, ItemPointer self,
             ^
pg_btree.c:45:13: warning: 'unused_bt_leafbuild' declared 'static' but never defined [-Wunused-function]
 static void unused_bt_leafbuild(BTSpool *, BTSpool *);
             ^
pg_btree.c:638:1: warning: 'BTSpoolGetNextItem' defined but not used [-Wunused-function]
 BTSpoolGetNextItem(BTSpool *spool, IndexTuple itup, bool *should_free)
 ^
pg_btree.c:806:1: warning: 'BTReaderGetNextItem' defined but not used [-Wunused-function]
 BTReaderGetNextItem(BTReader *reader)
 ^
pg_btree.c:858:1: warning: 'compare_indextuple' defined but not used [-Wunused-function]
 compare_indextuple(const IndexTuple itup1, const IndexTuple itup2,
 ^
pg_btree.c:924:1: warning: 'heap_is_visible' defined but not used [-Wunused-function]
 heap_is_visible(Relation heapRel, ItemPointer htid)
 ^
pg_btree.c:938:1: warning: 'remove_duplicate' defined but not used [-Wunused-function]
 remove_duplicate(Spooler *self, Relation heap, IndexTuple itup, const char *relname)
 ^
make[2]: *** [pg_btree.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.bRzoFr (%build)
    Bad exit status from /var/tmp/rpm-tmp.bRzoFr (%build)
make: *** [build] Error 1

I've set my postgresql 10 to 'without the need for a password' mode ,and can log on to the DB only with 'psql -d posteva -h localhost -p 5432 -U postgres' command. However, when I use pg_bulkloader(VERSION 3.1.14) with the command 'pg_bulkload -d posteva -h localhost -p 5432 -U postgres -w ./bulkload_raw_uemr.ctl', an error occured like this.

PS: When I use VERSION 3.1.13 with the same cmd on PG9.6, there's nothing wrong.
Is there any problem with my way of using new version pg_bulkload?

ERROR: query failed: ERROR: could not establish connection to parallel writer
DETAIL: could not translate host name "." to address: Name or service not known

HINT: Refer to the following if it is an authentication error. Specifies the authentication method to without the need for a password in pg_hba.conf (ex. trust or ident), or specify the password to the password file of the operating system user who ran PostgreSQL server. If cannot use these solution, specify WRITER=DIRECT.
DETAIL: query was: SELECT * FROM pg_bulkload($1)

Is there any package available for installing pg_backload to Ubuntu 16.04? From the documentation I only found RHL package. The option I found is only generating build from source. Please replay if anyone knows better way than source installation. Thanks!

I'm having trouble building on OpenBSD 6.1

When I try using OpenBSD make:

[jasper@puffy:~/pg_bulkload-VERSION3_1_13]$make
*** Parse error in /home/jasper/pg_bulkload-VERSION3_1_13: Need an operator in 'USE_PGXS' (Makefile:6)
*** Parse error: Need an operator in '""' (Makefile:9)
*** Parse error: Need an operator in 'endif' (Makefile:11)
*** Parse error: Need an operator in 'endif' (Makefile:12)
*** Parse error: Need an operator in 'USE_PGXS' (Makefile:14)
*** Parse error: Filename missing from "include" (Makefile:17)
*** Parse error: Need an operator in 'else' (Makefile:18)
*** Parse error: Could not find ../../src/Makefile.global (Makefile:20)
*** Parse error: Could not find /contrib/contrib-global.mk (Makefile:21)
*** Parse error: Need an operator in 'endif' (Makefile:22)

When I try using GNU make:

[jasper@puffy:~/pg_bulkload-VERSION3_1_13]$gmake
gmake[1]: Entering directory '/home/jasper/pg_bulkload-VERSION3_1_13/bin'
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pg_bulkload.o pg_bulkload.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o recovery.o recovery.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pgut/pgut.o pgut/pgut.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pgut/pgut-fe.o pgut/pgut-fe.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pgut/pgut-list.o pgut/pgut-list.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o -L/usr/local/lib -lpq -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -L/usr/lib -L/usr/local/lib -L/usr/local/lib  -L/usr/local/lib -Wl,-Bdynamic  -lpgcommon -lpgport -lssl -lcrypto -lz -lreadline -ltermcap -lm -o pg_bulkload
/usr/local/lib/libpq.so.6.9: warning: warning: strcpy() is almost always misused, please use strlcpy()
/usr/local/lib/libpq.so.6.9: warning: warning: sprintf() is often misused, please use snprintf()
/usr/local/lib/libpq.so.6.9: warning: warning: strcat() is almost always misused, please use strlcat()
gmake[1]: Leaving directory '/home/jasper/pg_bulkload-VERSION3_1_13/bin'
gmake[1]: Entering directory '/home/jasper/pg_bulkload-VERSION3_1_13/lib'
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o binary.o binary.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o logger.o logger.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o parser_binary.o parser_binary.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o parser_csv.o parser_csv.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o parser_function.o parser_function.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o parser_tuple.o parser_tuple.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pg_btree.o pg_btree.c
nbtree/nbtsort-9.6.c:181: warning: 'unused_bt_spooldestroy' defined but not used
nbtree/nbtsort-9.6.c:191: warning: 'unused_bt_spool' defined but not used
nbtree/nbtsort-9.6.c:202: warning: 'unused_bt_leafbuild' defined but not used
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pg_bulkload.o pg_bulkload.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o pg_strutil.o pg_strutil.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o reader.o reader.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o source.o source.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o writer.o writer.c
cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -pipe -fpic -DPIC -I../include -I/usr/local/include/postgresql -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/local/include   -c -o writer_binary.o writer_binary.c
writer_binary.c: In function 'open_output_file':
writer_binary.c:421: error: 'S_IRUSR' undeclared (first use in this function)
writer_binary.c:421: error: (Each undeclared identifier is reported only once
writer_binary.c:421: error: for each function it appears in.)
writer_binary.c:421: error: 'S_IWUSR' undeclared (first use in this function)
gmake[1]: *** [<builtin>: writer_binary.o] Error 1
gmake[1]: Leaving directory '/home/jasper/pg_bulkload-VERSION3_1_13/lib'
gmake: *** [Makefile:27: all] Error 2

OpenBSD comes with a very low GCC version number, so sometimes I have to build with clang:

[jasper@puffy:~/pg_bulkload-VERSION3_1_13]$cc --version
cc (GCC) 4.2.1 20070719
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[jasper@puffy:~/pg_bulkload-VERSION3_1_13]$clang --version
clang version 4.0.0 (tags/RELEASE_400/final)
Target: amd64-unknown-openbsd6.1
Thread model: posix
InstalledDir: /usr/local/bin

But running CC=clang CXX=clang++ gmake didn't help. Any suggestions?

Thanks @amitlan for quick fix for build break.

But, the regression tests are not passed...

I have found the recent postgres commit postgres/postgres@b8d7f05 which changes ExecQual.

Though I have modified lib/pg_btree.c similar to src/backend/executor/execIndexing.c as below, the tests are still not passed.

index 5b1ab48..0b61891 100644
--- a/lib/pg_btree.c
+++ b/lib/pg_btree.c
@@ -334,21 +334,21 @@ IndexSpoolInsert(BTSpool **spools, TupleTableSlot *slot, ItemPointer tupleid, 
                /* Check for partial index */
                if (indexInfo->ii_Predicate != NIL)
                {
-                       List               *predicate;
+                       ExprState          *predicate;
 
                        /*
                         * If predicate state not set up yet, create it (in the estate's
                         * per-query context)
                         */
                        predicate = indexInfo->ii_PredicateState;
-                       if (predicate == NIL)
+                       if (predicate == NULL)
                        {
-                               predicate = (List *) ExecPrepareExpr((Expr *) indexInfo->ii_Predicat
+                               predicate = ExecPrepareQual(indexInfo->ii_Predicate, estate);
                                indexInfo->ii_PredicateState = predicate;
                        }
 
                        /* Skip this index-update if the predicate isn'loader satisfied */
-                       if (!ExecQual(predicate, econtext, false))
+                       if (!ExecQual(predicate, econtext))
                                continue;
                }

Hi,

I have a CSV file with HEADER. There is no reference within documentation about how to indicate if the CSV file do contains or doesn't contains a HEADER. It seems like a CSV file can't contain a HEADER.

Please, how could I take on that..

Will the pg_bulkload supports the Postgres Enterprise Edition(EDB)? I want to try that in EDB.

Thanks

Hi, I'm using postgres 9.4 and pg_bulkload-master.
After running installcheck that's what I obtain:
postgres@burriac:/home/pbuenaposada/Escriptori/pg_bulkload-master$ make installcheck
make[1]: Entering directory `/home/pbuenaposada/Escriptori/pg_bulkload-master/bin'
/usr/lib/postgresql/9.4/lib/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --psqldir='/usr/lib/postgresql/9.4/bin' --dbname=contrib_regression --encoding=UTF8 init load_bin load_csv load_remote load_function load_encoding load_check load_filter load_parallel write_bin
(using postmaster on Unix socket, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test init ... ok
test load_bin ... FAILED
test load_csv ... FAILED
test load_remote ... FAILED
test load_function ... FAILED
test load_encoding ... FAILED
test load_check ... FAILED
test load_filter ... FAILED
test load_parallel ... FAILED
test write_bin ... FAILED

9 of 10 tests failed.

The differences that caused some tests to fail can be viewed in the
file "/home/pbuenaposada/Escriptori/pg_bulkload-master/bin/regression.diffs". A copy of the test summary that you see above is saved in the file "/home/pbuenaposada/Escriptori/pg_bulkload-master/bin/regression.out".

make[1]: *** [installcheck] Error 1
make[1]: Leaving directory /home/pbuenaposada/Escriptori/pg_bulkload-master/bin' make[1]: Entering directory/home/pbuenaposada/Escriptori/pg_bulkload-master/lib'
make[1]: Nothing to be done for installcheck'. make[1]: Leaving directory/home/pbuenaposada/Escriptori/pg_bulkload-master/lib'
make[1]: Entering directory /home/pbuenaposada/Escriptori/pg_bulkload-master/util' make[1]: Nothing to be done forinstallcheck'.
make[1]: Leaving directory `/home/pbuenaposada/Escriptori/pg_bulkload-master/util'
make: *** [installcheck] Error 2

the regression.diffs contains (only a part, is too big...):

*** /home/pbuenaposada/Escriptori/pg_bulkload-master/bin/expected/load_bin.out 2015-08-04 03:36:35.000000000 +0200
--- /home/pbuenaposada/Escriptori/pg_bulkload-master/bin/results/load_bin.out 2015-10-27 10:41:45.175456930 +0100

*** 1,28 ****
TRUNCATE customer;
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_logfile -l infile_logfile -P pbfile -u dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_pbfile -l logfile -P infile_pbfile -u dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_dbfile -l logfile -P pbfile -u infile_dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile_pbfile -P logfile_pbfile -u dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile_dbfile -P pbfile -u logfile_dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile -P pbfile_dbfile -u pbfile_dbfile
! NOTICE: BULK LOAD START
! ERROR: query failed: ERROR: INPUT, PARSE_BADFILE, DUPLICATE_BADFILE and LOGFILE cannot set the same file name.
! DETAIL: query was: SELECT * FROM pg_bulkload($1)
TRUNCATE customer;
SET enable_seqscan = off;
SET enable_indexscan = on;
--- 1,16 ----
TRUNCATE customer;
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_logfile -l infile_logfile -P pbfile -u dbfile
! sh: 1: pg_bulkload: not found
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_pbfile -l logfile -P infile_pbfile -u dbfile
! sh: 1: pg_bulkload: not found
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile_dbfile -l logfile -P pbfile -u infile_dbfile
! sh: 1: pg_bulkload: not found
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile_pbfile -P logfile_pbfile -u dbfile
! sh: 1: pg_bulkload: not found
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile_dbfile -P pbfile -u logfile_dbfile
! sh: 1: pg_bulkload: not found
! pg_bulkload -d contrib_regression data/bin1.ctl -i infile -l logfile -P pbfile_dbfile -u pbfile_dbfile
! sh: 1: pg_bulkload: not found
TRUNCATE customer;
SET enable_seqscan = off;
SET enable_indexscan = on;

*** 33,1201 ****
(0 rows)

! pg_bulkload -d contrib_regression data/bin1.ctl -i data/data1.bin -l results/bin1.log -P results/bin1.prs -u results/bin1.dup
! NOTICE: BULK LOAD START
! NOTICE: BULK LOAD END
! 1 Rows skipped.
! 8 Rows successfully loaded.
! 0 Rows not loaded due to parse errors.
! 0 Rows not loaded due to duplicate errors.
! 0 Rows replaced with new rows.
! awk -f data/adjust.awk results/bin1.log
!
! pg_bulkload 3.2devel on
!
! INPUT = .../data1.bin
! PARSE_BADFILE = .../bin1.prs
! LOGFILE = .../bin1.log
! LIMIT = 8
! PARSE_ERRORS = 0
! CHECK_CONSTRAINTS = NO
! TYPE = BINARY
! SKIP = 1
! STRIDE = 232
! COL = CHAR (10)
! COL = CHAR (4)
! COL = CHAR (1)
! COL = CHAR (16)
! COL = CHAR (2)
! COL = CHAR (16)
! COL = CHAR (70 + 20)
! COL = VARCHAR (50 + 20)
! COL = CHAR (90 + 20)
! COL = CHAR (2)
! COL = CHAR (9)
! COL = CHAR (16)
! COL = CHAR (19)
! COL = CHAR (2)
! COL = UNSIGNED SMALLINT (2)
! COL = INTEGER (4)
! COL = BIGINT (8)
! COL = FLOAT (4)
! COL = DOUBLE (8)
! COL = CHAR (10)
! COL = CHAR (39)
! OUTPUT = public.customer
! MULTI_PROCESS = YES
! VERBOSE = NO
! WRITER = DIRECT
! DUPLICATE_BADFILE = .../bin1.dup
! DUPLICATE_ERRORS = 0
! ON_DUPLICATE_KEEP = NEW
! TRUNCATE = NO
!
!
! 1 Rows skipped.
! 8 Rows successfully loaded.
! 0 Rows not loaded due to parse errors.
! 0 Rows not loaded due to duplicate errors.
! 0 Rows replaced with new rows.
!
! Run began on
! Run ended on
!
! CPU s/u sec elapsed sec
SET enable_seqscan = on;
SET enable_indexscan = off;
SET enable_bitmapscan = off;
SELECT * FROM customer ORDER BY c_id;
! c_id | c_d_id | c_w_id | c_first | c_middle | c_last | c_street_1 | c_street_2 | c_city | c_state | c_zip | c_phone | c_since | c_credit | c_credit_lim | c_discount | c_balance | c_ytd_payment | c_payment_cnt | c_delivery_cnt | c_data
! ----------+--------+--------+---------+----------+------------------+------------+----------------------+----------------------+---------+-----------+------------------+--------------------------+----------+--------------+------------+-----------+---------------+---------------+----------------+-----------------------------------------
! 16777217 | 217 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777218 | 218 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777219 | 219 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777220 | 220 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777221 | 221 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777222 | 222 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777223 | 223 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777224 | 224 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! (8 rows)

SET enable_seqscan = off;
SET enable_indexscan = on;
SET enable_bitmapscan = off;
SELECT * FROM customer ORDER BY c_id;
! c_id | c_d_id | c_w_id | c_first | c_middle | c_last | c_street_1 | c_street_2 | c_city | c_state | c_zip | c_phone | c_since | c_credit | c_credit_lim | c_discount | c_balance | c_ytd_payment | c_payment_cnt | c_delivery_cnt | c_data
! ----------+--------+--------+---------+----------+------------------+------------+----------------------+----------------------+---------+-----------+------------------+--------------------------+----------+--------------+------------+-----------+---------------+---------------+----------------+-----------------------------------------
! 16777217 | 217 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777218 | 218 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777219 | 219 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777220 | 220 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777221 | 221 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 | AAAAAAAAAAAAAAAAAAAA | AA | AAAAAAAAA | AAAAAAAAAAAAAAAA | Sun Jan 01 12:34:56 2006 | AA | 32768 | 12345 | 12345 | 12345.7 | 12345.7 | 12345.6789 | 123456789012345678901234567890123456789
! 16777222 | 222 | 0 | ABCDEFG | AA | AAAAAAAAAAAAAAAA | c_street_2 | c_street_1 |

In direct mode, if i don't use postgresql script, what result will be ? @amitlan

Hi, I cannot build in a docker container against postgres 10.1, I receive the error below. It seems some packages are missing, but I have installed everything stated in installation

My docker file is Dockerfile.postgres.txt

Thank you for your help

Step 4/4 : RUN cd pg_bulkload && make
---> Running in c898b35fc114
make[1]: Entering directory '/pg_bulkload/bin'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c -o pg_bulkload.o pg_bulkload.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c -o recovery.o recovery.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c -o pgut/pgut.o pgut/pgut.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c -o pgut/pgut-fe.o pgut/pgut-fe.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -I../include -I/usr/include/postgresql -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c -o pgut/pgut-list.o pgut/pgut-list.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-YdsZNw/postgresql-10-10.1=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer pg_bulkload.o recovery.o pgut/pgut.o pgut/pgut-fe.o pgut/pgut-list.o -L/usr/lib/x86_64-linux-gnu -lpq -L/usr/lib/x86_64-linux-gnu -Wl,-z,relro -Wl,-z,now -L/usr/lib/x86_64-linux-gnu/mit-krb5 -Wl,--as-needed -Wl,--build-id -lpgcommon -lpgport -lpthread -lselinux -lpam -lssl -lcrypto -lgssapi_krb5 -lz -ledit -lrt -lcrypt -ldl -lm -o pg_bulkload
/usr/bin/ld: cannot find -lselinux
/usr/bin/ld: cannot find -lssl
/usr/bin/ld: cannot find -lcrypto
collect2: error: ld returned 1 exit status
make[1]: *** [pg_bulkload] Error 1
/usr/lib/postgresql/10/lib/pgxs/src/makefiles/pgxs.mk:300: recipe for target 'pg_bulkload' failed
make[1]: Leaving directory '/pg_bulkload/bin'
Makefile:27: recipe for target 'all' failed
make: *** [all] Error 2

I just noticed that if a target table has check constraints, they are not checked by pg_bulkload by default (although NOT NULL constraints of individual columns are). Should that be reverse?

We recently could get rid of the gh-pages branch in the pg_rman repository. The requirement for the new GitHub pages feature was to have a doc/index.html in the master branch. We do have for one in the pg_bulkload repository too, but we don't update it when we make releases. Rather, we update pg_bulkload.html which is linked from index.html. I just noticed that we forgot to mention information about releases 3.1.10 onward, so pushed a fix to the copy of pg_bulkload.html in the master and VERSION3_1_STABLE branches, which didn't result in updating what I can read here. Pushing the same fix in the gh-pages branch worked.

I'm wondering which files are picked up by ossc-db.github.io/pg_bulkload/? To be specific, which branch of the repository. I mean, since we have doc/index.html in the master branch, and if that's what is being used, then why isn't pg_bulkload.html in the same branch used to link to?

Hi,

I have a source file with following order without header.

COLA;COLB;COLC
1;2;3

Table column order:

COLB
COLA
COLC

Here COLA should be mapped to COLA, COLB should be mapped to COLB and COLC should be mapped to COLC.

This case can be accomplished without FILTER option? Why because, we are not defining anywhere, which column to load.

Thanks

I'm using pg_bulkload to import a bunch of large (3-5 million row x 4 column) CSV files into a PostgreSQL database. The first two import very quickly and without issue, but subsequent ones fail with the following error:

ERROR: copy failed: ERROR:  faild in tuplesort_performsort

For reference, my control file looks like this:

OUTPUT = spectra_data
TYPE = CSV
DELIMITER = ","

...and the command I'm running looks like this.

pg_bulkload -d leaf_spectra control.ctl -i stdin < data.csv

(Actually, the command and files are generated automatically by an R system call...but that's effectively what it is).

The table I'm trying to import into looks like this:

                                  Table "public.spectra_data"                                                                                                         [0/1862]
    Column     |  Type   |                              Modifiers
---------------+---------+----------------------------------------------------------------------
 spectradataid | bigint  | not null default nextval('spectra_data_spectradataid_seq'::regclass)
 spectraid     | bigint  | not null
 wavelength    | numeric |
 spectravalue  | numeric |
Indexes:
    "spectra_data_pkey" PRIMARY KEY, btree (spectradataid)
Check constraints:
    "legal_wavelength" CHECK (wavelength > 0::numeric)
Foreign-key constraints:
    "spectra_data_spectraid_fkey" FOREIGN KEY (spectraid) REFERENCES spectra_info(spectraid) ON DELETE CASCADE

Any idea why?

FYI, I'm running Postgres version 9.6.1 (latest) on Arch Linux, with an install of pg_bulkload from the master branch earlier today (Feb 7, 2017).

I tried building v. 3.19 on Windows (10 x64) for PostgreSQL 9.5.2 (compiled by Visual C++ build 1800, 64-bit) in the following env:

1. MSVC 2013 (no Win SDK 7.1 installed, I change Build Toolset to v120)
2. MSVC 2010 (no Win SDK 7.1 installed, I change Build Toolset to v100)
3. MSYS2+mingw64

There is one common error I keep getting on all 3 of them:

binary.c:423:1: warning: no previous prototype for 'BinaryParam' [-Wmissing-prototypes]
 BinaryParam(Field **fields, int *nfield, char *value, bool preserve_blanks, bool length_only)
 ^
binary.c:423:1: error: 'BinaryParam' redeclared as different kind of symbol
In file included from D:/Programs/Dev/Env/msys64/mingw64/x86_64-w64-mingw32/include/rpc.h:104:0,
                 from D:/Programs/Dev/Env/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:88,
                 from D:/Programs/Dev/Env/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23,
                 from D:/Programs/Dev/Env/msys64/mingw64/include/postgresql/server/pg_config_os.h:37,
                 from D:/Programs/Dev/Env/msys64/mingw64/include/postgresql/server/c.h:101,
                 from D:/Programs/Dev/Env/msys64/mingw64/include/postgresql/server/postgres.h:47,
                 from ../include/pg_bulkload.h:9,
                 from binary.c:11:
D:/Programs/Dev/Env/msys64/mingw64/x86_64-w64-mingw32/include/rpcasync.h:88:5: note: previous declaration of 'BinaryParam' was here
   } BinaryParam;
     ^
<builtin>: recipe for target 'binary.o' failed
mingw32-make[1]: *** [binary.o] Error 1
mingw32-make[1]: Leaving directory 'D:/Programs/Dev/Env/msys64/home/nick/pg_bulkload_319/lib'

When trying to build on MSVC I get that and a lot of syntax errors pointing to this line of logger.h code:

__attribute__((format(PG_BULKLOAD_PRINTF_ATTRIBUTE, 2, 3)));

I can attach any build log if it is required, but I think I'm missing something really simple.

Hello
we are trying to replace oracle (sqlldr )

is it possible that the csv has 4 columns an I can import them in a table with only 3 columns? the last column would be ignored

we just upgraded to 9.5 as well so we a 9.5 package would be great

Hi,

I need to pass the Filename, File Path, Host, Port, Database, Username, Schema, Password as arguments, something like below;

pg_bulkload filename.ctl -h $1 -U $2 -d $3 -p $4 $5 $6

$5 denotes File Path
$6 denotes File Name

These parameters will be passed by scheduler, during run time. Kindly help me, how this can be done?

Thanks

I was just installing pg_bulkload and would like to suggest adding the dependencies for popular distributions in the install section.
To install on ubuntu I needed build-essential, git, postgresql-server-dev-x.x, libpam0g-dev, libedit-dev.

Hello, since some weeks ago (maybe data log increased), we notice sometime an "out of memory" error in pg_bulkload (3.1.9 version) when we use, as input, a postgresql function
pg_bulkload 3.1.9 on 2016-10-04 09:18:23.033388+02

INPUT = pg_bulkload_function_name()
WRITER = DIRECT
..........
out of memory

When loading other types of data (until 10 millions of row with a CSV file), everything works.
It seems a memory leak when using this function...
Someone has experienced this problem?

Thank you, best regards!
Francesco

I'm getting this error when I try to run pg_bulkloader,

ERROR: query failed: ERROR:  shmget(id=1949974951) failed: Invalid argument
DETAIL: query was: SELECT * FROM pg_bulkload($1)

And when I try to run the tests (make installcheck), I see the same error in regressions.diff. I'm using postgres 9.4.3.