pgq / skytools-legacy

Obsolete, see https://github.com/pgq/ for maintained code.

Generic Queue for PostgreSQL

PgQ is PostgreSQL extension that provides generic, high-performance lockless queue with simple API based on SQL functions.

• Supported PostgreSQL versions: 10..16
• Deprecated but might work: 9.x

skytools-legacy's Issues

Can I use skytools for postgres 8.4?

Can I use skytools for postgres 8.4?

How to remove/delete events? event_failed() function is missing.

Hello,

there was a function event_failed() in PgQ/Skytools2 but got dropped in version 3 (but is still documented in /doc/pgq-sql.txt). So how to remove events? For example i'm trying to process an event 3 times. After that i would like to completely remove/delete the event. I did this before with the event_failed() function.

I'm using PgQ version: 3.1.3

Thanks,
Patryk

KeyError with CoopConsumer and pgq_lazy_fetch=0

I get a
Traceback (most recent call last):
File "/home/avtech/skytools/python/skytools/scripting.py", line 568, in run_func_safely
return func()
File "/home/avtech/skytools/python/pgq/baseconsumer.py", line 260, in work
self._finish_batch(curs, batch_id, ev_list)
File "/home/avtech/skytools/python/pgq/coopconsumer.py", line 76, in _finish_batch
self._flush_retry(curs, batch_id, list)
File "/home/avtech/skytools/python/pgq/consumer.py", line 76, in _flush_retry
if ev._status == EV_RETRY:
File "/home/avtech/skytools/python/pgq/event.py", line 49, in getattr
return self._event_row[_fldmap[key]]
KeyError: '_status'
when running a CoopConsumer with pgq_lazy_fetch=0.

I believe that the cause of the problem is that pgq.baseconsumer.py:_load_batch_events_old() instantiates events directly using the Event-constructor, but CoopConsumer expects "RetriableEvent"s instead. That is, the Event-instances do not have a _status-field whereas RetriableEvents do.

When pgq_lazy_fetch > 1 the events are instantiated by the _batch_walker_class-field in _load_batch_events() so there is no crash (because CoopConsumer supplies its own BatchWalker which creates RetriableEvents).

Maybe events should always be created by some function _make_event that subclasses can override?

londiste3 should complain if triggers are not in place

Should londiste complain loudly if the triggers are not in place but the table is added locally? I was trying out the simple example using pgbench. pgbench will drop the tables and recreate them which removes the triggers. londiste however did not complain in any way. 'londiste3 $ini tables' shows the merge_state as 'ok' even if there are no triggers in place... seems like a dangerous situation. Just thinking out load here.

simple londiste3 replication standby server error

I'm trying to configure simple replication and my standby server log:

2012-11-12 11:12:17,372 4435 ERROR Job st3_l3simple_db2 crashed: format requires a mapping
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/skytools/scripting.py", line 563, in run_func_safely
return func()
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 199, in work
return Consumer.work(self)
File "/usr/local/lib/python2.7/dist-packages/pgq/consumer.py", line 280, in work
self._launch_process_batch(db, batch_id, ev_list)
File "/usr/local/lib/python2.7/dist-packages/pgq/consumer.py", line 309, in _launch_process_batch
self.process_batch(db, batch_id, list)
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 172, in process_batch
self.process_remote_batch(src_db, tick_id, event_list, dst_db)
File "/usr/local/lib/python2.7/dist-packages/londiste/playback.py", line 363, in process_remote_batch
self.restore_fkeys(dst_db)
File "/usr/local/lib/python2.7/dist-packages/londiste/playback.py", line 926, in restore_fkeys
self.log.info('Creating fkey: %(fkey_name)s (%(from_table)s --> %(to_table)s)' % row)
TypeError: format requires a mapping

londiste doesn't copy all significant digits with double presicion data type

I'm working on a server migration with londiste. We've been having trouble with a table which has a unique constraint that always failed on getting applied. It's erroring about duplicate values which are not on the provider node.

Taking a pg_dump of the table in question showed that the tuples unique key were different on the provider.

Playing around I found that londiste uses the default value for extra_float_digits which is zero. This means some significant values for float fields get lost. This is very bad, as people would expect to have the same exact data copied over as what they have on the provider node.

Workaround is to just change the connection string on the root node (or on all nodes) so it sets extra_float_digits to 3. Just add something like option='-c extra_float_digits=3'

node 2 worker crashing - baseconsumer getting 4 args instead of 3.

Hi there,

I've pasted the trace back below, but basically the worker on the clone continually dumping this error:

2013-01-24 12:01:31,047 20806 INFO {idle: 301.2509}
2013-01-24 12:01:31,059 20806 ERROR Job 83_92_sa_replication_clone crashed: init() takes exactly 3 arguments (4 given)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/skytools/scripting.py", line 567, in run_func_safely
return func()
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 199, in work
return BaseConsumer.work(self)
File "/usr/local/lib/python2.7/dist-packages/pgq/baseconsumer.py", line 256, in work
self._launch_process_batch(db, batch_id, ev_list)
File "/usr/local/lib/python2.7/dist-packages/pgq/baseconsumer.py", line 285, in _launch_process_batch
self.process_batch(db, batch_id, list)
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 172, in process_batch
self.process_remote_batch(src_db, tick_id, event_list, dst_db)
File "/usr/local/lib/python2.7/dist-packages/londiste/playback.py", line 375, in process_remote_batch
CascadedWorker.process_remote_batch(self, src_db, tick_id, ev_list, dst_db)
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/worker.py", line 149, in process_remote_batch
for ev in event_list:
File "/usr/local/lib/python2.7/dist-packages/pgq/baseconsumer.py", line 56, in iter
ev = self._event_class(self, self.queue_name, row)
TypeError: init() takes exactly 3 arguments (4 given)

role pgq_admin cannot bulk_insert_events

I tried to use bulk_insert_events from the pgq library, but it does not work out of the box, even after applying the newgrants_pgq.sql permissions.

Since it is special command, I assume that only pgq_admin should be allowed to use it (or maybe pgq_writer as well). It can serve for writing tests for example, or in some special cases.

I worked around the issue by assigning additional grants to pgq_admin:

\i /usr/share/postgresql/9.3/contrib/newgrants_pgq.sql

GRANT INSERT ON pgq.event_template TO pgq_admin;
ALTER DEFAULT PRIVILEGES IN SCHEMA pgq GRANT USAGE ON SEQUENCES TO pgq_admin;

PK update

Hi,

there is any reason to block primary key updates?

Make possible to write on branch node

I'm not sure if is a correct approach to write at branch node, but makes sense for my use case:

I have an local instance that is my master and a web instance that is the queue consumer, which should be able to write rows that don't need to be replicated to anyone. All conflicts are already treated on application logic layer.

I have the replication configured and working, but when I write at branch node i get the following message:

ERROR:  Insert into queue disallowed
CONTEXT:  PL/pgSQL function pgq.insert_event(text,text,text,text,text,text,text) line 24 at RETURN
SQL statement "select pgq.insert_event($1, $2, $3, $4, $5, $6, $7)"

Seems an expected behavior, however I didn't expect that branch node would try to insert something into queue.

Shouldn't this be optional for a branch or leaf node?

---

master.ini

[londiste3]
db = user=postgres host=127.0.0.1 dbname=postgres
queue_name = myappq
loop_delay = 0.5
logfile = log/master.log
pidfile = pid/master.pid

slave.ini

[londiste3]
db = user=postgres host=127.0.0.1 dbname=londiste
queue_name = myappq
loop_delay = 0.5
logfile = log/slave.log
pidfile = pid/slave.pid

ticker.ini

[pgqd]
base_connstr = user=postgres host=127.0.0.1
database_list = postgres
logfile = log/ticker.log
pidfile = pid/ticker.pid

londiste3 can not load dispatch handler under python 2.4 (RHEL5)

The londiste3 dispatch handler can not be loaded because the handler makes use of functools.partial which was introduced in 2.5. Unfortunately we're still on RHEL5 here which is still on python 2.4. Note that I'm not using this handler but I do get warned continually.

What is the lowest version of python supported for londiste3?

creating leaf sets node location incorrectly

when replicating between 2 boxen, passing a provider connection string that has more than just the dbname, fails to register correctly.

For eg on node2:

londiste3 someini.ini create-leaf node2 dbname=my_clone --provider "dbname=my_master host=db-master user=just_me"

creates a node_location entry on my_clone.pgq_node.node_location for node1 of "dbname=my_master".

Similarly (and I'm not sure at what point) node1 has a node2 entry for node_location of "dbname=my_clone".

In order to get things working, I just manually add a step and update the node_location column with correct db connection details.

I'm happy to drill in a bit and see if I can fix this (good chance to learn some python) and shoot you a pull request, however can you confirm my understanding of how this should work is correct?

PgQ python framework can not be used anymore with python2.4

Hello,

Since version 3.1.5, PgQ python framework can not be used anymore with python2.4.
The use of finally instruction in admin.py file is blocking.

-bash-3.2$ ./QMover_axfrak4_pgq_emv_newtrans_axis2proxy_QA_RHA.py QMOVER.ini --stop
Traceback (most recent call last):
File "./QMover_axfrak4_pgq_emv_newtrans_axis2proxy_QA_RHA.py", line 11, in ?
from qmover import QueueMover
File "/var/lib/pgsql/9.2/data/PgQ_ReplicationRHA_RC6/qmover.py", line 27, in ?
import pgq
File "/usr/lib64/python2.4/site-packages/pgq/init.py", line 14, in ?
import pgq.cascade.admin
File "/usr/lib64/python2.4/site-packages/pgq/cascade/admin.py", line 495
finally:
^
SyntaxError: invalid syntax

In order to keep compatibility, is it possible to use the following code in python/pgq/cascade/admin.py :

def load_node_status (self, name, location):
""" Load node info & status """
# must be thread-safe (!)
if not self.node_alive(name):
node = NodeInfo(self.queue_name, None, node_name = name)
return node
try:
db = None
db = skytools.connect_database (location)
db.set_isolation_level (skytools.I_AUTOCOMMIT)
curs = db.cursor()
curs.execute("select * from pgq_node.get_node_info(%s)", [self.queue_name])
node = NodeInfo(self.queue_name, curs.fetchone())
node.load_status(curs)
self.load_extra_status(curs, node)
if db: db.close()
except DBError, d:
msg = str(d).strip().split('\n', 1)[0].strip()
print('Node %r failure: %s' % (name, msg))
node = NodeInfo(self.queue_name, None, node_name = name)
if db: db.close()
return node

instead of:

def load_node_status (self, name, location):
""" Load node info & status """
# must be thread-safe (!)
if not self.node_alive(name):
node = NodeInfo(self.queue_name, None, node_name = name)
return node
try:
db = None
db = skytools.connect_database (location)
db.set_isolation_level (skytools.I_AUTOCOMMIT)
curs = db.cursor()
curs.execute("select * from pgq_node.get_node_info(%s)", [self.queue_name])
node = NodeInfo(self.queue_name, curs.fetchone())
node.load_status(curs)
self.load_extra_status(curs, node)
except DBError, d:
msg = str(d).strip().split('\n', 1)[0].strip()
print('Node %r failure: %s' % (name, msg))
node = NodeInfo(self.queue_name, None, node_name = name)
finally:
if db: db.close()
return node

Best Regards.

Incomplete string interpolation

Two incomplete string interpolations in file python/londiste/setup.py:

...
self.log.info("Sequence %s gone but exists on subscriber, removing")
...
self.log.info("Table %s gone but exists on subscriber, removing")
...

Please give values for %s.

londiste3 create-root failed with "could not access file "$libdir/pgq_lowlevel""

I have installed skytools3, to test replication with Postgres 9.3. Here's my server environment & related packages version:

server: debian 7.6
postgres: postgresql-9.3 9.3.5-1.pgdg70+1, postgresql-contrib-9.3, postgresql-plperl-9.3
skytools: skytools3 3.1.5-1.pgdg70+1 (from repo: http://apt.postgresql.org/pub/repos/apt/)
python:
- python-minimal 2.7.3-4+deb7u1
- python-pgq3 3.1.5-1.pgdg70+1
- python-psycopg2 2.4.5-1
- python-skytools3 3.1.5-1.pgdg70+1
- python2.7 2.7.3-6+deb7u2
- python2.7-minimal 2.7.3-6+deb7u2

By following londiste3 tutorial, it failed at "londiste3 .... create-root ..." with this error:

[root@shuwnyuan st3simple]$ londiste3 /root/st3simple/st3_l3simple_primary.ini create-root node1 "dbname=l3simple host=localhost user=postgres port=5434"
2014-09-18 08:19:07,123 23596 INFO plpgsql is installed
2014-09-18 08:19:07,123 23596 INFO Installing pgq
2014-09-18 08:19:07,123 23596 INFO Reading from /usr/share/skytools3/pgq.sql
2014-09-18 08:19:07,201 23596 ERROR Job st3_l3simple_db1 got error on connection 'db': could not access file "$libdir/pgq_lowlevel": No such file or directory. Query: CREATE OR REPLACE FUNCTION pgq.insert_event_raw(
queue_n ...
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/skytools-3.0/skytools/scripting.py", line 568, in run_func_safely
return func()
File "/usr/lib/pymodules/python2.7/skytools-3.0/skytools/adminscript.py", line 62, in work
fn(*cmdargs)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/admin.py", line 147, in cmd_create_root
return self.create_node('root', args)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/admin.py", line 198, in create_node
self.install_code(db)
File "/usr/lib/pymodules/python2.7/skytools-3.0/londiste/setup.py", line 29, in install_code
CascadeAdmin.install_code(self, db)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/admin.py", line 425, in install_code
skytools.db_install(db.cursor(), objs, self.log)
File "/usr/lib/pymodules/python2.7/skytools-3.0/skytools/sqltools.py", line 531, in db_install
obj.create(curs, log)
File "/usr/lib/pymodules/python2.7/skytools-3.0/skytools/sqltools.py", line 490, in create
curs.execute(stmt)
File "/usr/lib/python2.7/dist-packages/psycopg2/extras.py", line 123, in execute
return _cursor.execute(self, query, vars)
OperationalError: could not access file "$libdir/pgq_lowlevel": No such file or directory

Is there any version incompatibility issue for the packages I installed? Does skytools3.1.5 supported Postgres 9.3? Appreciate if you can advice how to proceed here. Thanks!

tarball is missing libusual files

Hi,

In order to create Skytools3 RPMs, I downloaded 3.2.6 tarball. However, since it is missing configure script, I wanted to run autogen.sh, but then the tarball is missing the files in the lib/ directory -- which should be fetched from libusual directory.

In short, the tarball is incomplete, and I cannot produce the RPMs. A solution would be creating a tarball that includes everything (+, if possible, a tarball that includes configure script)

Thanks!

Regards, Devrim

Add support for polling PgQ via LISTEN/NOTIFY

I would to be able to write daemon's that use PgQ for messages, but would rather not have to run in an infinite loop with 'sleep' calls. Instead, I would rather block on via libpq to recieve a notification from a 'pgq' channel. I currently have a prototype of this working nicely, whereby I added the following:

create or replace function notify_pgq_tick() returns trigger as $$ begin notify pgqticker; return null; end; $$ language plpgsql;
create trigger pgqtick after insert or update or delete on pgq.tick for each statement  execute procedure notify_pgq_tick();

My application can then block on notifications from the 'pgqticker' channel.

Thoughts?

Bug with a branch node worker

I'm trying to setup root->branch replication using this guide (https://github.com/markokr/skytools/blob/master/doc/howto/londiste3_cascaded_rep_howto.txt)

Everything goes well until launching branch-node worker.
It crashes immediately after startup
2013-01-17 17:00:15,052 5567 DEBUG init
2013-01-17 17:00:15,057 5568 DEBUG Connect 'db' to 'dbname=db2 host=127.0.0.1 port=5431 [...]
'
2013-01-17 17:00:15,062 5568 DEBUG exec_cmd: select * from pgq_node.get_consumer_state('replik
a', 'londiste_db2')
2013-01-17 17:00:15,065 5568 DEBUG 100 Ok
2013-01-17 17:00:15,066 5568 DEBUG exec_cmd: select * from pgq_node.set_consumer_error('replik
a', 'londiste_db2', NULL)
2013-01-17 17:00:15,066 5568 DEBUG 100 None
2013-01-17 17:00:15,067 5568 DEBUG Connect '_provider_db' to 'dbname=db1 host=127.0.0.1 [...
]'
2013-01-17 17:00:15,068 5568 DEBUG exec_cmd: select * from pgq_node.get_node_info('replika')
2013-01-17 17:00:15,074 5568 DEBUG 100 Ok
2013-01-17 17:00:15,086 5568 INFO pgq.maint_operations is installed
2013-01-17 17:00:15,087 5568 DEBUG Sync tables
2013-01-17 17:00:15,088 5568 DEBUG New event: id=1 / type=pgq.location-info / data=slave / ext
ra1=replika
2013-01-17 17:00:15,088 5568 DEBUG got cascade event: pgq.location-info(slave)
2013-01-17 17:00:15,090 5568 DEBUG New event: id=2 / type=pgq.location-info / data=slave / ext
ra1=replika
2013-01-17 17:00:15,090 5568 DEBUG got cascade event: pgq.location-info(slave)
2013-01-17 17:00:15,091 5568 DEBUG Connect 'db' to 'dbname=db2 host=127.0.0.1 port=5431 [...]'
2013-01-17 17:00:15,096 5568 DEBUG exec_cmd: select * from pgq_node.set_consumer_error('replika', 'londiste_db2', 'Writing to queue disabled')
2013-01-17 17:00:15,099 5568 DEBUG 100 Consumer londiste_db2 error = Writing to queue disabled
2013-01-17 17:00:15,099 5568 ERROR Job londiste_db2 got error on connection 'db': Writing to queue disabled. Query: select pgq.current_event_table('replika')
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/skytools-3.0/skytools/scripting.py", line 563, in run_func_safely
return func()
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/consumer.py", line 199, in work
return Consumer.work(self)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/consumer.py", line 280, in work
self._launch_process_batch(db, batch_id, ev_list)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/consumer.py", line 309, in _launch_process_batch
self.process_batch(db, batch_id, list)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/consumer.py", line 175, in process_batch
self.finish_remote_batch(src_db, dst_db, tick_id)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/worker.py", line 362, in finish_remote_batch
self.flush_events(dst_curs)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/cascade/worker.py", line 419, in flush_events
bulk_insert_events(dst_curs, self.ev_buf, flds, st.target_queue)
File "/usr/lib/pymodules/python2.7/skytools-3.0/pgq/producer.py", line 31, in bulk_insert_events
curs.execute(q, [queue_name])
File "/usr/lib/python2.7/dist-packages/psycopg2/extras.py", line 123, in execute
return _cursor.execute(self, query, vars)
InternalError: Writing to queue disabled

Both DBs are on the same server

Test server configuration:
Ubuntu 12.04 LTS
PostgreSQL 9.1.5
Skytools 3.1.2
Python 2.7.3
psycopg2 2.4.5

Ini files:

cat pgqd.ini
[pgqd]
database_list = db1,db2
logfile = /tmp/pgqd.log
pidfile = /tmp/pgqd.pid

cat db1.ini
[londiste3]
job_name = londiste_db1
db = dbname=db1 host=127.0.0.1 port=5431 password=*****
queue_name = replika
logfile = /tmp/%(job_name)s.log
pidfile = /tmp/%(job_name)s.pid

pgq_autocommit = 1
pgq_lazy_fetch = 0

cat db2.ini
[londiste3]
job_name = londiste_db2
db = dbname=db2 host=127.0.0.1 port=5431 password=*****
queue_name = replika
logfile = /tmp/%(job_name)s.log
pidfile = /tmp/%(job_name)s.pid

pgq_autocommit = 1
pgq_lazy_fetch = 0

What database permissions does Skytools 3 require?

It would be useful to list this, particularly for cases like RDS in AWS where you can't add whatever permissions you like.

londiste3 error with import pkgloader

My server is running Debian 7.8, with Postgresql-9.3 (9.3.6-1.pgdg70+1)

I want to use Londiste replication. I have installed the following Skytools package via apt-get install:

python-skytools3 3.2.6-3.pgdg70+1
skytools-modules-9.3 2.1.13-4.pgdg70+2
skytools3 3.2.6-3.pgdg70+1

After installation, upon running londiste command, or even service skytools3 start, I run into this error:

Traceback (most recent call last):
File "/usr/bin/scriptmgr3", line 48, in
import pkgloader
ImportError: No module named pkgloader

Trying locate pkgloader.py give me this:
/usr/lib/python2.7/site-packages/pkgloader.py

Any pointers here will be greatly appreciated, thanks!

Skytools SQL has a missing function

Hi,

I was today going through a server which had quite some lag and was forced to run "replay --rewind". All this with the surprise that I got this error:

-bash-4.1$ londiste.py ../londiste_main.ini replay --rewind
2015-02-17 16:27:55,650 10308 INFO Rewinding queue
2015-02-17 16:27:55,699 10308 CRITICAL Job main_replicate crashed: <class 'psycopg2.ProgrammingError'>: 'function pgq.register_consumer_at(unknown, unknown, integer) does not exist
LINE 1: select pgq.register_consumer_at('main_replicate', 'main...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.' (<traceback object at 0x2213098>: [' File "/usr/pgsql-9.1/lib64/python2.6/site-packages/skytools/scripting.py", line 459, in run\n self.startup()\n', ' File "/usr/pgsql-9.1/lib64/python2.6/site-packages/pgq/consumer.py", line 378, in startup\n self.rewind()\n', ' File "/usr/pgsql-9.1/lib64/python2.6/site-packages/pgq/consumer.py", line 471, in rewind\n src_curs.execute(q, [self.pgq_queue_name, self.consumer_id, dst_tick])\n', ' File "/usr/lib64/python2.6/site-packages/psycopg2/extras.py", line 121, in execute\n return super(DictCursor, self).execute(query, vars)\n'])

Interesting thing was that there is no pgq.register_consumer_at function in the SQL for skytools 2 (running skytools 2.1.13), but there is a call to such a function in the python code, specifically when using --rewind.

I was able to find the pgq.register_consumer_at in the skytools3 pgq.register_consumer.sql

I will be checking the code to see if the new register_consumer* functions are compatible with skytools2, but wanted to let you know of the bug

unique key names are not escaped

one of our programmers created a table with a column called "iso_639-1", it replicates fine.
There's a uniqe key on that column caled "unq_iso_639-1", which causes londiste to crash when adding the table to the subscriber.

2013-01-09 12:48:07,082 11289 INFO Creating unq_iso_639-1
2013-01-09 12:48:07,101 11289 CRITICAL Job londiste-woordenboek crashed: <class 'psycopg2.ProgrammingError'>: 'syntax error at or near "-"
LINE 1: ...LY dictionary.languages ADD CONSTRAINT unq_iso_639-1 UNIQUE ...
^' (<traceback object at 0x7f8af1168758>: [' File "/var/lib/python-support/python2.5/skytools/scripting.py", line 510, in run_once\n return self.work()\n', ' File "/var/lib/python-support/python2.5/pgq/consumer.py", line 175, in work\n self._launch_process_batch(db, batch_id, list)\n', ' File "/var/lib/python-support/python2.5/pgq/consumer.py", line 207, in _launch_process_batch\n self.process_batch(db, batch_id, list)\n', ' File "/var/lib/python-support/python2.5/pgq/consumer.py", line 409, in process_batch\n self.process_remote_batch(db, batch_id, event_list, dst_db)\n', ' File "/var/lib/python-support/python2.5/londiste/playback.py", line 304, in process_remote_batch\n self.sync_tables(dst_db)\n', ' File "/var/lib/python-support/python2.5/londiste/playback.py", line 348, in sync_tables\n res = self.sync_from_copy_thread(cnt, dst_db)\n', ' File "/var/lib/python-support/python2.5/londiste/playback.py", line 448, in sync_from_copy_thread\n self.do_copy(t)\n', ' File "/var/lib/python-support/python2.5/londiste/table_copy.py", line 79, in do_copy\n dst_struct.create(dst_curs, objs, log = self.log)\n', ' File "/var/lib/python-support/python2.5/skytools/dbstruct.py", line 350, in create\n curs.execute(sql)\n', ' File "/usr/lib/python2.5/site-packages/psycopg2/extras.py", line 88, in execute\n return _cursor.execute(self, query, vars, async)\n'])

Strange behavior of the option ignore

Good day

I'm interested in is described in the documentation (http://skytools.projects.postgresql.org/skytools-3.0/pgq/files/triggers-sql.html) option IGNORE.

This option allows you to update the expressions do not look at these columns. That is, If the query involves only columns ignored - the event is not created.

The problem is that the event is ignored updates only if the column value is changed (old is not equal to new)
That, in my opinion is not logical, and all the profit from the opportunities lost.
A feeling that the code just a mistake!
Ideally, it should ignore the update expression in that case if it involves only the columns from the list, ignore list, and all, in spite of everything else.

I would like that the event would be ignored and if the column is not updated from the ignore list, but the value does not change! (This is a dream, it is not declared and not implemented)

Test from the developers did not reveal the problem. Because there are updates on new meaning. And if it's an update statement run a second time, the event will be generated.
https://github.com/markokr/skytools/blob/master/sql/pgq/sql/logutriga.sql

create table udata (
id serial primary key,
txt text,
bin bytea
);

create trigger utest after insert or update or delete on udata
for each row execute procedure pgq.logutriga ('udata_que', 'backup', 'ignore = bin');

insert into udata values ​​(1, 'txt', 'bin');
NOTICE: insert_event (udata_que, I: id, id = 1 & txt = txt, public.udata)
update udata set txt = 'txt';
NOTICE: insert_event (udata_que, U: id, id = 1 & txt = txt, public.udata)
update udata set txt = 'txt2', bin = 'bin2';
NOTICE: insert_event (udata_que, U: id, id = 1 & txt = txt2, public.udata)
update udata set bin = 'bin2'
no events!
update udata set bin = 'bin2'
NOTICE: insert_event (udata_que, U: id, id = 1 & txt = txt, public.udata) #!!!??? bug?

I do not understand this error, omission, or I would not have understood correctly stated functionality.