pgq / pgqd

PgQ maintenance daemon

pgqd

Maintenance daemon for PgQ

Manual section:	1

Synopsis

pgqd [-qvd] config

pgqd [-skr] config

pgqd --ini|-h|-V

Description

Runs both ticker and periodic maintenence for all databases in one PostgreSQL cluster.

Options

-q	Do not log to stdout
-v	Verbose log
-d	Daemonize process
-s	Send SIGINT to running process to stop it
-k	Send SIGTERM to running process to stop it
-r	Send SIGHUP to running process to reload config
-h	Show help
-V	Show version
--ini	Show sample config

Configuration

Config uses ini file syntax:

[pgqd]
logfile = ~/log/pgqd.log
pidfile = ~/pid/pgqd.pid

Options:

logfile

Filename to log to. Default: empty.

pidfile

Filename to store pid, required when daemonizing. Default: empty.

base_connstr

Connect string without dbname= Default: empty.

initial_database

Startup db to query other databases. Default: template1

database_list

Limit ticker to specific databases. Default: empty, which means all database.

syslog

Whether to log into syslog. Default: 1

syslog_ident

Name to use for syslog. Default: pgqd

check_period

How often to check for new databases, in seconds. Default: 60.

retry_period

How often to flush retry queue, in seconds. Default: 30

maint_period

How often to do maintentance, in seconds. Default: 120

ticker_period

How often to run ticker, in seconds. Default: 1

Install

pgqd uses autoconf based build system:

./configure --prefix=/opt
make
make install

Dependencies: libevent, python3-docutils

pgqd's Issues

lib folder is empty in v3.5 release

maybe add it back or notice users to download it

PgQD license?

Hello! This repo doesn't have license file. How is it licensed?

Non-descriptive error with pgqd 3.3

I'm running a custom built pgqd and running into a weird error I could not track down in the code, as there is no message:

2020-01-10 20:42:16.699 27 ERROR 

Full output with -v

2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: [pgqd]
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'syslog' = '0'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'syslog' = '0' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'base_connstr' = '***'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'base_connstr' = ***' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'initial_database' = 'app'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'initial_database' = 'app' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'database_list' = 'app'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'database_list' = 'app' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'check_period' = '60'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'check_period' = '60' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'retry_period' = '30'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'retry_period' = '30' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'maint_period' = '120'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'maint_period' = '120' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'ticker_period' = '1'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'ticker_period' = '1' ok:1
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'connection_lifetime' = '3600'
2020-01-10 20:42:16.606 27 DEBUG parse_ini_file: 'connection_lifetime' = '3600' ok:1
2020-01-10 20:42:16.606 27 LOG Starting pgqd 3.3
2020-01-10 20:42:16.607 27 DEBUG app: launch_ticker
2020-01-10 20:42:16.673 27 DEBUG app: select 1 from pg_catalog.pg_namespace where nspname='pgq'
2020-01-10 20:42:16.675 27 DEBUG app: select pgq.version()
2020-01-10 20:42:16.678 27 LOG app: pgq version ok: 3.2.6
2020-01-10 20:42:16.678 27 DEBUG app: launch_maint
2020-01-10 20:42:16.681 27 DEBUG app: launch_retry
2020-01-10 20:42:16.692 27 DEBUG app: starting maintenance
2020-01-10 20:42:16.693 27 DEBUG app: select 1 from pg_proc p, pg_namespace n where p.pronamespace = n.oid   and p.proname = 'maint_operations'   and n.nspname = 'pgq'
2020-01-10 20:42:16.694 27 DEBUG app: select func_name, func_arg from pgq.maint_operations()
2020-01-10 20:42:16.697 27 DEBUG app: starting retry event processing
2020-01-10 20:42:16.697 27 DEBUG app: select * from pgq.maint_retry_events()
2020-01-10 20:42:16.699 27 ERROR 
2020-01-10 20:42:16.699 27 DEBUG app: close_maint, 20.000000
2020-01-10 20:42:16.701 27 DEBUG app: close_retry, 30.000000

Dockerfile:

FROM alpine:3.10

ARG PGQD_URL=https://github.com/pgq/pgqd.git

RUN apk add --update --no-cache libevent libpq \
 && apk add --update --no-cache --virtual devdeps autoconf automake gcc git libevent-dev libtool make musl-dev postgresql-dev linux-headers \
 && git clone https://github.com/pgq/pgqd.git /tmp/pgqd \
 && cd /tmp/pgqd \
 && git submodule update --init \
 && ./autogen.sh \
 && ./configure \
 && make \
 && make install \
 && cd / \
 && rm -rf /tmp/pgqd \
 && apk del --purge devdeps \
 && mkdir -p /opt/pgqd

COPY etc/ /etc/

CMD /usr/local/bin/pgqd -v /etc/pgqd.ini

I did notice that pgqd is 3.3 and pgq is 3.2.6, will go hunt that down and see if it's related.

Any thoughts as to what this could be?

Multiple database list

Previously I used the skytools version 3.2.6.
I had replication from one central server to several nodes. And also replication from all nodes to one central server.
For this I used 2 pgqd.
The configuration file (pgqd_center.conf and pgqd_nodes.conf) looked like this:

pgqd_center.conf:
[pgqd]
base_connstr = dbname=centralnode host=192.168.1.10 port=5432 user=postgres

pgqd_nodes.conf:
[pgqd]
database_list = simplenode dbname=simplenode host=192.168.1.1 port=5432 user=postgres, simplenode dbname=simplenode host=192.168.1.2 port=5432 user=postgres, simplenode dbname=simplenode host=192.168.1.3 port=5432 user=postgres

everything worked well.

But when I needed to use new versions of londiste, pgq, pgqd-3.5, these configuration files do not work.
I'm getting errors:
WARNING [simplenode dbname=simplenode host=192.168.1.1 port=5432 user=postgres]: default timeout
LOG {ticks: 0, maint: 0, retry: 0}
LOG {ticks: 0, maint: 0, retry: 0}
ERROR connection error: PQconnectPoll
ERROR libpq: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: database "simplenode dbname=simplenode host=192.168.1.1 port=5432 user=po" does not exist

How in the new version of pgqd it is necessary to specify a list of several databases located on different hosts?

Failing database connection should fail the pgqd process

Our pgqd couldn't connect to the database (couldn't resolve the DB hostname due to networking issues).
The pgqd process keeps retrying, but doesn't fail. So if we just monitor the pgqd process/service, it looks healthy.

Should pgqd fail after a few retries?