sqrtminusone / nginx-logs-to-postgres

A simple program to store nginx logs to PostgreSQL

nginx-logs-to-postgres

A simple program to store nginx logs to PostgreSQL.

Usage

git clone [email protected]:SqrtMinusOne/nginx-logs-to-postgres.git
cd ./nginx-logs-to-postgres
go build nginx-logs

Set nginx log format as follows:

log_format json_combined '{"time_local": "$time_local", '
    '"path": "$request_uri", '   
    '"ip": "$remote_addr", '
    '"server_name": "$server_name", '
    '"remote_user": "$remote_user", '
    '"remote_port": "$remote_port", '
    '"time": "$time_iso8601", '
    '"user_agent": "$http_user_agent", '
    '"user_id_got": "$uid_got", '
    '"user_id_set": "$uid_set", '
    '"request": "$request", '
    '"status": "$status", '
    '"body_bytes_sent": "$body_bytes_sent", '
    '"request_time": "$request_time", '
    '"request_method": "$request_method", '
    '"geoip_country_code": "$geoip_country_code", '
    '"http_referrer": "$http_referer" }';
access_log /var/log/nginx/access.log json_combined;

To change the set of fields, edit the createTemplate string, parseLine and storeLogs functions.

Set the following variables to .env file:

• DATABASE_URL - PostgreSQL connection URI
• TABLE_SCHEMA, TABLE_NAME
• LOG_FILE - path to the nginx log file. Make sure it exists and the program can read it
• FLUSH_MS - rate limit to inserts

Run the program

./nginx-logs

Example supervisor config

[program:nginx-logs]
command=/home/pavel/bin/nginx-logs-to-postgres/nginx-logs
directory=/home/pavel/bin/nginx-logs-to-postgres/
autostart=true