Coder Social home page Coder Social logo

pfstats's Introduction

pfstats

This is a simple Postfix mail log parser I wrote in python that can filter postfix mail log lines based on date, status, sender, and subject.

In my machine (Ubuntu 14.04LTS, 2GB Ram, 2x Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz) it processed 5 million lines in 1.2-1.5 mins. Script was run without filtering emails by status (meaning it calculated all: bounced, sent, deferred and rejected) or sender, but I specified the date (not today's date (default)), which means that the script had to combine gziped log lines from two files and then process them.

Requirements

Logrotate

Daily logrotate created in this format for postfix logs:

/var/log/postfix/*.log {
    rotate 366
    dateext
    daily
    missingok
    notifempty
    compress
    delaycompress
}

Example of postfix log directory files:

mail.log
mail.log-20170724.gz
mail.log-20170725.gz
mail.log-20170726.gz
mail.log-20170727.gz
mail.log-20170728.gz
mail.log-20170729.gz
mail.log-20170730.gz
mail.log-20170731.gz
mail.log-20170801.gz
mail.log-20170802.gz
mail.log-20170803.gz
mail.log-20170804

Postfix subject message in log

This is a little trick for Postfix, it lets you log the subject, from and to of all the emails postfix sends (or which pass through it if you run it as a relay). It comes in handy when you need to debug an email issue and need to confirm your mailserver has sent the message.

First create the file /etc/postfix/header_checks and insert this into it:

/^subject:/      WARN

Now, in your postfix /etc/postfix/main.cf add the following to the end of the file:

header_checks = regexp:/etc/postfix/header_checks

And restart postfix:

$ /etc/init.d/postfix restart

You will hopefully now get log items like below, and if not you have a problem with your mailserver:

Dec  4 08:23:05 localhost postfix/cleanup[2278]: 90CA714: warning: header Subject: This is a testmail which gets logged from localhost[127.0.0.1]; from=<root@localhost> to=<root@localhost> proto=ESMTP helo=<localhost>

Usage

$ pfstats.py -h
usage: pfstats.py [-h] [-d] [-t] [-s] [-m] [-l] [--log-dir]
                  [--output-directory] [--output-delimiter]
                  [--output-filetype]

Filter and format Postfix log files.

optional arguments:
  -h, --help           show this help message and exit
  -d , --date          Specify different date. Default is current date.
                       Format: Jan 20 (note one space) &
                       Jan  2 (note two spaces).
                       Default is todays date: Aug  7

  -t , --type          Type of email status: bounced, sent, rejected, deferred.
                       Default is all.

  -s , --sender        Specify senders address in order to query logs matching this parameter

  -m , --message       Postfix default log format must be changed for this option to work.
                       Add subject message in logs, and then you can use this option to query
                       those emails with specific subject message.

  -l , --log           Specify the log file you want to use.
                       Default is: /var/log/postfix/mail.log

  --log-dir            Specify the log directory.
                       Default is: /var/log/postfix/

  --output-directory   Specify the generated file(s) directory.
                       Default is current working directory: /home/myuser/pfstats

  --output-delimiter   Specify the generated output delimiter.
                       Default is ";"

  --output-filetype    Specify the generated output file type.
                       Default is "csv"

Example 1

$ python pfstats.py -d 'Jul 26' -s '[email protected]'


************************* RESULTS *************************

bounced:        5079            bounced-261841.csv
deferred:       2570            deferred-261841.csv
sent:           9397            sent-261841.csv
rejected:       0               rejected-261841.csv

-----
Total:          17046

***********************************************************
--- 69.95447611808777 seconds ---

Example 2

$ python pfstats.py -d 'Jul 26' -t 'bounced' -s '[email protected]'


************************* RESULTS *************************

bounced:        5079            bounced-882924.csv

-----
Total:          5079

***********************************************************
--- 19.795605659484863 seconds ---

Output Examples

Sent

date;sender;receiver;message_id;subject;host_message
Jul 26;[email protected];[email protected];96CF641D9A;Hello World;(250 2.6.0 <[email protected]> [InternalId=18245021073525, Hostname=mail.altinukshini.com] 82359 bytes in 0.119, 672.660 KB/sec Queued mail for delivery)

Bounced

date;sender;receiver;message_id;subject;host_message
Jul 26;[email protected];[email protected];9018142B3F;Hello World;(host mx2.hotmail.com[207.46.8.199] said: 550 Requested action not taken: mailbox unavailable (in reply to RCPT TO command))

pfstats's People

Contributors

altinukshini avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.