onebip / recruiter

Job Queue Manager: high performance, high volume, persistent, fault tolerant. 100% PHP/MongoDB, 100% Awesome

Disclaimer

This is a work in progress not ready to be used or seen. This work is sponsored by Onebip

Recruiter

It's a Job Queue Manager built with PHP meant to be used in PHP projects. Features and characteristics:

• Jobs are made persistent on MongoDB
• Jobs are retriable with complex and customizable strategies
• Multiple queues are supported through tagging
• Jobs are stored by default in an history collection for after the fact inspection and analytics
• Built to be robust, scalable and fault tolerant

History

Onebip is a payment system (think PayPal with mobile devices in place of credit cards), things like: payment notifications, subscription renewals, remainder messages, … are really important. You cannot skip or lose a job (notification are idempotent but payments are not). You cannot forgot to have completed a job (customer/merchant support must have data to do their job). You need to know if and when you can retry a failed job (external services have rate limits and are based on agreements/contracts). We have developed internally our job/queue solution called Recruiter. After a year in production and many billions of jobs we have decided to put what we have learned into a stand alone project and to make it available to everyone.

recruiter's Issues

Consider using queue interprop as abstraction for queues

Hello!

Please look at https://github.com/queue-interop/queue-interop project. Using interfaces from it allows us reuse some implementations like enqueue, which supports a lot of transports. You can outsource some code

Timezone switch bug

The function \Recruiter\RetryPolicy\TimeTable::estimateHowManyRetriesIn assumes that the default timezone is UTC (which should be the best practice).

In case the default timezone is different than UTC (please refer to date_default_timezone_get() to known the default time zone of your PHP runtime) the estimateHowManyRetriesIn will fall in a subtle bug which causes a time travel in the future when an amount of time is subtracted from another instance of time (at [1] you can read one of the many reports about this bug).

Please note that supporting only UTC is a good idea. Maybe we should just check the result of the function date_default_timezone_get and if it is different from UTC we should throw an exception.

[1] - https://bugs.php.net/bug.php?id=68286