tpitale / mail_room Goto Github PK

View Code? Open in Web Editor NEW

194.0 194.0 51.0 313 KB

Forward mail from gmail IMAP to a callback URL or job worker, simply.

License: MIT License

Ruby 100.00%

mail_room's People

Stargazers

Watchers

Forkers

helpershub sahild yiftee stuartj akhilsengar shinobidevs tbreier i108 hatkirby apsislabs douwem flyeven javex stewartmckee zaratropia pcarranza godfat jacobvosmaer brodock galet 5minpause kmwhite craigmiskell-gitlab cablett jessiel-hacke gmanley happyfuncorp tfyre higrow mjang stanhu gogainda jarkak 0x2c7 cajun clairton forwardfinancing nevans ryochin alexwiththescar tjaartvdwalt shernade erwin suupic martiandominic wrozee adz jessecai33 zach-brown-hpa dnitza zzorin

mail_room's Issues

Allow log path to be configured to STDOUT/STDERR

For structured logging, we may want to direct output to STDOUT/STDERR and not just a file path.

Raise a configuration error if required fields are missing on a mailbox

In the validate! for a mailbox we should check for required fields like name. Raise a ConfigurationError otherwise.

Reconnect to IMAP if the server disconnects us

If the server disconnects, we get a connection closed Net::IMAP::Error. Rather than failing out, or silently swallowing it, we should reconnect.

Handle encoding issues from email in JSON-using delivery methods

Both Que and Sidekiq dump to json. Encodings other than properly recognized UTF-8 will trip json up.

From @DouweM:
A reproduction of the specific case GitLab users were having trouble with is this:

message = "é"
message.force_encoding("ASCII-8BIT")

Note that in this case, the message was actually UTF-8-encoded, but for some reason Ruby thought it was ASCII-8BIT, and when trying to do JSON.generate, it tried to convert this string that was actually UTF-8 encoded from ASCII-8BIT to UTF-8, yielding the "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError) error you see in https://gitlab.com/gitlab-org/gitlab-ce/issues/2698.

Reproduce this by doing:

JSON.generate(message) # => error

In this case, simply doing message.force_encoding('UTF-8') would work, but not in this case, where the original message was ISO-8859-1 rather than UTF-8:

message = "é".encode('ISO-8859-1')
message.force_encoding('ASCII-8BIT')
message.force_encoding("UTF-8")
JSON.generate(message) # => error

Encoding issues with Faraday

I am using mail_room in combination with the postback delivery method (hence using Faraday). I somehow noticed that emails containing a % got "lost" (they were marked as 'seen' in GMail, but the specified delivery_url was apparently never called).

I digged a little into it and, by enabling logging for Faraday, confirmed my suspicion that there might be an encoding issue: When an email containing a % (for example: An email with the message body test %) gets POSTed through Faraday, the message delivery silently fails:

I, [2015-03-23T14:46:12.456947 #98851]  INFO -- : post https://xxxxxxx.xxxxxxx
D, [2015-03-23T14:46:12.457528 #98851] DEBUG -- request: User-Agent: "Faraday v0.9.1"
Authorization: "Token token=\"my_token\""
Content-Type: "application/x-www-form-urlencoded"
I, [2015-03-23T14:46:13.057616 #98851]  INFO -- Status: 400
D, [2015-03-23T14:46:13.057934 #98851] DEBUG -- response: transfer-encoding: "chunked"
connection: "close"
status: "400 Bad Request"

That kind of makes sense, because a % cannot stand on its own using x-www-form-urlencoded. (Try, for example http://stackoverflow.com/test%test).

I seem to be able to successfully POST an email with a % if I pass URI.encode_www_form_component(message) instead of message to Faraday. I am a little confused by that though, because I would have assumed that Faraday automatically handles encoding...?

Error: `require': cannot load such file -- charlock_holmes

After update to last version 0.6.1 - got an error _cannot load such file -- charlock_holmes_.
It works if I manually add charlock_holmes gem to my Gemfile. Anybody know why I lost dependency?

/home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery/sidekiq.rb:4:in `require': cannot load such file -- charlock_holmes (LoadError)
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery/sidekiq.rb:4:in `<top (required)>'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery.rb:4:in `require_relative'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery.rb:4:in `[]'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:50:in `delivery_klass'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:58:in `delivery'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:75:in `deliver'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:19:in `block in process'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:17:in `each'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:17:in `process'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:166:in `process_mailbox'
    from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:146:in `block in run'

Update ruby version requirements, drop imap backport

@DouweM What ruby versions does gitlab support? If I dropped support from mail_room prior to 2.3 would that be an issue?

Possible race condition in mailbox_watcher.rb quit method - undefined method `join' for nil:NilClass

In our GitLab deployment we are running into the following error:

2016-07-26_08:31:54.08717 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/mailbox_watcher.rb:50:in `quit': undefined method `join' for nil:NilClass (NoMethodError)
2016-07-26_08:31:54.08721       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:30:in `each'
2016-07-26_08:31:54.08721       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:30:in `quit'
2016-07-26_08:31:54.08722       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:25:in `ensure in run'
2016-07-26_08:31:54.08722       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:25:in `run'
2016-07-26_08:31:54.08722       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/cli.rb:52:in `start'
2016-07-26_08:31:54.08723       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/bin/mail_room:5:in `<top (required)>'
2016-07-26_08:31:54.08723       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2016-07-26_08:31:54.08724       from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'

It seems to be caused by some kind of race condition.

Original issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/20273

Possible problem with postback delivery method in Rails apps

Logging works as a delivery method, but postback doesn't seem to send POST request to locally running Rails app (no trace of any POST request in the console/log).

See https://github.com/ppppppachyderm/mail_room_rails_example for attempted implementation and http://stackoverflow.com/questions/21138237/mail-room-with-ruby-on-rails for additional details.

I've been trying the Rails 4.0.2 barebone implementation above, an identical Rails 3.2.16 implementation and implementing mail_room into the users controller of an existing 4.0.2 app. In every case logging works but postback doesn't seem to.

Accept mailbox configuration for one account directly on the CLI

If we want to start a simple postback delivery, we could accept the configuration for it from the CLI directly, without the need for a configuration file.

When script is restarted or server rebooted messages can be missed

Not sure if this is in scope for this awesome gem. Could mail query IMAP for NEW instead of UNSEEN? this would have the upside of catching missed messages and the downside of reading all the messages in a previously used mailbox.

Configurable IMAP server support

We should be able to support configuration for non-gmail IMAP servers. I don't think there is any particular thing specific to gmail with IMAP.

Based on the work by @matte in #9.

Add readme note about using -q with upstart/god/monit/etc

If -q is used, and no configuration is found or the app is misconfigured, it will not run (no mailboxes). This is better than running and sleeping continuously, but it may just cause a process manager to continually try to restart.

Make a note in the readme of this behavior, and about throttling process restarts.

Support sending in format that can be received by griddler

https://github.com/thoughtbot/griddler accepts POST in a format from sendgrids parse api hooks. When configured, mail_room should be able to send in that format.

mail_room periodically gets stuck communicating with IMAP servers

This stems from a GitLab issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/13357

It seems a number of users are having issues with mail_room getting stuck and not processing any messages. gdb backtraces are available in the notes.

It looks like there may be two issues here with Ruby's Net::IMAP implementation:

May block indefinitely in connect(): https://github.com/rubysl/rubysl-net-imap/blob/2.0/lib/rubysl/net/imap/imap.rb#L1454
May sit indefinitely in IDLE state, waiting for messages that never arrive

You can replicate the first problem by just pointing mail_room to a local server on port 5000 by running nc -l 127.0.0.1 5000. It just sits there.

Regarding the second case, it looks like Ruby 2.3 added a timeout parameter to the idle method perhaps to help avoid this issue. We're using Ruby 2.1 now and won't be at 2.3 for some time.

I can see a couple of solutions that would help:

Use/fork a different IMAP client (one that has stricter timeouts)
Add a thread to monitor status of IMAP threads (e.g. ping occasionally, retry IDLE command, etc.)
Enable TCP keepalives
Monkey-patching IMAP socket?

Thoughts?

Redis sentinel support

Could you include support for defining a sentinel-enabled connection?
redis-rb already support so, all you have to do is pass "sentinels" array of hashes connection param.

Ideally you would change "redis_url" YAML key to just "redis" and accept every redis possible setting:

url
host
port
password
db
namespace
sentinels

See: https://github.com/redis/redis-rb#sentinel-support

Using Rails '3.2.21' Ruby '2.0.0-p0'

When I got mail into my inbox I am not able to see anything else into the logs except below..

$ bundle exec foreman start -p 3000
18:27:25 web.1  | started with pid 27546
18:27:25 mail.1 | started with pid 27547

18:27:37 web.1  | [RAILS-DEV-BOOST] Unable to start rails-dev-boost in an asynchronous mode. Are you running on an OS not supported by the `listen` gem?
18:27:37 web.1  | [RAILS-DEV-BOOST] To get rid of this message disable the rails-dev-boost's async mode by putting the following code in a Rails initializer file (these are found in config/initializers directory):
18:27:37 web.1  | [RAILS-DEV-BOOST] 
18:27:37 web.1  | [RAILS-DEV-BOOST]     RailsDevelopmentBoost.async = false if defined?(RailsDevelopmentBoost)
18:27:37 web.1  | [RAILS-DEV-BOOST] 
18:27:38 web.1  | [2014-12-13 18:27:38] INFO  WEBrick 1.3.1
18:27:38 web.1  | [2014-12-13 18:27:38] INFO  ruby 2.0.0 (2013-02-24) [i686-linux]
18:27:38 web.1  | [2014-12-13 18:27:38] INFO  WEBrick::HTTPServer#start: pid=27546 port=3000
18:28:22 web.1  | [deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.

Arbitration locking issue

I think there's a problem with the redis arbitration locking, and we're seeing unread messages get stuck in our inbox as a result.

The 'lock' is taken in deliver? which is called during the search for UNSEEN messages (new_message_ids). If something goes wrong during the actual message fetch, then the message will not be marked SEEN. When a mail_room calls new_message_ids (the restarted one, or another instance) before the key expires (10min by default), we'll call deliver? which will increment the key and reset the TTL, and return false (the value will != 1). If we never go 10 minutes without receiving mail (or with no mail_room processes running), the key will keep on being incremented and the TTL keep on being reset, and the message will never be picked up again.

I don't know what the solution is I'm afraid, but I'm happy to try and help figure it out.

Document usage with Griddler

https://robots.thoughtbot.com/griddler-is-better-than-ever

Transactional SEEN/UNSEEN

Delivery errors have so far not been a problem that simply restarting mail_room doesn't fix. But it's possible we would not want those errors on delivery to stop fetching, or to mark messages as SEEN.

When using UNSEEN as the search command, it's impossible to have transactional mail because Google handles marking as SEEN. Whereas NEW/DELETED are controlled by mail-room after delivery.

Structured logging of message fetching and delivery

It may be valuable to know that messages are being fetched/processed/delivered to a remote location in a structured log. Still undecided on this. If using que/sidekiq new jobs will be processed; if using logger all messages are already logged. Only in a remote delivery case (callbacks) would it be especially hard to know if new messages were being fetched/delivered without a log if the remote service is not within the users control.

Add docs/support for ActionMailbox

https://edgeguides.rubyonrails.org/action_mailbox_basics.html#sendgrid (for example)

Add configuration to readme
Forward mail to the rails url https://example.com/rails/action_mailbox/{{service}}/inbound_emails
Figure out which format to use to submit
Add any code module to use mail_room as an ingress configuration

Que support

Handle EOFError

/Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1150:in `get_tagged_response': end of file reached (EOFError)
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1210:in `block in send_command'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1192:in `send_command'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1345:in `block in search_internal'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1341:in `search_internal'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:768:in `search'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:39:in `new_message_ids'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:29:in `new_messages'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:17:in `process'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_watcher.rb:152:in `process_mailbox'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_watcher.rb:129:in `block in run'

Provide way to add new delivery method

If someone is running this code as a library (rather than just from the CLI) allow them to load and extend the registry of delivery methods available.

This could maybe be done with gems as plugins for each delivery method? Pure speculation.

Having N mail_rooms on N application servers will result in 1 message being processed N times

In the case of GitLab, this means every "Reply by email" message would result in N comments being created.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/2870.

@tpitale What's your take on this? Our (@jacobvosmaer and me) solution has mail_room use Redis to keep track of what messages have been processed, so that only one mail_room ever processes a specific message, but this does add complexity to mail_room that a lot of use cases (like yours, it appears) don't need.

Add quiet CLI flag `-q`

If -q is set, do not raise an error if there is no configuration or configuration is missing the top level mailboxes key.

Create example Rails endpoint

To further help newcomers, we could add an example of a simple endpoint with auth for the postback delivery method.

Structured logging of arbitration

It may be valuable to know which decision arbitration is making for a given message uid. Though, it is a boolean option … either it's in the list or not. So, undecided there.

We should log this to STDOUT if configuration for arbitration includes logging: true (or something, undecided). Logs should be in a standardized, structure form that is machine readable (research required for that).

Do not silently swallow disconnects from the server in the idling thread.

Add example upstart script

http://code.alexreisner.com/articles/upstart.html

Add arbitration to readme

scenarios for arbitration
no guarantee of non-duplicates, just our best effort to reduce
pre-reqs, aka redis at this time
advice for handling dups, in case (idempotent)

Resource usage for large mailboxes

Add debug mode to diagnose issues

It would be really helpful to have a debug mode that shows what mail_room is doing so that we can diagnose whether there is an issue with the configuration, mail_room, or the IMAP server. It can just be a configuration flag (e.g. -d or -v).

/cc: @godfat

IDLE not working correctly

I've been using this gem for a few days now, and it seems that it works for maybe 30 minutes, but then it seems to stop noticing any new emails coming through. Could it be possible that IDLE isn't working as intended? Restarting mail_room then correctly pulls in emails for another period of time until it stops again.

After seeing that emails weren't coming through, I checked mail_room was still running, it just wasn't noticing any new emails.

Incorrectly configured Mailroom uses 100% CPU

If the Mailroom can't connect to the mailserver due to an SSL error it will keep attempting to connect to it going so far as to dedicate a CPU core to the task of banging the mailserver.

To be precise, the command /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/bundle exec mail_room -c /var/opt/gitlab/gitlab-rails/etc/mail_room.yml is using 100% CPU.

My system is Ubuntu Server 14.04 and I am using the omnibus distribution of GitLab.

Here is an excerpt from my mailroom log file:

2015-12-05_16:28:10.65112 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:10.65116   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:10.65116   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:10.65117   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:10.65117   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:10.65118   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:10.65121   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:10.65122   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:10.65123   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:10.65123   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:10.65124   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:10.65124   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:10.65125   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:10.65125   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:10.82077 Runit: starting mail_room
2015-12-05_16:28:23.33484 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:23.33493   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:23.33494   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:23.33496   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:23.33498   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:23.33499   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:23.33501   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:23.33502   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:23.33504   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:23.33506   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:23.33507   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:23.33509   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:23.33510   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:23.33512   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:23.51205 Runit: starting mail_room
2015-12-05_16:28:36.27779 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:36.27782   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:36.27783   from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:36.27783   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:36.27783   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:36.27784   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:36.27784   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:36.27785   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:36.27787   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:36.27787   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:36.27787   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:36.27788   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:36.27788   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:36.27788   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:36.44894 Runit: starting mail_room

Originally reported in https://github.com/gitlabhq/gitlabhq/issues/9891

Mailroom crash on lost connection to IMAP

https://gitlab.com/gitlab-org/gitlab-ce/issues/50842

bundler: failed to load command: mail_room (/home/git/gitlab/vendor/bundle/ruby/2.4.0/bin/mail_room)
Net::IMAP::ByeResponseError: AXIGEN Proxy front1 IMAP4rev1: cannot connect to AXIGEN
  /usr/local/lib/ruby/2.4.0/net/imap.rb:1188:in `get_tagged_response'
  /usr/local/lib/ruby/2.4.0/net/imap.rb:1248:in `block in send_command'
  /usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  /usr/local/lib/ruby/2.4.0/net/imap.rb:1230:in `send_command'
  /usr/local/lib/ruby/2.4.0/net/imap.rb:434:in `login'
  /home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:84:in `log_in'
  /home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:68:in `setup'
  /home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:54:in `rescue in wait'
  /home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:44:in `wait'
  /home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/mailbox_watcher.rb:34:in `block in run'

Investigate concurrent-ruby to encapsulate threaded behavior more safely.

The concurrent-ruby library appears to have reached a reasonable level of stability and is in use by a number of major ruby projects.

It may be worth investigating to see if it could be leveraged to improve the code around threaded behaviors in mail_room. It would be our only required dependency.

Areas that may be improved:

coordinator
mailbox_watcher
connection (?)
…

Stopped responding, generated a ThreadError when terminated

I was running mail_room overnight, to test it, outputting to logger. It worked for awhile, but when I woke up in the am, it hadn't pulled in any of the new messages that had come in. Ctrl+C and I got this error:

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:185:in `lock': can't be called from trap context (ThreadError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:185:in `mon_enter'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:209:in `mon_synchronize'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/imap.rb:941:in `idle_done'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/mailbox_watcher.rb:69:in `stop_idling'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/mailbox_watcher.rb:90:in `quit'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:22:in `each'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:22:in `quit'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:45:in `stop'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:34:in `block in start'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `call'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `sleep'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `run'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:41:in `start'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/bin/mail_room:5:in `<top (required)>'
from /usr/bin/mail_room:23:in `load'
from /usr/bin/mail_room:23:in `<main>'

Pulling some config from environment vars

Is it possible to have the mailroom.yml config file read from env vars? We'd rather not have credentials and tokens in our source tree.

StartTLS support

I'm not a ruby guy so I can't implement this myself (but it seems the change would be primitive). My mail server only supports StartTLS (it doesn't acceppt non-encrypted connections and 993 is closed). So to connect to it, the STARTTLS upgrade needs to be performed.

From what I gather, Net::IMAP#starttls is a simple command to execute after connecting so it would be cool to have an option to trigger this once the connection is established.

Address issues of raised errors on delivery crashing all of mail_room

MailRoom should not crash when a message or delivery raises an error. The mailbox watcher should be properly supervised such that it simply restarts on that mailbox. Let it fail, and make a new watcher.

Need to call expunge() or close() sometime to delete messages

I can't find anywhere that mail_room tells the IMAP server to actually delete messages flagged as DELETED; my understanding is that it would be by calling close() or expunge(). I can confirm that our gmail inbox has a large number (>200K) messages, nearly all marked DELETED but never actually expunged.

Am I missing something?

It's not possible to run mail_room over SSL, even if the host is localhost

/usr/local/lib/ruby/2.1.0/openssl/ssl.rb:240:in `post_connection_check': hostname "localhost" does not match the server certificate (OpenSSL::SSL::SSLError)
        from /usr/local/lib/ruby/2.1.0/net/imap.rb:1456:in `start_tls_session'
        from /usr/local/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:20:in `new'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:20:in `imap'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:69:in `setup'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:132:in `run'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/coordinator.rb:19:in `each'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/coordinator.rb:19:in `run'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/cli.rb:52:in `start'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/bin/mail_room:5:in `<top (required)>'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/mail_room:23:in `load'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/mail_room:23:in `<main>'

Configurable IMAP search command

By default we use Gmail IMAP's UNSEEN. It may be useful to configure this to be NEW in some cases.

Does it work with any IMAP server?

I’m quite confused. It uses IMAP, so it probably should work with any IMAP server (e.g. Dovecot), but the readme is full of Gmail… Please clarify this, Google is not the only IMAP server in the world.

Add a Prometheus metrics endpoint

Currently it's difficult to view the performance of the mail_room gem. If the gem emitted metrics, we can start measuring performance and other relatable data in order to alert Engineers if things are going wrong. This can also be useful as an endpoint could be utilized for monitoring the health of the service in general, whether it be a docker daemon or Kubernetes healthcheck.

Also consider that it may be wise to include metrics of monitoring the mailbox for which mail_room is configured to look at. This prevents the need for persons to stand up a secondary system such as https://github.com/camptocamp/imap-mailbox-exporter to monitor the mailbox.

run tests against redis 4 - fails with TypeError: no implicit conversion of Symbol into Integer

Failures:

  1) MailRoom::Delivery::Sidekiq#options when sentinel is specified client has same specified sentinel params
     Failure/Error: expect(redis.client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel

     TypeError:
       no implicit conversion of Symbol into Integer
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:346:in `establish_connection'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:106:in `block in connect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:306:in `with_reconnect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:105:in `connect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:381:in `ensure_connected'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:231:in `block in process'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:319:in `logging'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:230:in `process'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:125:in `call'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:222:in `block in client'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `block in synchronize'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `synchronize'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:221:in `client'
     # ./spec/lib/delivery/sidekiq_spec.rb:68:in `block (4 levels) in <top (required)>'

  2) MailRoom::Arbitration::Redis redis client connection params when sentinel is present client has same specified sentinel params
     Failure/Error: expect(client.client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel

     TypeError:
       no implicit conversion of Symbol into Integer
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:346:in `establish_connection'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:106:in `block in connect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:306:in `with_reconnect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:105:in `connect'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:381:in `ensure_connected'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:231:in `block in process'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:319:in `logging'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:230:in `process'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:125:in `call'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:222:in `block in client'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `block in synchronize'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `synchronize'
     # /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:221:in `client'
     # ./spec/lib/arbitration/redis_spec.rb:120:in `block (4 levels) in <top (required)>'

Finished in 0.0533 seconds (files took 0.35497 seconds to load)
51 examples, 2 failures

Failed examples:

rspec ./spec/lib/delivery/sidekiq_spec.rb:67 # MailRoom::Delivery::Sidekiq#options when sentinel is specified client has same specified sentinel params
rspec ./spec/lib/arbitration/redis_spec.rb:119 # MailRoom::Arbitration::Redis redis client connection params when sentinel is present client has same specified sentinel params

Randomized with seed 3049

IMAP should be properly disconnected when stopping mail room

Attempt to reduce scope of error catching during idle

Currently, if a disconnect occurs during idle, net/imap raises the superclass Error that is used for many other errors https://github.com/ruby/ruby/blob/trunk/lib/net/imap.rb#L3599-L3645 for more specific failures.

We only want to reset the connection if it's specifically a disconnect during idle.

tpitale / mail_room Goto Github PK

mail_room's People

Stargazers

Watchers

Forkers

mail_room's Issues

Recommend Projects

Recommend Topics

Recommend Org