IDLE support for listening to incoming mails about mailkit

[jstedfast]

Yea, that's something I haven't figured out yet. I've been busy focusing on other stuff.

Unfortunately IDLE doesn't quite work like every other IMAP command, so I'm not sure how best to handle it...

Feel free to take a stab at it.

[ziahamza]

I will probably send a pull request by a day or 2 :)

[ziahamza]

I have made some progress. here:
https://github.com/ziahamza/MailKit/compare/jstedfast:master...master
Its a blocking Idle function which blocks until the cancellation token is cancelled. The only problem is, which is also the case with all cancellationTokens in mailkit that when the token is cancelled, the operations still block until the tcp stream becomes readable or writable. So when the user cancels the Idle request, it doesnt finish until the server sends over some data. Stream.Read blocks.

The proper way would be to completely change all Stream.Read and Write calls to async Stream.ReadAsync and Stream.WriteAsync which accept CancellationTokens. One added benefit of this would be that the entire codebase will be proper async, rather than being async using a hack (starting a new thread). This may also improve perf as (if I understand correctly) the network async functions could use the underlaying polling syscalls (epoll) rather than creating a new thread and marshalling data back from a different thread.

this would be a big change-set though, and I wanted your take on this. Do you suggest any other route for this?

[ziahamza]

Other than the cancellation problem, the IDLE extention is mostly implemented I would say. A person could add event handlers to the MailFolder to get the untagged events, and call client.IDLE in a different thread. Then cancel the call after a while when necessary/

[ziahamza]

I decided to refactor the entire IMAP codebase anyways, into async functions all the way down. Removing the synchronous parts all together. Will update my fork when things are usable :)

[jstedfast]

I don't think that ReadAsync() is the solution for cancellation. This is Mono's implementation of Stream.ReadAsync():

public virtual Task<int> ReadAsync (byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
    if (cancellationToken.IsCancellationRequested)
        return TaskConstants<int>.Canceled;

    return Task<int>.Factory.FromAsync (BeginRead, EndRead, buffer, offset, count, null);
}

[jstedfast]

Also, I'd prefer to stick with .NET 4.0 if possible.

[ziahamza]

Hmm, thats a bummer. So cancellation token has pretty much no effect when there is a pending IO call. And yes, that would bump up the version of the framework to 4.5.

So basically, the only option left for the IDLE call might be to just let it hang until response comes by, or to dispose the connection completely from another thread and and build up the imap connection again which would seem very hacky. Actually, reading other IMAP implementations of Android Mail and reading about IOS mail app, they all maintain multiple IMAP connections in other threads, so an implementation in that regard would also solves this by having idle imap connection in another thread for events only.

I would implement it either way because I really want that feature. Which implementation, if any, would you prefer to merge in (should save me hassle of implementing either, haha)?

[jstedfast]

The patch I just committed plus the changes to ImapStream.ReadAhead() that I made a few hours ago I think will work...

I have not tested this yet, so I don't know if it works or not, but I think the approach will work...

[jstedfast]

I had a bug in the Socket.Poll() logic, but other than that... the code works fine. I just tested it.

Woot!

[ziahamza]

It works, brilliant!! Thanks a lot for the swift change!

[jstedfast]

I just came up with an even better way of doing it w/o the need for polling... :-)

The problem with polling is that it could break for SslStreams. It works for me in my testing against GMail, but since the SslStream buffers data, it could easily get into a state where we go to poll the socket and hang there indefinitely waiting for more data to arrive when in fact the SslStream already has data we can read from it (because it has remaining buffered data).

[ziahamza]

Hmm .. so I guess polling now happens at the underlying ReadAhead method to propagate the token cancel changes rather than the top layer. Neat!

[jstedfast]

Actually, the main thing I wanted to bring attention to is the fact that CancellationToken has a Register() method which I am now using as a way to register a callback that will send "DONE\r\n" to the server.

Using this approach, it doesn't matter if I block in a stream.Read() call in ReadAhead() (which will happen for SslStream since there's no way to determine if there is any more buffered data, so I can't poll the socket).

When the mail client decides to end the IDLE loop, it'll fire off the callback (in whatever thread the CancellationTokenSource is being used from), sending the "DONE\r\n", which will force the server to send more data, freeing up the Read() that is blocking.

[ziahamza]

aah, thats clever! Now it make sense. That will indeed force server to send in more data. I didnt catch that before. Thanks for the update!! :)

[jstedfast]

Figured you might be interested in that change :-)