Coder Social home page Coder Social logo

imap-d's Introduction

Build Status Build status Coverage

IMAP for D / SIL and JMAP for SIL

Status - usable alpha

I am starting to use this daily, but there are probably many bugs - please file issues. PRs welcomed.

IMAP Features that should work (ones without a check may work but be more awkward to use)

  • session capabilities
  • copy message
  • create mailbox
  • createQuery (higher level API for IMAP search)
  • delete_ mailbox
  • esearch
  • examine
  • expunge
  • fetchFast
  • fetchFields
  • fetchFlags
  • fetchHeader
  • fetchPart
  • fetchRFC822 - whole email fetch and parse
  • fetchSize
  • fetchStructure
  • fetchText
  • idle
  • list
  • login
  • logout
  • lsub
  • move
  • moveUIDs
  • multiMove
  • multiSearch - search multiple mailboxes (sadly not widely supported on IMAP servers)
  • noop
  • raw IMAP command
  • search
  • searchQuery
  • select
  • status
  • store
  • subscribe
  • unsubscribe

JMAP Features that should work (ones without a check may work but have not yet been tested)

  • Email
  • Contacts
  • Calendar

Heavily inspired by imapfilter.

Limitations

  • currently IMAP literals are only parsed for returns of type responseBody
  • only fetchRFC822 works for parsing results (works if you want whole email including attachments)
  • more abstraction and better design needed for parsing
  • synchronous and single threaded: use processes to start multiple connections for performance
  • JMAP depends on internal Symmetry Variable type and will require changes to work without it

Features to add

  • more IMAP extensions
  • more JMAP capabilities

Example use

import imap;

int main(string[] args) {
    import std.conv : to;
    import std.stdio : writeln, stderr;
    import std.range : back;
    import std.process : environment;
    import std.string : join;
    import std.datetime : Date;

    if (args.length != 4) {
        stderr.writeln("imap-example <server> <port> <mailbox>");
        return -1;
    }

    // user is set in IMAP_USER environmental variable
    // password is set in IMAP_PASS environmental variable

    auto user = environment.get("IMAP_USER", "");
    auto pass = environment.get("IMAP_PASS", "");

    if (user.length == 0 || pass.length == 0) {
        stderr.writeln("IMAP_USER and IMAP_PASS environment variables must be set.");
        return -1;
    }

    auto server = args[1];
    auto port = args[2];
    auto mailbox = args[3];

    auto login = ImapLogin(user, pass);
    auto imapServer = ImapServer(server, port);

    auto session = new Session(imapServer, login);
    session.sslProtocol = ProtocolSSL.ssl3;
    session.options.debugMode = true;
    session = session.openConnection;
    session = session.login();

    // Select Inbox
    auto INBOX = new Mailbox(session, mailbox);
    auto result = session.select(INBOX);

    // search all messages since 29 Jan 2019 and get UIDs using raw query interface
    auto searchResult = session.search("SINCE 29-Jan-2019");
    writeln("--- Raw 'SINCE 29-Jan-2019' search results:");
    writeln(searchResult.value);
    writeln("--- Search result IDs: ", searchResult.ids);

    // search all messages from GitHub since 29 Jan 2019 and get UIDs using high level query interface
    auto query = new SearchQuery()
        .and(DateTerm(DateTerm.When.Since, Date(2019, 1, 29)))
        .and(FieldTerm(FieldTerm.Field.From, "GitHub"));
    searchResult = session.search(query.to!string);
    writeln("--- Structured 'since:Date(2019,1,29),fromContains:\"GitHub\"' search results:");
    writeln(searchResult.value);

    // fetch one of the messages from above
    if (searchResult.ids.length > 0) {
        auto exampleId = searchResult.ids.back.to!string;
        auto messageResult = session.fetchText(exampleId);
        writeln("--- Message text for ID ", exampleId, ":");
        writeln(messageResult.value);

        // just fetch the fields we care about
        auto relevantFields = ["FROM", "TO"];
        auto fieldsResult = session.fetchFields(exampleId, relevantFields.join(" "));
        writeln("--- Message 'from' and 'to' fields for ID ", exampleId, ":");
        writeln(fieldsResult.value);
    }
    return 0;
}

imap-d's People

Contributors

9il avatar adamdruppe avatar john-colvin avatar kaleidic avatar kinke avatar laeeth avatar maxhaton avatar otrho avatar rmanthorpe avatar skoppe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

imap-d's Issues

Session API is a bit verbose.

The API for this library revolves around the imap.Session. It holds the context for an active connection to a server and is needed for the majority of the library methods, which all take a mutable reference to it in case it needs to be updated.

In D this fine and most of the time you'd use UFCS to call e.g., session.select(mailbox). Looks good.

In SIL however, the wrapping will see the ref Session and produce an out parameter for it. The wrapped methods return a pair of an 'out parameter' and a 'return value'. This ends up being quite verbose in practice and we almost never want to bother with the Session out parameter as it's really just an opaque handle. Strictly speaking though we should be taking it and passing it on in subsequent calls.

Ideally I'd like to have the session as an opaque handle which can be passed around, and in the SIL context is essentially immutable. If the Session was in fact a pointer then it would be an opaque immutable handle, but I'm not sure if this is viable or desirable.

Actually, I'm not sure what the best approach to improve this problem is. In essence, I'd like to create the session (per server connection) and pass it around like a handle, and to not have to worry about using the verbose and ugly .returnValue property after each API call.

imap-d should implement IMAP extensions

  • LITERAL+
  • ID
  • ENABLE
  • ACL
  • RIGHTS=kxten
  • QUOTA
  • MAILBOX-REFERRALS
  • UIDPLUS (partial implementation so far)
  • NO_ATOMIC_RENAME
  • UNSELECT
  • MULTIAPPEND
  • BINARY
  • CATENATE
  • CONDSTORE
  • SEARCH=FUZZY
  • SORT
  • SORT=DISPLAY
  • SORT=UID
  • THREAD=ORDEREDSUBJECT"
  • THREAD=REFERENCES
  • THREAD=REFS
  • ANNOTATE-EXPERIMENT-1
  • METADATA
  • LIST-EXTENDED
  • LIST-STATUS
  • LIST-MYRIGHTS
  • LIST-METADATA
  • WITHIN
  • QRESYNC
  • SCAN
  • XLIST
  • MOVE (should be implemented)
  • SPECIAL-USE
  • CREATE-SPECIAL-USE
  • DIGEST=SHA1
  • X-REPLICATION
  • STATUS=SIZE
  • OBJECTID
  • SAVEDATE
  • X-CREATEDMODSEQ
  • PREVIEW=FUZZY
  • XAPPLEPUSHSERVICE
  • LOGINDISABLED
  • XCONVERSATIONS
  • COMPRESS=DEFLATE
  • X-QUOTA=STORAGE
  • X-QUOTA=MESSAGE
  • X-QUOTA=X-ANNOTATION-STORAGE
  • X-QUOTA=X-NUM-FOLDERS
  • XMOVE

Keepalive

If we don't implement we should implement HTTP keepalive

We have two Travis integrations?

I'm not sure whether I accidentally did anything, but it looks to me like we have two integrations with Travis-CI. One with .com and one with .org.

Apparently the .org used to be for public repos and the .com for (what used to be paid for) private repos. And they're going to turn the .org site off at the end of the year. So I guess we don't really have to do anything.

But right now it's doing double the work on Travis. I think it should be possible to go into the settings and change things on an 'integrations' tab, but I'm not seeing it there. Maybe I don't have permissions?

Remove SIL configs from dub.sdl.

Having the SIL registration code in this repo introduces dependencies on sil-lang which are marked as optional in dub.sdl because there's a dub bug or something if they're not.

Ideally we should remove all the SIL stuff from here and keep it as just an IMAP D library and the SIL stuff can go in a dedicated plugin library hosted elsewhere which depends on this one.

Improve TLS functionality.

Right now the TLS code via libssl works but is not quite finished. In particular server cert verification is not enabled, mostly due to it being a bit hard on Windows (or actually, I've never done it on Windows and am not sure how to make it work).

Also, it might be worthwhile making it extremely hard to use--or flat out banning--unencrypted traffic since it's 2020 and all.

Automatically reconnect

Might be I didn't set the option but if we dont implement already should be able to have a parameter that makes it reconnect and retry if socket closes.

Useful for imap idle

Reading from the socket tends to block.

Much of the response reading/parsing code uses a default timeout of 20 seconds (from session.d: Options.timeout) but I'm seeing the code just blocking if we try to read when there's no data.

Right now e.g., in append() when reading a continue response but getting a TRYCREATE it doesn't parse properly and blocks. I'll fix that specifically but there are probably many places in the reading/parsing which could block in a similar way (even with a timeout, which is strange.)

SearchQuery high level struct interface needs refining

working

  • for boolean OR, pass an array of SearchQuery
  • for boolean AND, set conditions within SearchQery
  • for boolean NOT, set not in SearchQuery to true: this will NOT all conditions within the struct

missing

  • no way to specify A and NOT B.

SSL socket error

Sometimes I get this error on longrunning process using IDLE. More info to come.

mailrules.service: Main process exited, code=exited, status=255/EXCEPTION
Error: reading data through SSL; EOF in violation of the protocol

CI bitrot

4 month ago CI still passed, but a recent run has certificate and compiler issues.

I propose to switch to github actions.

Timeouts

Aren't set up properly between secure and regular sockets

Ditch AppVeyor

In #102 @maxhaton moved from travis over to github actions.

The moment the request was merged, the CI on master failed for appveyor. Since github actions supports windows as well I figured we can migrate completely to it.

I opened #103, but it doesn't quite work. On windows it complains about missing libssl. I see there is already an issue #96 for it.

I noticed in the appveyor config we build the example using the appveyor config.

We should probably include that at the root library.


Another reason for ditching appveyor is the complex config file as well as the fact that it is currently setup to run on @otrho account.

Dogfooding

Could @otrho @jamtho @John-Colvin write their email rules in SIL. I have a systemd service that restarts SIL when occasionally the socket is closed.

But after the outstand pull requests have been accepted for me it sort of works to use SIL for IMAP rules using IDLE mode.

Value comes when we can have functions that categorise emails received across the firm and default ways of handling such messages.

Linking on Windows

To link on Windows it needs libssl and libcrypto , so the libs needs extending:

libs "ssl" "crypto" platform="posix"
libs "libssl" "libcrypto" platform="windows-x86_64"
libs "libssl" "libcrypto" platform="windows-x86_mscoff"

This requires the libs to be on the linker path, which, when building as a SIL plugin requires lflags to be extended:

configuration "plugin" {
    . . .
    lflags "/LIBPATH:$KAL_EXT_LIB_WIN64" platform="windows-x86_64"
    lflags "/LIBPATH:$KAL_EXT_LIB_WIN32" platform="windows-x86_mscoff"
}

FETCH only supports UID.

For MR37 one of the commits introduced an optional way to allow message sequence IDs to be used for FETCH requests.

The RFC specifies that message UIDs must be numbers, if I'm reading it correctly, especially since they must increment in value for newer messages.

In imap-d the message IDs are strings as this is simpler than casting back and forth from integers and it simpler for SIL. I suggested that for the FETCH calls in the API the message ID could be prefixed with a '#' to indicate that it should be interpreted as a message sequence ID (i.e., 1 for the first message in the mailbox, 2 for the second...), and a missing '#' just indicates a UID as before. Since UIDs must be numbers there can't be a conflict and so this should be backwards compatible and work fine.

Selecting numbers by message sequence ID is useful -- I wanted it to simply iterate through the messages in a mailbox. The alternative is to either do a blanket search to fetch all the UIDs in a mailbox, or do to a FETCH UID n for each message before the UID FETCH ... itself, which wasteful.

If the '#' proposal is too on-the-nose then I guess the alternative is to have fetchByUid or fetchBySeqId alternatives for each of the FETCH functions, which feels a bit verbose and wouldn't suit SIL. Or perhaps to have an extra parameter indicating the type of ID or to use a special type for the ID which indicates whether it's UID or SeqID.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.