Automated testset about mailu

[kaiyou]

I do not agree with running tests inside the Mailu container network. Email is tightly related to network services being properly exposed, spam management and tests would need to try and exchange emails with external services.

I would prefer a testing suite that runs from a separate user and simulates real users that indeed access Mailu from anywhere over the Internet. It could also interact with a couple test accounts from various providers.

[muhlemmer]

During the works from the last 2 weeks, I was already pretty annoyed about manual testing different setups before sending a PR. This is not a complaint, just a personal experience. I tried to set up Docker auto-builds for my own fork. but since all Dockerfiles reside in one Github tree, every directory and branch is to be specified. This is not feasible if I deploy a quick testing branch to try some options.

@kaiyou: For the most tests we don't really need a full real-world simulation. You'd just want to see if an updated query is still performing, if an update script is not breaking the install, if postifx can still receive and send mail etc etc. It makes life more easier for people to contribute and for you to accept pull requests. Implementing something local would still be better what there is now: nothing. Close to a final version, release an alpha and beta version for real-world testing by the users.

My 2 cents:

TravisCI

The mail advantage, .travis.yml recides in the repository. Any contributor would be able to set up a travisCI account and test their own setups more easy.

Features:

1. Docker testing is supported and documented. Inlcuding support for docker-compose.
2. We can write Selenium scripts for web interface testing, and hook them up to Sauce labs through travis. It's free for open source. (Covers the advanced topics in the original issues)
3. All PR's at least passed some kind of testing before merging, hopefully taking out the majority of errors.

@HorayNarea already proposed to start setting something up. We can start simple, just let travisCI to build all Pull requests. We can extend this step by step with testing scripts. I'll be willing to assist as well. I guess we just need your (@kaiyou) okay.

Define the testing scope

What is Mailu's testing scope? Upstream also has their testing suites. It would be pointless for us to test all the various webmail functions. Just the Mailu-specific settings. Same goes for the other packages.

Beta-testing

I can see that this project is picking up popularity. Let's use the crowd to do the real-world testing. We can write up a QC-like testing procedure and a kind of reporting standard, so that beta testers will report something useful and not just "x does not work anymore".

Development policy

Once we get to a reasonable stage of the testing suite, you could enforce policies to keep the testing up-to-date. If a PR includes new functionality, it should have also the proper documentation and testing script included.

[pesimon]

I think we need to decide on a testing framework first. To me a separate testing contianer which runs inside the mailu container network and tests services (integration/acceptance test).

I did a small poc with a ruby-container and rspec test based on https://github.com/wildbit/MailHandler It sends an Email via smtp and checks receiving via IMAP.

[benyanke]

Beta testing

An additional piece of this would be actually knowing there's a beta to test. If we're prepping for a release, is there a place to be notified?

[kaiyou]

@muhlemmer you have my definite okay about starting a Travis build and build up from there with testing scripts. Regarding building, the Compose file comes in really handy with building everything from the single repo.

I will setup a Travis org right now and link it to this repo. I was thinking about making it simply public, what do you think?

[kaiyou]

Well I even had forgotten I had a Travis project setup already for updating the docs website. I am not well aware of the technics to build multiple pipelines using Travis, but a PR is more than welcome for basic builds

[muhlemmer]

So, we have a basic test set for now. We will work hard to extend it's features. But for now I would like to mark is as completed in the 1.6 milestone. Keep this issue open though

[kaiyou]

This is largely implemented recently