Testing Strategy about ansible-collection-smallstep

[maxhoesel]

I've added tests for the roles for now, but everything else is still up for debate: #27

[maxhoesel]

Added integration and sanity tests for the modules in 8fe4b19. This should cover most of the issues that might arise with this collection. Unit tests for modules are still an option for the future, so i'll leave this issue open, in case anyone has a suggestion for how to approach that

[eengstrom]

Perhaps this is the wrong place for this, but I'm trying to follow your current README.md and CONTRIBUTING.md docs and I'm unable to get the testing to work on what is currently the main branch. Is the right invocation to simply run ./test.sh? If yes, then I end up with an error on the main branch with no modifications:

TASK [step_ca_provisioner_claims : Check server health] ************************
fatal: [testhost]: FAILED! => {"changed": false, "cmd": ["step-cli", "ca", "health"], "delta": "0:00:00.018545", "end": "2021-07-09 21:00:08.083501", "msg": "non-zero return code", "rc": 1, "start": "2021-07-09 21:00:08.064956", "stderr": "The certificate authority encountered an Internal Server Error. Please see the certificate authority logs for more info.\nRe-run with STEPDEBUG=1 for more info.", "stderr_lines": ["The certificate authority encountered an Internal Server Error. Please see the certificate authority logs for more info.", "Re-run with STEPDEBUG=1 for more info."], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
testhost                   : ok=16   changed=9    unreachable=0    failed=1    skipped=6    rescued=0    ignored=0

NOTICE: To resume at this test target, use the option: --start-at step_ca_provisioner_claims
NOTICE: To resume after this test target, use the option: --start-at step_ca_renew
ERROR: Command "ansible-playbook step_ca_provisioner_claims-r09nnp99.yml -i inventory -v" returned exit status 2.
Run command: docker exec 7dfde1d26befb42b9489c1dd6537e32657ddf88e56458aa04a707222a5dac80a tar czf /root/output.tgz --exclude .tmp -C /root/ansible_collections/maxhoesel/smallstep ...
Run command: docker exec -i 7dfde1d26befb42b9489c1dd6537e32657ddf88e56458aa04a707222a5dac80a dd if=/root/output.tgz bs=65536
Run command: tar oxzf /var/folders/79/7jgnn_hn7l5c3178vw7_m8v00000gp/T/ansible-result-abl5ovb1.tgz -C /Users/engstrom/.ansible/collections/ansible_collections/maxhoesel/smallstep ...
Run command: docker rm -f 7dfde1d26befb42b9489c1dd6537e32657ddf88e56458aa04a707222a5dac80a
ERROR: Command "docker exec 7dfde1d26befb42b9489c1dd6537e32657ddf88e56458aa04a707222a5dac80a /usr/bin/env ANSIBLE_TEST_CONTENT_ROOT=/root/ansible_collections/maxhoesel/smallstep LC_ALL=en_US.UTF-8 /usr/bin/python3.6 /root/ansible/bin/ansible-test integration -v --python 3.6 --metadata tests/output/.tmp/metadata-d97o1bac.json --truncate 183 --redact --color yes --requirements --allow-destructive" returned exit status 1.
ERROR: InvocationError for command /Users/engstrom/Projects/sysops/ansible/dev/collections/ansible_collections/maxhoesel/smallstep/.tox/ansible/bin/ansible-test integration --requirements --docker --color -v --python 3.6 (exited with code 1)
_______________________________________________________________________________________ summary _______________________________________________________________________________________
ERROR:   integration: commands failed

My development environment is macos, Catalina, fully patched, updated docker, brew, etc...

[maxhoesel]

Yes, just running test.sh should work fine. That said, there's definitely something wrong with the tests right now - I just re-ran the CI workflow and it fails with the same error message. It ran just fine a few days ago. My best guess is that one of test dependencies got updated and broke the workflow. Going to take a look at it tomorrow.

Also, thanks for all the Issues and PRs! I'll try to get to them in the coming days, but i'm a bit busy IRL right now. These improvements are very much appreciated!

EDIT: Issues with the CI have been resolved