mozilla-services / services-test Goto Github PK

We need to add a test-catalog.ini to services-test.

The test-catalog.ini in dev would "activate" the jenkins-job-manager to create a corresponding job for every test listed in the catalog in the jenkins-dev.
Likewise, a test-catalog.ini file in the master branch, would activate corresponding jobs for every test listed in the master jenkins.
Forcing contributors to explicitly add their tests to this list would permit contributors to retain tests within services-test that still can be housed in github, but would not have a corresponding Jenkins job.

Example:
https://github.com/rpappalax/jenkins-job-manager/blob/dev/test_catalog.ini

flag lines with legit >80 length, then remove E501

script downloads to maintenance workspace and setups up symlink

please include:

#!/usr/bin/env python
#!/usr/bin/env node
#!/usr/bin/env cobol

so then we can also leave off the file extension.
Since all our jenkins jobs are generated dynamically (and we can't assume one language), let's just drop the extension altogether.

For treeherder integration, we will probably need to go with either python or node:
see: https://bugzilla.mozilla.org/show_bug.cgi?id=1213542#c8

README should also be updated accordingly

seems like largely converting print to print()

https://bugzilla.mozilla.org/show_bug.cgi?id=1206254

@chartjes would these be schema-checks?

https://github.com/mozilla-services/services-test/tree/dev/normandy/study_tests

Currently we have a couple "floating" _temp folders

1. the root /_temp/ folder is used for temporary Firefox Profiles (which we can clean up in the nightly refresh job).
2. We have a /_utils/_temp/ folder is used for temporary Firefox binaries (added by #55 in /_utils/firefox_download.py:41)

We should maybe consider using /_temp/profiles/ and /_temp/firefoxes/, or something.
This will keep everything close, and not have a bunch of random temporary folders peppered throughout the repo.

We have some dummy code for this in print statements.
Need to flesh it out

STR:

1. Go to https://github.com/mozilla-services/services-test
2. Click 'example run file' or 'example manifest.ini'

Expected results:
See docs

Actual results:
404

Looking at ./_shared/install_ff_nightly.sh, it may be slightly simpler if we use the lovely firefox-get npm module (assumes the host images all have Node.js installed).

$ curl $(firefox-get -v beta) | tar xz --directory ff_beta

NOTE: This assumes you already have an ./ff_beta/ directory.

We could also possibly abstract this one step further by creating some UNIX-y function where we pass a target "version" of Firefox that we want to download and it will ensure the directory is created for us.

Then we can do one/all of the following:

• $ download_firefox release
• $ download_firefox beta
• $ download_firefox nightly

From there we can create some ENV variables to point to the different Firefox versions, where $FF_BETA points to ./ff_beta/firefox/firefox-bin.

mozilla/mozdownload#345

Mozdownload now fixed for OSX. we should look at replacing our custom one:
https://github.com/mozilla-services/services-test/blob/dev/_utils/firefox_download.py

also add more to manifest.ini for job definition

right now firefox_downloads.py will download all new version of FF for the Jenkins slave env.
This tool would be more useful for general testing if we could specify specific version via args.

Various projects have inert run(.sh) files that were added as a "first attempt" at scoping out how services-test would operate. Now that we have a working architecture, these should all be removed.

• cd /home/jenkins-slave/workspace/sync_e2e-test_prod/mozilla-central/testing/tps
  ././sync/e2e-test/run: line 30: cd: /home/jenkins-slave/workspace/sync_e2e-test_prod/mozilla-central/testing/tps: No such file or directory
• exit 1

We don't need any of the files that are now in ff-tool and should remove them from here
https://github.com/mozilla-services/services-test/tree/dev/_utils

For ease-of-use, we should probably consider moving the download, install, uninstall and profile manager mods bundled into their own package w/ 1 or 2 console entry points. It would be easiest if we start with 1 menu of args and see if that looks practical. I looked through quite a few other github repos around firefox and found some other partial attempts at this, but I think if we put our tools together, it might be fairly useful or others outside of our team as well.

INSTALLERS
https://github.com/chef-cookbooks/firefox
https://github.com/bellpeterm/firefox
https://github.com/zanardo-ansible/firefox
https://github.com/boxen/puppet-firefox

HACKS
https://github.com/omgmog/install-all-firefox/blob/master/install-all-firefox.sh

PROFILE MANAGERS
https://github.com/Puppet-Finland/firefox

We have some dummy code for this in print statements.
Need to flesh it out

MSISDN-Gateway tests no longer needed

https://bugzilla.mozilla.org/show_bug.cgi?id=1262454

moved to:
https://github.com/rpappalax/autopush-loadtests

@chartjes looks like no further deploys for loop? If so, can archive the tests in a release and or tag?

Add a directory to store tests that are checking the Normandy API responses for valid schema and properties

[DEFAULT]
browser.safebrowsing.blockedURIs.enabled = true

and add: mozplugin2-block-digest256" to urlclassifier.blockedTable

This is the Opengraph test I have finished and have been requested to merge here.

/msisdn-gateway/e2e-test/fabfile.py:1-2 has the following:

from fabric.api import *
import fabric

Flake8 doesn't like dis, and complains:

• ./msisdn-gateway/e2e-test/fabfile.py:1:1: F403 'from fabric.api import *' used; unable to detect undefined names
• ./msisdn-gateway/e2e-test/fabfile.py:2:1: F401 'fabric' imported but unused

We should try and improve the glob import to avoid flake8 errors and avoid having to do weird hacks like adding # noqa to get flake8 to ignore our code.

Ref: https://github.com/mozilla-services/services-test/blob/dev/sync/e2e-test/manifest.ini

True vs true
cron formatting
prod vs production
job_timeout must be integer
required fields present
manifest file must define at least 3 envs (dev, stage, prod)

Me thinks we can use Travis-CI to do this, although not sure of the optimal way. I can try experimenting with this.

Was also looking at Tox since I've seen that used in other projects, but not sure if that is overkill.

As for checking JavaScript files, we could use ESLint, but that may get tricky if we plan on excluding oodles of files/folders (in which case we may need Gulp or Grunt, and we're starting to add a few dependencies to our Travis env).

Probably best to just dive in and start doing a bunch of PRs and see what falls out.

We can now use ff-tool to e2e-test shavar, but will need to first convert and add shavar prefs files.

TODO:
figure out how to handle multiple pref sets

The deployments-tests look good, but should prob be named consistently per the services-test README to: "schema check":
https://github.com/mozilla-services/services-test/blob/dev/README.md

@chartjes on the other hand, would it make more sense to create a more descriptive test type for tests like these called "config-check"?

https://github.com/mozilla-services/services-test/search?utf8=%E2%9C%93&q=virtualenv&type=Code

We're doing virtualenv . in a couple places, which can make things a bit messy and hard to exclude when trying to grep or use globs in certain tools.

This test was a proof-of-concept that no longer does anything.