jeads / datasource Goto Github PK

It appears that I need to create table named data_sources_test_data to run tests. It'll be nice if we had a wrapper script that does this automatically. Also, it appears to me that we should let developer specify username & password as in:

run-tests.py --host=localhost --username=test-user --password=passwd.

Given that we already have ability to pull in custom datasource.json, it should be fairly easy to generate this json file automatically given username, password, & etc...

Datasource requires MySQL-python, which isn't compatible with Python 3, and also not maintained. However it's been forked, and the fork (repo) does support Python 3.

After switching to the fork, there are then errors like:

  File "/home/travis/virtualenv/python3.5.1/lib/python3.5/site-packages/datasource/bases/BaseHub.py", line 342, in <module>
    os.path.walk(procs_path, BaseHub.load_builtin_procs, {})
AttributeError: module 'posixpath' has no attribute 'walk'

We should:

1. Fix errors under Python 3.5
2. Add Python 3.5 to the Travis run
3. Add the PyPI Python 3 classifier to setup.py to indicate support

In https://bugzilla.mozilla.org/show_bug.cgi?id=1125464#c8 it was found that we were using the master_host for everything, even though we had thought we had specified read_host for the appropriate selects in the SQL proc file.

Unfortunately it turned out we were using the property 'host' rather than 'host_type'., eg:

"get_performance_series": {

    "sql":"SELECT `interval_seconds`, `series_signature`, `type`, `last_updated`, `blob`
           FROM `performance_series`
           WHERE `interval_seconds` = ? AND `series_signature` = ?",

    "host":"read_host"

},

This issue would have been much easier to spot, if Datasource validated the keys and rejected any that it did not recognise.

Alternatively, if that is not something you wish to do - having a way to opt out of the default_host_type would mean we could catch any cases where we were not explicitly passing the desired host_type (possibly a good feature in its own right).

There should probably be some easy way to set the configuration for running tests, without modifying a file that's checked-in to git.

python setup.py install
Traceback (most recent call last):
File "setup.py", line 20, in
long_description=read('README'),
File "setup.py", line 10, in read
return open(os.path.join(os.path.dirname(file), fname)).read()
IOError: [Errno 2] No such file or directory: 'README'

mv README.md to README worked to finish install :)

Currently if host_type is not set, Datasource falls back to the default of "master_host".

It would be great to optionally allow the caller to disable this fallback, so accidental omissions can be spotted, to avoid unnecessary master host load.

This would have been another way to avoid the problems in #20

Hi @jeads hope you are well :-)

Could you switch Travis on for this repo using the switch at https://travis-ci.org/profile please?
The .travis.yml config is in place so it should just work - I'm just not allowed to turn it on myself, not being the owner.

Thanks :-)