yhat / bandit Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://yhat.github.io/bandit/
Home Page: https://yhat.github.io/bandit/
Bandit doesn't have very user-friendly default metadata values.
This makes it hard for the user to conceptualize what these features are useful for or how they are intended to be used.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""new_bandit_instance.py
"""
from bandit import Bandit
BANDIT_API_KEY = os.environ['BANDIT_API_KEY']
BANDIT_URL = 'http://00.000.000.000/'
bandit = Bandit("austin", BANDIT_API_KEY, BANDIT_URL)
# newly created bandit instance should have more logical starting values
# Useful example to go by would be pandas
# import json; import pandas as pd; print json.dumps(vars(pd.options), indent=2)
# FOR EXAMPLE...
bandit.job # should return None
bandit.email # should return None
# bandit has several `set_{{value}}()` type methods
# therefore it must also have an equal number of `get_{{value}}()` properties
bandit.job.get_status() # should throw error
bandit.job.get_name() # should return None
bandit.job.properties # should return {"id": {{random_id_hash}}}
As a user, I want to be able to assign arbitrary data properties to my bandit jobs using python dot notation.
for example, i might want to record statistical performance metrics for a model like this:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""classification_algorithm_evaluation_metrics.py
* Classification Accuracy
* Logarithmic Loss
* Area Under ROC Curve
* Confusion Matrix
* Classification Report
"""
# Cross Validation Classification Accuracy
import sys
import os
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from bandit import Bandit
BANDIT_API_KEY = os.environ['BANDIT_API_KEY']
BANDIT_URL = 'http://00.000.000.000/'
bandit = Bandit("austin", BANDIT_API_KEY, BANDIT_URL)
colnames = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
df = read_csv('pima-indians-diabetes.data.csv', names=colnames)
feature_names = df.columns[df.columns!='class'].tolist()
target_name = 'class'
bandit.metadata.description = """For classification metrics, the Pima Indians onset of diabetes dataset is used as demon- stration.
This is a binary classification problem where all of the input variables are numeric.
"""
bandit.metadata.feature_names = feature_names
bandit.metadata.target_name = target_name
X = df.ix[:, feature_names]
y = df[target_name]
# use kfold
kfold = KFold(n_splits=10, random_state=7)
# tell bandit about your cross val method
bandit.metadata.cross_val.method = "kfold"
bandit.metadata.cross_val.n_splits = 10
bandit.metadata.cross_val.random_state = 7
# use logistic regression
model = LogisticRegression()
# tell bandit about your model
bandit.metadata.model_name = "glm-pima-indians-diabetes"
# compute arbitrary statistics about your model performance
results = cross_val_score(model, X, y, cv=kfold, scoring='accuracy')
# tell bandit about your metrics
# NOTE before i run the next line, metadata.metrics should return None
# print json.dumps(dict(bandit.metadata), indent=2)
# {}
bandit.metadata.metrics.accuracy.mean = results.mean()
bandit.metadata.metrics.accuracy.std = results.std()
# after i tell bandit that i want a substructure called metadata.metrics, this is what it should look like
# print json.dumps(dict(bandit.metadata), indent=2)
# {
# "metrics": {
# "std": 0.048410519245671947,
# "accuracy": {
# "mean": 0.76951469583048526
# }
# }
# }
# bandit should complain if you try to put anything that's NOT json serializable into bandit.metadata
steps:
observe:
Here's the UI
And here are the logs i see when i click on the one that i've pointed to with the arrow
��Requirement already satisfied: pandas==0.19.1 in /root/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 2))
�=Collecting tweepy==3.2.0 (from -r requirements.txt (line 3))
�0 Downloading tweepy-3.2.0-py2.py3-none-any.whl
��Requirement already satisfied: psycopg2==2.6.2 in /root/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 4))
��Requirement already satisfied: python-dateutil in /root/anaconda/lib/python2.7/site-packages (from pandas==0.19.1->-r requirements.txt (line 2))
��Requirement already satisfied: pytz>=2011k in /root/anaconda/lib/python2.7/site-packages (from pandas==0.19.1->-r requirements.txt (line 2))
�ICollecting six==1.7.3 (from tweepy==3.2.0->-r requirements.txt (line 3))
�- Downloading six-1.7.3-py2.py3-none-any.whl
�NCollecting requests==2.4.3 (from tweepy==3.2.0->-r requirements.txt (line 3))
�: Downloading requests-2.4.3-py2.py3-none-any.whl (459kB)
�WCollecting requests-oauthlib==0.4.1 (from tweepy==3.2.0->-r requirements.txt (line 3))
�; Downloading requests_oauthlib-0.4.1-py2.py3-none-any.whl
�hCollecting oauthlib>=0.6.2 (from requests-oauthlib==0.4.1->tweepy==3.2.0->-r requirements.txt (line 3))
�, Downloading oauthlib-2.0.1.tar.gz (122kB)
�1Building wheels for collected packages: oauthlib
�5 Running setup.py bdist_wheel for oauthlib: started
�I Running setup.py bdist_wheel for oauthlib: finished with status 'done'
�k Stored in directory: /root/.cache/pip/wheels/e7/4e/b5/c36a97caf4408037eb35aeb1e0a80d95ea0c521a562c3488a3
�nSuccessfully built oauthlib
Installing collected packages: six, requests, oauthlib, requests-oauthlib, tweepy
�* Found existing installation: six 1.10.0
�� DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
�� Uninstalling six-1.10.0:
�* Successfully uninstalled six-1.10.0
�/ Found existing installation: requests 2.11.1
�" Uninstalling requests-2.11.1:
�/ Successfully uninstalled requests-2.11.1
�dSuccessfully installed oauthlib-2.0.1 requests-2.4.3 requests-oauthlib-0.4.1 six-1.7.3 tweepy-3.2.0
�L[Errno 2] No such file or directory: '/jobs/output-files/tweets-to-load.csv'```
User should be able to invoke bandit.run("job_name")
to kick-off (aka to run) an existing bandit job.
steps:
"my_job_000001"
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""run_existing_job.py
"""
import sys
import os
from bandit import Bandit
BANDIT_API_KEY = os.environ['BANDIT_API_KEY']
BANDIT_URL = 'http://54.201.192.120/'
bandit = Bandit("austin", BANDIT_API_KEY, BANDIT_URL)
print bandit.run(" hello") # job which previously completed successfully in the UI
results
$ python run_existing_job.py
<html>
<head>
<title>Bandit</title>
<link href="/css/bulma.css" rel="stylesheet" />
<link href="/css/font-awesome.css" rel="stylesheet" />
</head>
<body>
<section class="hero is-bold is-dark is-fullheight">
<div class="hero-body">
<div class="container has-text-centered">
<div class="columns is-vcentered">
<div class="column is-5">
<figure class="image is-4by3">
<img src="http://placehold.it/800x600" alt="Description">
</figure>
</div>
<div class="column is-6 is-offset-1">
<h1 class="title is-2">
Whoops!
</h1>
<h2 class="subtitle is-4">
The page you're looking for doesn't exist!
</h2>
<br>
<p class="has-text-centered">
<a href="/" class="button is-dark is-inverted is-large">
<span class="icon">
<i class="fa fa-home"></i>
</span>
<span>Take Me Home</span>
</a>
</p>
</div>
</div>
</div>
</div>
<!-- Hero footer: will stick at the bottom -->
<!-- <div class="hero-foot">
<nav class="tabs is-boxed is-fullwidth">
<div class="container">
<ul>
<li class="is-active"><a>Overview</a></li>
<li><a>Modifiers</a></li>
<li><a>Grid</a></li>
<li><a>Elements</a></li>
<li><a>Components</a></li>
<li><a>Layout</a></li>
</ul>
</div>
</nav>
</div> -->
</section>
</body>
</html>
Expected:
Observed
In [40]: bandit.get_jobs()
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
<ipython-input-40-270c11c54d58> in <module>()
----> 1 bandit.get_jobs()
/usr/local/lib/python2.7/site-packages/bandit/bandit.pyc in get_jobs(self)
73 """
74 r = requests.get(self.url + "/jobs", auth=(self.username, self.apikey))
---> 75 jobs = r.json()['jobs']
76 return [Job(**j) for j in jobs]
77
/usr/local/lib/python2.7/site-packages/requests/models.pyc in json(self, **kwargs)
824 # used.
825 pass
--> 826 return complexjson.loads(self.text, **kwargs)
827
828 @property
/usr/local/lib/python2.7/site-packages/simplejson/__init__.pyc in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
514 parse_constant is None and object_pairs_hook is None
515 and not use_decimal and not kw):
--> 516 return _default_decoder.decode(s)
517 if cls is None:
518 cls = JSONDecoder
/usr/local/lib/python2.7/site-packages/simplejson/decoder.pyc in decode(self, s, _w, _PY3)
372 if _PY3 and isinstance(s, binary_type):
373 s = s.decode(self.encoding)
--> 374 obj, end = self.raw_decode(s)
375 end = _w(s, end).end()
376 if end != len(s):
/usr/local/lib/python2.7/site-packages/simplejson/decoder.pyc in raw_decode(self, s, idx, _w, _PY3)
402 elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf':
403 idx += 3
--> 404 return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.