godatadriven / asekuro Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 3.0 92 KB

A utility tool to automate certain tasks with Jupyter notebooks.

License: MIT License

Python 66.80% Jupyter Notebook 32.22% Makefile 0.99%

asekuro's People

Contributors

Stargazers

Watchers

Forkers

fokko hgrif rensdimmendaal

asekuro's Issues

switch to click

using subprocess for testing can be a tad bit ... wonky. instead it would be better to use click such that we can offer some more strict checking mechanisms.

put it on pypy

might be better to make sure people can install it too.

klopt also needs to accept python files

currently klopt only allows the first thing to be listed to be a notebook followed by python scripts. this feels like a limit we won't need. klopt needs to run files in order to check what is going wrong.

? issues

asekuro willen currently break without warning if it sees the following pattern while performning the check command.

you cannot pass it just a file it needs a path

(venv) ➜  01-streaming-algorithms git:(timeseries-nuon-revamp) ✗asekuro test 01-01-introduction-streaming.ipynb 
2018-07-31 13:50:02,183 [commandline.py:test_notebook:113] DEBUG - about to test 01-01-introduction-streaming.ipynb
2018-07-31 13:50:02,183 [commandline.py:_cwd:24] DEBUG - directory of script calling /Users/vincentwarmerdam/Development/accelerator/lecture09_timeseries/lecture/01-streaming-algorithms
Traceback (most recent call last):
  File "/Users/vincentwarmerdam/Development/accelerator/venv/bin/asekuro", line 11, in <module>
    sys.exit(main())
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/asekuro/commandline.py", line 129, in main
    'clean': clean_notebook
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/fire/core.py", line 127, in Fire
    component_trace = _Fire(component, args, context, name)
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire
    component, remaining_args)
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable
    result = fn(*varargs, **kwargs)
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/asekuro/commandline.py", line 114, in test_notebook
    folder, filename = _cwd(nbpath)
  File "/Users/vincentwarmerdam/Development/accelerator/venv/lib/python3.6/site-packages/asekuro/commandline.py", line 27, in _cwd
    os.chdir(folder)
FileNotFoundError: [Errno 2] No such file or directory: ''

enable docker caching in circle ci

it makes sense to investigate if the builds can be faster. 4 mins is a bit long.

Logging configuration should go out of the library

In commandline.py, this should be removed:

stdout_handler = logging.StreamHandler(sys.stdout)
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s [%(filename)s:%(funcName)s:%(lineno)d] %(levelname)s - %(message)s',
    handlers=[stdout_handler]
)

and maybe be replaced by:

logging.getLogger(__name__).addHandler(logging.NullHandler())

See https://docs.python-guide.org/writing/logging/#logging-in-a-library.

it cannot handle wildcards

preferably we have a nice way to just give it a folder or certain files to run. currently we have to run file by file. this is a nice to have, i don't mind being explicit. but we do want to have it at some point.

add ignore flags

for stability reasons it would be nice if we can add a flag to klopt and test to remove ipython magic cells in a clean manner.

replace logging with plain stdout

for added bonus it might be nice to have some colors in there as well.

add copyright functionality

currently we cannot add a copyright logo/text. we want to add this.

add markdown functionality

be able to take the output of a notebook and turn it into a markdown file with images seperated.

autoreload breaks asekuro

this is a serious bug

if this is in a notebook it gets mistaken for %load

%load_ext autoreload

Replacing cells with %load fails if the cells contain an additional new line

A cell with

%load bla.py

Will give a FileNotFoundError: [Errno 2] No such file or directory: 'bla.py\n'.

On one hand, we might only want to have cells with a nice single line with %load. On the other hand, we often get feedback after a pipeline of 15 mins so it could be nice to be more lenient (and I don't know what will happen if you have a trailing whitespace). A .strip()here could save some hassle.