leotrs / decu

decu stands for "Decu is a Experimental Computation Utility"

decu

decu stands for "Decu is a Experimental Computation Utility". decu is a suite of command line tools to automate the menial tasks involved in the development of experimental computation projects.

Experimental Computation

We define a "experimental computation" script as a script that reads some data, performs an experiment (run an algorithm, train a model, plot a figure, etc), saves some results to disk, and then quits. Some tasks that are usually involved in this are:

1. file I/O: choosing file names for results and plots
2. multiprocessing: running many experiments/scripts in parallel
3. timekeeping: how long algorithms take to run
4. cloud service integration
5. logging

decu was born from the realization that none of these tasks have anything to do with (and in fact get in the way of) the actual experimentation being done. Furthermore, experimental computation scripts tend to mix together code dedicated to these tasks with code intended to run the actual algorithms of interest, thus making said algorithms harder to maintain and debug.

The main goal of decu is to provide standardization and automation of these tasks, with end code that clearly separates experimental computation from bookkeeping and other auxiliary code.

Installation

Clone this repo, cd to the decu directory and do

$ pip install .

Now you have a local installation of decu. If you are going to make edits to decu, don't forget to use the -e flag.

Usage

For a simple example, please see the quick start page. For more, see the tutorial and the documentation page.

Best practices

decu is built with Best Practices for Data Science in mind. For more, see

Wilson, Greg, et al. Good Enough Practices in Scientific Computing. PLOS Computational Biology, vol. 13, no. 6, 2017, doi:10.1371/journal.pcbi.1005510.

Wilson, Greg, et al. Best Practices for Scientific Computing. PLoS Biology, vol. 12, no. 1, July 2014, doi:10.1371/journal.pbio.1001745.

decu's Issues

we don't need the Script class

can we add a pytest-discoverable test that runs `python -m decu src/testscript.py` inside the test/testscript directory?

leave the __init__.py file empty and separate everything in files

Reason being that now init.py and main.py use the same code to load the config parser. If in the future we need to change that code (for example we may change the interpolation kw arg), we will probably enter a bug by changing one and not the other. One of the new files this should have a read_config function that both use.

add "saved figure to file" message to log

Would using string templates make the configuration file more readable?

All the {} in the strings in config.ini make no sense. Would using string templates improve this?

shorten the result file name in log output

add a test case that checks that a @experiment or @figure have been saved to disk correctly

Implement the run_parallel that works on @experiment decorated methods

should we have named tuples for experimental results?

add a `project init` executable

add a CLI option for the executable that reads some data and runs a @figure-decorated method on it

Right now, __main__ creates a Script object by passing NOW. Should the Script class itself handle this?

add "data registering" functionality

log experimental results

in results/

add config file options to change the log file, figures file, results file (filenames, not dir names)

add a test case for testing the show and save arguments of @figure

The experiment decorator needs to know the NOW time to generate the result file name. Where is it going to get it from?

define a __all__

add a test case for command line options

for example, if we call decu inspect or decu exec without a file parameter, decu should print an error.

return decorated methods that receive self as first argument, instead of doing obj=args[0]

add a test case checking that the dictionary returned by run_parallel has the elements in the correct order

that is, check that every value corresponds to the key it was associated to
equivalently, make sure that pool.starmap doesn't change the order of the results with respect to the order of the iterable

make the py_cmd in __main__.inspect into a decu.cfg option

functions exp_start_msg, exp_end_msg, wrote_results_msg, make_figure_file, wrote_fig_msg are basically the same. How can we refactor them into one?

add GCP integration

Have a separate config file instead of hard coding strings

The logs have simple quotes appearing in weird places.

This happened right after we started using the config.ini file.

Make a test case for log outputs.

should we support a `run_parallel` call on a data generator function?

add support for writing different types of results to disk

Right now, all we have is np.savetxt()

Add a delimiter option in the Script file of the config file. Add to the documentation the fact that none of the other strings (time, module_name, etc) can contain the delimiter as a substring

add a CLI option for the executable that loads results and dumps into ipython

have premade plot functions like: given a set of values, plot the mean, 95% shadows, etc

we need better documentation for this project

add test case for the "suffix" parameter of @figure decorated methods

stop hardcoding the path to the config.ini file

In fact, we should have a system where the user can have a config.ini file in the project dir that overrides the defaults.

how do we test `decu inspect`?

add a test case where we run `decu exec testscript.py`

Before doing that, make sure the logs, results, pics direcotires are empty.
After doing that, we can also do decu inspect <file> --plot <func> to check that a new figure has been added.

add devel and publish modes

add test case for local config files

Make a standalone executable

No need to always use python -m

Have a not-so-simple test project that is more complicated than the current testscript

the name Script.py is a misnomer

move the TestScript class from __init__ to somewhere in test/

use exceptions

standardize the use of parameter, argument, value, etc

How to best expose the experiment run id to the user?

Right now we have class.exp_method.run - see test/test_project/scr/script.py

support experiments with an arbitrary number of parameters

should the variables LOGS_PATH and so on be spelled in lower case?

add a test case that changes the options in the config file

Implement the @figure decorator

add support for local config files

The user should be able to customize the config file by having a config.ini file in the project dir, which overrides any default value. Add test cases for overriding.