shaynali / aipen

Evolutionary engine for easier hyper-parameter optimization of AI and ML models.

Aipen

master

dev

aipen's Issues

Documentation

Document original, undocumented code in the repo's GitHub wiki.

Talk to Essex about digital forensics

Arena enhancements

Allow the Aipen arena to optimize instance of itself.

Fix importing

Initial data module

Management of data storage and ids
Should interface with visualization module and use numpy types as underlying data storage

Logger is bugged (pre-rollback)

Bugs:

• Does not log to arena and build logs
• Does not give console output for build tests
• Does not log to absolute location of log (known behavior, needs modification)

Model serialization

Add option for save and load to/from pickled file and JSON/other readable file format.

Write more models - linear, fitting, neural nets

Initial visualization module

Write some basic visualization processes using an appropriate third party library. Should be as decoupled from third party library as possible so we can switch to another if needed.

RandomModel distribution only works for min/max distributions

Relevant code

Visualization module extensions

A follow up to #10 to explore enhancements to the visualization module.

Some other options which are interactive, and could be good for the presentation:

• Bokeh
• Chaco

Also see visualization libraries discussed originally in #10.

Reconsider abstract classes + methods for agents

Either improve or remove the abstract classes and methods.

Interview digital forensics contacts

Gradient lattice descent

Optimize evolutionary learning and intra-round learning.

Data processing

Following up to #8, add processing functions such as padding, noise reduction, transforms, etc as they are needed

Set up Travis CI

This task relates to the test coverage tasks and involves setting up the .travis.yml file to integrate with Travis CI and set up any related testing scripts.

Command option parsing

Should add cmd line options and parsing, so that the user can have a rich way of choosing how modules run, what the console outputs (e.g. listen to evaluation log instead), if a log outputs to a new console, etc.

See #15 and #40.

Consider using Click.

Refactor and clean up

Various bits of code, especially in the arena code should be clean up and refactored, renaming as required and breaking apart functionality.

Fix gitkeeps

The line !**.gitkeep in .gitignore does not serve to maintain empty folders (e.g. the logging folder), so either this should be fixed or else a utils file should be created to manage path and file creation.

Upgrade to Python 3.7

Get Travis to run without pip installing in dev mode

Add builds for multiple configurations

E.g. by using a matrix

Move setup.py to top level dir

RandomStaticModels do not pass serialization tests

Should also making serialization better in general.

Additional activities

OpenAI gyms, image recognition, etc

Abstract model enhancements

Write the AgentModel, refactor others

Logging tests

Test that log functionality works by expected, can use a special log channel

Should conform with system being developed in #5

Write more rigorous tests + testing system

Write additional tests for uncovered modules, implement using test cases as a dict key=input, val=expected_ouput. Can have multiple input/outputs.

Problem and agent/model abstracts

Implement MLProblem and AgentProblem (and come up with a better name if you can) plus at least one implementation of each, based on a toy problem.

Consider making SupervisedProblem and UnsupervisedProblem interfaces with slightly difference APIs.

Recommend implementing noisy linear data fitting for MLProblem and Battleship for AgentProblem.

Find a way to move setup.py and requirements.txt to root

Previous attempts have broken pip install ., running into the No module named * issue.

Add option to open live logs as subprocesses/etc

Instead of a single log monopolizing the console, we should have an option for logs to open consoles in other windows which listen to them (e.g. an arena console window, an evaluation console window, ...)

Top performing models, similar problems, etc

Create a 'social-like' side of Aipen, to find the best models for a given problem

Initial data generation

Write a data generation module capable of generating simple bits of data, e.g. random, linear with noise

Parameter parameterization

Add dict/function/etc for parameters (for learning) and hyper-parameters (for evolution) to models. Allows learning optimizers and evolutionary engine to spawn models better than random.

Data generation

Follow up to initial data generation module, grabbing data from actual sources/problems/etc

Should include an id system, even if it's just an initial implementation. Could use in-memory sqlite3.

Cloud API and web interface

Allow Aipen to be accessed as a web API with a web interface, so that users can provide data and run evolutionary algorithms online without installing anything.

Test coverage detection and warning

Look at a testing system which checks for test coverage and emits warnings when test coverage is low in a commit.

Logging channels

Fix console_logging=True behavior (locks out other logs).

Implement a source -> channel -> listener system for logging.

Refactor directory structure

Be wary of import errors with some directory structures: https://stackoverflow.com/a/50559323/7195043