stasinos / machine_learning Goto Github PK

Define and implement an way to provide as a prior the knowledge that an unknown function is periodic (or likely periodic) and use this when constructing the networks (cf. Issue #20)

Have a look at exp01. This experiment defines the network fn1(x) + fn2(2x), where fn1 and fn2 are different instances of our "universal" approximator. The network is trained on data generated from sin(x) + sin(2x).

Inspect visually the test plot for fn1, fn2, and the overall network. fn1 and fn2 vaguely resemble sin(x) and sin(2x), so although far from perfect, they are promising (See below, but also generate them to ensure we all get the same results)

Experiment with different function combinations AND combinators (composition, addition) to draw conclusions about what works and what doesn't. Look for the hyperparams and training set size that seem to work best, but must work for all combinations. (That is, find a single set of hyperparams that works best for all complex functions, not a different set of hyperparams for each complex function)

sin(x) + sin(2x):

fn1(x) + fn2(2x):

fn1(x):

fn2(x):

The current implementation of the make_one_experiment function has several issues related to the accuracy calculations and the decision tree visualization.

1. The accuracy calculation during the k-fold cross-validation is not consistently organized. Some accuracies are missing, and the method of averaging (/20) cannot be generalized (k-dependent).
2. The creation of the decision tree visualization using export_graphviz is incomplete.

The current implementations of the make_square and make_long functions may result in errors, as the generated square or long shape could exceed the specified max_dim limit along the x or y axis. This issue arises from the calculation of the ending positions (x2/y2) being derived from the starting positions (x1/y1) and delta, which could potentially be 99. Therefore the ending positions may exceed 100.

The training data should be pairs of graph language inputs and the resulting graphs (bitmaps). It is important that there are no training data whatsoever with isolated functions, only complex expressions. The purpose of the learner is to construct the definitions of the (hypothetically unknown) elementary functions sine(), sinc(), exp(), and ln() although it can only observe their behaviour in the context of complex expressions.

We shall leave poly() outside the scope of the training and provide the system with its definition. We shall use the composition with poly() and the meaning of multi-line programs as a means to build complex linear and non-linear combinations of the elementary functions (and compositions of more than one elementary function).

All four elementary functions must appear in every single example, in order to not allow the learner to isolate sub-problems and have to learn all four functions simultaneously.

One way to build the training dataset is the following: Write a dozen or so graph language program templates that combine all four functions, leaving 5-10 numerical parameters as placeholders, and then generate pairs of instantiated programs & their bitmaps by randomly filling the parameters.

A short slide deck with the most important points of the term paper.
Please do not commit here.

You can see an example for sine committed, but feel free to experiment. In any case, you should confirm in advance that the same structure and hyperparameters (the one in the example or some other) will be able to approximate any of the four elementary functions. These will be the "unknown functions" that your system will learn by observing the outputs of complex expressions of these functions, so we should first confirm that they are learnable.

Curate the autodiff-history, merge, and update the exercise definition to describe the next step for next year's students

Further work is needed on the graph language:

• Extend with if-then-else branching, to better demonstrate what autodiff does
• Implement in the executor the functionality of automatically writing out pyTorch code as the result of compiling a graph language program
• Define and implement the parser as a proper DCG

The make_production_dataset function attempts to create long shapes with a specified max_dim limit, but due to changes in the make_long function (#23) , these long shapes can now fit within the allowed axis limits.

Prepare a short document (about 4 pages) that presents the results. Please do not commit here drafts of the document, to avoid leaking too much to next year's students.

• Introduction: Description of the problem
• Background: A few words about autodiff how it allows interleaving explicit code with machine-learned functions
• Experiments: The elementary functions and combinators used, in the experiments, and plots with the results. Per-experiment discussion of what worked and what didn't, with some theorizing about why stuff didn't work.
• Conclusions: Higher-level discussion of what kinds of combinators work better than others (e.g., addition works better than composition?)

This issue addresses the need for a more comprehensive understanding of the training process and its relation to model performance. Currently missing from the codebase are calculations for accuracy drops, failure probabilities, and hypotheses space complexities. This hinders a clear insight into the dynamics of training and the underlying complexities of the model.

The testing data should be pairs of graph language inputs and the resulting graphs (bitmaps). As the purpose of the learner is to construct the definitions of the (hypothetically unknown) elementary functions sine(), sinc(), exp(), and ln(), the testing dataset only needs to have the graphs of the elementary functions in isolation.

One way to build the training dataset is to generate pairs of instantiated programs & their bitmaps by randomly filling the parameters that make sense for each function. Composition with poly() can be used for this.

Implement an evaluation function that computes a scalar similarity between two plots. This should behave like a heuristic, so that is can be used as a loss function.

The make_training_dataset and make_production_dataset functions trigger SyntaxWarnings due to the use of "is" with string literals.