plonerma / wann-genetic Goto Github PK

• How many weights would be needed?

• If we use the number of generations for weight array size, how many more evaluations would be required?

• No heuristics needed for individuals with different numbers of weights that influences the fitness

• How many calculations would it save on slightly lower numbers?

• Model cost / advantage of this method

Saving a confusion matrix for each iteration blows up, if there are many generations.
Either compute metrics and cache those or just evaluate the mean confusion matrix.
Relevant questions here:

• Is the mean of f-scores (and other metrics), the same as the f-score of the mean confusion matrix?
• Is caching the calculated metrics a suitable alternative?

Should rewann be adpated to use sklearn? Check out sklearn-genetic and compare pros and cons

The amount of code to maintain should be reduced. Lean on standard libs where possible. Remove unnecessary bloaty concepts.

Check-out:
sklearn-genetic

• Enforce policy about which
  data to store
• Represented by a unique dir on fs

Store:

• hyperparams were used on task
  • num processes
  • hyperparams for evol. process
• population (if condition met)
• time series of events:
  • best and mean performances in
    each generation
  • cpu usage
  • outputs, warnings, infos, etc.

• Stores series of performance records
• Produced by task adapter ( #18 )
• Kept by experiment class ( #21 )
• Referenced by individual via experiment

Mglw. weniger relevant, sobald negative Gewichte zugelassen sind.

• logging should not require passing the log obj via environment
• use root logger and configure on env init

• Use streamlit to enable during and post experiment inspection
• Write interface for headless execution of experiments

• Take np.array as weight and calculate all steps for each weight (adds dimension during network application)

• Which activation functions will return the same value if the shared weight is sufficiently high enough and only 1 (or 2) of multiple edges propagate a signal?

https://github.com/google/brain-tokyo-workshop/blob/dfa8e09cbb44a7b75c70795b9d220cb2fea4b34b/WANNRelease/prettyNeatWann/neat_src/neat.py#L69

Performance metrics will be influenced by the introduction of multiple tests results for randomly sampled weights ( #25 ).

Peformance module should:

• store confusion matrices for each weight that the individual was evaluated on
• compute averages (accuracy, f-score, etc.)
• provide interface to fitness considerations (elite & tournament selection)
• be able to compute averages etc. for lists of individuals (populations)

Update experiment defaults with rewann_default.toml -> this allows for environment specific default values (useful for debug, storage, and num_workers settings).

Knoten, die schon lange bestehn, eher verbinden (possibility matrix multiplizieren)

Individual

Individual is a representation for networks. Networks can be in different states, and not all states should be constantly in memory. To still be able to easily be able to work with each possible expression of the network (genotype, phenotype, etc.) translations should implicitly be made.

Where suitable, certain expressions will be saved to the file system for later inspection.

Class

Data

Representations

• genotype with historical markings
• phenotype (topologically ordered weight matrix)
• torch network

Other

• fitness measurements / performance stats

Functions

IO

Enable loading & saving of all representations from & to FS

Expression

Translate expressions, save all translated versions

Evaluation with task

Evaluate Individual on a given task (possibly using certain seed / other parameters)

Individual

• Median Performance
• Confusion-Matrix?
• Best Weight (Tuned Shared Weight)
• Tuned Weights

Population

• Ensemble model

• Generate new generation
• Mutation Operator
• Keep track of generations
  and metadata of their overall
  performances

Sprechnen Sie uns an, wenn Sie lange Rechnungen aufsetzen, die nicht unterbrechbar sind. Rechnerbetriebsgruppe

Originally posted by @plonerma in #2 (comment)

Should I implement culling?

https://en.wikipedia.org/wiki/Culling

https://github.com/google/brain-tokyo-workshop/blob/55d8c1990dc446e3278a70e3a0964aee6faf8dbb/WANNRelease/WANN/wann_src/_variation.py#L55