Simple example for learning curve plotting about syne-tune

[mseeger]

Hello Andreas, thanks for your interest.

In the context of TrialsOfExperimentResults, a setup maps to an experiment, whose trial learning curves are shown in one of the subplots. In the example, we have setups ASHA and HYPERTUNE-INDEP, which are two different HPO methods. We get two sub-plots, and can compare how they differ.

You can also plot these results for a single experiment (one method, one seed). In the example above, to plot results for ASHA only, just use SETUPS_TO_COMPARE = ("ASHA",).

[geoalgo]

We just merge a tool to do those plots directly from ExperimentResult, if you can pull from mainline, you should be able to do:

...
tuner.run()

from syne_tune.experiments import load_experiment
exp = load_experiment(tuner.name)
exp.plot_trials_over_time()

If you cant pull from mainline and want to use the latest version from pip, you can use the snippet I gave above.

Hope those help, feel free to reopen if not :-)

[mseeger]

Can you attach the launcher script you were using?

[amueller]

Thank you for the quick reply Matthias! My launcher script is something like

import logging

from syne_tune import Tuner, StoppingCriterion
from syne_tune.backend import LocalBackend
from syne_tune.config_space import randint, loguniform, uniform, lograndint, choice
from syne_tune.optimizer.baselines import ASHA, MOBSTER
root = logging.getLogger()
root.setLevel(logging.DEBUG)

# hyperparameter search space to consider
config_space = {
    'learning-rate': loguniform(1e-5, 1e-1),
    'n-layers': randint(1, 10),
    'hidden-size': lograndint(4, 2048),
    'dropout-rate': uniform(0, 1),
    'weight_decay': loguniform(1e-7, 1e-1),
    'onehot': choice([True, False]),
    'epochs': 10000,
}

tuner = Tuner(
    trial_backend=LocalBackend(entry_point='train_parity.py'),
    scheduler=MOBSTER(
        config_space,
        metric='accuracy',
        resource_attr='epoch',
        max_resource_attr="epochs",
        search_options={'debug_log': False},
        mode='max',
    ),
    results_update_interval=5,
    stop_criterion=StoppingCriterion(max_wallclock_time=30 *60),
    n_workers=1,  # how many trials are evaluated in parallel
    tuner_name="parity-test"
)
tuner.run()

I think I'm confused where the names for the setups come from. Are the they the name of the scheduler classes?

[amueller]

any suggestions?

[geoalgo]

Hi Andreas,

Sorry for the delay, we just realize that Matthias was out of the office, let me try to help you on this.

This code assumes that experiments has been scheduled with python benchmarking/examples/benchmark_hypertune/launch_remote.py \ --experiment_tag docs-1 --random_seed 2965402734 --num_seeds 15, the names "ASHA" and "HYPERTUNE-INDEP" comes from this file https://syne-tune.readthedocs.io/en/latest/benchmarking/benchmark_hypertune.html which defines the experiments to run).

From your example, it seems you just want to plot the trials after an experiments, you can use the following code for instance (I will add a method in ExperimentResults to make this easier):

from syne_tune.experiments import load_experiment
import matplotlib.pyplot as plt
from syne_tune.constants import ST_TUNER_TIME

# Replace with your experiment name here
# In your case, it will start with "parity-test" and be suffixed with a time-stamp for unicity
expname = "funky-earthworm-ASHA-0-nas201-cifar10-2023-10-10-12-38-03-865"

exp = load_experiment(expname)

fig, ax = plt.subplots(1, 1, figsize=(12, 4))
df = exp.results
metric = exp.metric_names()[0]
for trial_id in sorted(df.trial_id.unique()):
    df_trial = df[df.trial_id == trial_id]
    df_trial.plot(x=ST_TUNER_TIME, y=metric, marker=".", ax=ax, legend=None, alpha=0.5)
df_stop = df[df['st_decision'] == "STOP"]
plt.scatter(df_stop[ST_TUNER_TIME], df_stop[metric], marker="x", color="red")
plt.xlabel("Wallclock time (s)")
plt.ylabel(metric)
plt.title("Trial value over time")

[amueller]

perfect, thank you!