Comparing scalar charts about clearml

[bmartinn]

@SeanNaren Thanks! 😄

I'm not sure I follow you, the code snippet above will change the names of experiment in the trains-server back-end (this is system wide rename affecting both experiment table and comparison/graph legend).

Are you also suggesting that if we have multiple experiments with the same name we add the ID to the name column ? The ID itself is searchable, so if you are looking for a specific experiment ID you can quickly find it, also if you click the ID icon next to the task name in the right panel, you will get the experiment ID.

BTW, the following code will rename all experiments and add the experiment ID to the experiment name (so now you do not have to it manually):

tasks = Task.get_tasks(project_name='examples', task_name='train')
for t in tasks:
    t.name += ' {}'.format(task.id)

Or if you want you can add it directly to your training code:

task = Task.init('examples', 'train')
task.name += ' {}'.format(task.id)

[bmartinn]

Hi @SeanNaren ,
We've just released Trains-Server v0.13.0, with added support for comparing experiments with the same name :)

[bmartinn]

Thanks @SeanNaren, greatly appreciated :)
Regrading Task names, you can go and change the names of the tasks after you executed them.

Just open the task info, double click on the name and edit it, the charts should refresh with the new names, hopefully helping to better differentiate between the different runs.

Sounds good?

BTW:
We just released v0.12 🎉 of both trains , trains-server, and the new addition trains-agent.
You can now have Full AutoML pipelines, including cluster management, with literally only pip install trains_agent on any machine you like to connect to the cluster.

As by now you probably know, the experiment already stores everything that is needed in order to replicate the run and now you have the ability to change the hyper-parameters, and schedule execution.

[SeanNaren]

Thanks for your quick response man! This is great, I assume if I wanted this to be possible for future experiments, I would need to change the task name in the python code everytime I run an experiment?

[bmartinn]

I can't think of anything else to put in the legend (other than the Task ID, which is a random set of alpha numeric characters, not very descriptive...)

Do you have something else in mind?

p.s
yes, you could change the python code to pick a new name on-the-fly based on some logic, that sounds reasonable to me. You can always change it later :)

[SeanNaren]

So our currently solution is using MLFlow, which gives us the random set of alpha numeric characters, but usually what we do is open the corresponding experiments in a separate tab to see our hyper-parameters etc (to be able to compare between runs and know which runs are which):

We could imitate this if there was a way to name the experiment with the same ID that Trains generates, but you may be able to help us come up with a different solution!

[bmartinn]

Hmm I like it, so if we have a legend with two or more exact entries, add the Task ID after the experiment name.
Sounds good?

[SeanNaren]

Sounds like a good idea to me! Is there anyway we can do this ourselves currently?

[bmartinn]

I guess the easiest solution is renaming (either through the UI or in the python code).

I'll make sure we add the feature request for the next release.
As you can see from the release history, I'm guessing it won't take long :)

Meanwhile, may I suggest an automation pipeline with a naming convention ?
There is a simple example here

[SeanNaren]

You guys are awesome! Thanks for the example, is there anyway you could rename the task via the python code to use the task ID as the task name?

[bmartinn]

I assume you mean post execution, in that case, my apologies, there still isn't. But good point, we will think on how to add these types of automation.

[SeanNaren]

Thanks! It would just make referring to the experiment/graphs easier when comparing imo, but modifying the names manually for now works!

[elinep]

Hi,
I am a new user of your great tool. I'm also facing the issue of distinguishing experiments when comparing them in the scalar plot.

I'm wondering how practical it would be to let the user select fields from the hyperparameter list to make a more informative legend name.

What do you think?

[bmartinn]

Hi @elinep, so it seems everyone agrees that the easiest solution is adding the experiment ID next to the experiment name in the legend.

This fix will be part of the next trains-server version (due to be released in a couple of weeks).

You could also add custom columns in the experiment page, including the hyperparameter and the loss/accuracy, see example here

In the meantime, if you upgrade to trains==0.12.1 you could do the following

tasks = Task.get_tasks(project_name='examples', task_name='my_automl_experiment')
for t in tasks:
    params = t.get_parameters()
    if 'my_secret_parameter' in params:
        t.name += ' my_secret_parameter={}'.format(params['my_secret_parameter'])

This code will add "my_secret_parameter=???" suffix to the names of all the experiments named "my_automl_experiment". Now when you compare them, you'll have the correct context :)

p.s.
@SeanNaren I'm guessing this solution applies to you as well.

[SeanNaren]

@bmartinn thanks for your help and response times man!

Just on a side will this fix make it to the table view? I've had to manually change the names in the table for the associated experiments to the experiment ID to be able to refer to the correct experiments (currently all our experiments default names are train).

[SeanNaren]

The last code snippet does exactly what I wanted, thanks! I think for me this is enough to close the issue, If you'd like to keep it open for any reason feel free but I'm more than happy to close this

[bmartinn]

Excellent!
closing issue :)