Comments (17)
@SeanNaren Thanks! 😄
I'm not sure I follow you, the code snippet above will change the names of experiment in the trains-server back-end (this is system wide rename affecting both experiment table and comparison/graph legend).
Are you also suggesting that if we have multiple experiments with the same name we add the ID to the name column ? The ID itself is searchable, so if you are looking for a specific experiment ID you can quickly find it, also if you click the ID
icon next to the task name in the right panel, you will get the experiment ID.
BTW, the following code will rename all experiments and add the experiment ID to the experiment name (so now you do not have to it manually):
tasks = Task.get_tasks(project_name='examples', task_name='train')
for t in tasks:
t.name += ' {}'.format(task.id)
Or if you want you can add it directly to your training code:
task = Task.init('examples', 'train')
task.name += ' {}'.format(task.id)
from clearml.
Hi @SeanNaren ,
We've just released Trains-Server v0.13.0, with added support for comparing experiments with the same name :)
from clearml.
Thanks @SeanNaren, greatly appreciated :)
Regrading Task names, you can go and change the names of the tasks after you executed them.
Just open the task info, double click on the name and edit it, the charts should refresh with the new names, hopefully helping to better differentiate between the different runs.
Sounds good?
BTW:
We just released v0.12 🎉 of both trains , trains-server, and the new addition trains-agent.
You can now have Full AutoML pipelines, including cluster management, with literally only pip install trains_agent
on any machine you like to connect to the cluster.
As by now you probably know, the experiment already stores everything that is needed in order to replicate the run and now you have the ability to change the hyper-parameters, and schedule execution.
from clearml.
Thanks for your quick response man! This is great, I assume if I wanted this to be possible for future experiments, I would need to change the task name in the python code everytime I run an experiment?
from clearml.
I can't think of anything else to put in the legend (other than the Task ID, which is a random set of alpha numeric characters, not very descriptive...)
Do you have something else in mind?
p.s
yes, you could change the python code to pick a new name on-the-fly based on some logic, that sounds reasonable to me. You can always change it later :)
from clearml.
So our currently solution is using MLFlow, which gives us the random set of alpha numeric characters, but usually what we do is open the corresponding experiments in a separate tab to see our hyper-parameters etc (to be able to compare between runs and know which runs are which):
We could imitate this if there was a way to name the experiment with the same ID that Trains generates, but you may be able to help us come up with a different solution!
from clearml.
Hmm I like it, so if we have a legend with two or more exact entries, add the Task ID after the experiment name.
Sounds good?
from clearml.
Sounds like a good idea to me! Is there anyway we can do this ourselves currently?
from clearml.
I guess the easiest solution is renaming (either through the UI or in the python code).
I'll make sure we add the feature request for the next release.
As you can see from the release history, I'm guessing it won't take long :)
Meanwhile, may I suggest an automation pipeline with a naming convention ?
There is a simple example here
from clearml.
You guys are awesome! Thanks for the example, is there anyway you could rename the task via the python code to use the task ID as the task name?
from clearml.
I assume you mean post execution, in that case, my apologies, there still isn't. But good point, we will think on how to add these types of automation.
from clearml.
Thanks! It would just make referring to the experiment/graphs easier when comparing imo, but modifying the names manually for now works!
from clearml.
Hi,
I am a new user of your great tool. I'm also facing the issue of distinguishing experiments when comparing them in the scalar plot.
I'm wondering how practical it would be to let the user select fields from the hyperparameter list to make a more informative legend name.
What do you think?
from clearml.
Hi @elinep, so it seems everyone agrees that the easiest solution is adding the experiment ID next to the experiment name in the legend.
This fix will be part of the next trains-server
version (due to be released in a couple of weeks).
You could also add custom columns in the experiment page, including the hyperparameter and the loss/accuracy, see example here
In the meantime, if you upgrade to trains==0.12.1
you could do the following
tasks = Task.get_tasks(project_name='examples', task_name='my_automl_experiment')
for t in tasks:
params = t.get_parameters()
if 'my_secret_parameter' in params:
t.name += ' my_secret_parameter={}'.format(params['my_secret_parameter'])
This code will add "my_secret_parameter=???" suffix to the names of all the experiments named "my_automl_experiment". Now when you compare them, you'll have the correct context :)
p.s.
@SeanNaren I'm guessing this solution applies to you as well.
from clearml.
@bmartinn thanks for your help and response times man!
Just on a side will this fix make it to the table view? I've had to manually change the names in the table for the associated experiments to the experiment ID to be able to refer to the correct experiments (currently all our experiments default names are train
).
from clearml.
The last code snippet does exactly what I wanted, thanks! I think for me this is enough to close the issue, If you'd like to keep it open for any reason feel free but I'm more than happy to close this
from clearml.
Excellent!
closing issue :)
from clearml.
Related Issues (20)
- "413 Request Entity Too Large" when uploading files to ClearML HOT 4
- legend titles broken in experiment comparison HOT 1
- Preview text files HOT 1
- Registering models from lightning not working (different than pytorch-lightning) HOT 2
- GPU monitoring failed getting GPU reading, switching off GPU monitoring HOT 6
- async variant of get_mutable_local_copy HOT 2
- Light theme for the dashboard HOT 1
- Plot comparison in a single figure not working for plots other than barplots HOT 3
- API calls fail for model with deleted parent task. HOT 1
- Scalar logging bug with Fire HOT 6
- multilanguage support HOT 1
- is it possible to upload bytes object instead files? HOT 1
- Add retries and chunking parameters to upload_folder HOT 2
- Add_external_files for Hyperdatsets HOT 1
- Add Progress Bar for downloading (and maybe uploading) artifacts HOT 2
- [ERROR] [CLEARML.auth] Error getting token - FileServer
- Many ClearML Dataset functions and parameters not available for Hyperdatasets, also it would be clearer to change name of allegroai Dataset to Hyperdataset HOT 1
- How to generate api services content?
- Scalars, Plots, and Debug Samples hang and stop logging with large git diff and Tensorboard Images HOT 2
- Relative path is incorrect when using `Dataset.add_external_files` for local file without the `file://` prefix HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml.