Comments (5)
@johnaohara
As discussed, shall we rename 'EXP_STOP' to 'EXP_REMOVE' as it deletes the experiment?
If this operation is used, it means we are deleting the experiment irrespective of whether it is complete or not. It even deletes the plots the completed experiment has generated.
If the user wants to restart the same experiment, the user has to do EXP_REMOVE and then EXP_START. The caveat of this operation would be the user will lose the available data for that experiment.
The user can view the experiment status before applying this operation.
For REST service, the user can view the listExperiments page (http://localhost:8085/listexperiments) to view the status of the experiment. We can have similar functionality for gRPC as well.
from hpo.
@kusumachalasani , thanks for opening this issue.
The original intention for EXP_STOP
was to stop a running experiment. As the service was stateless, and an experiment was essentially paused until the next EXP_TRIAL_GENERATE_NEW
call, stopping an experiment semantically meant removing it from the running experiments. I agree that the naming should be clarified to represent what the action is, i.e. removing it from HPO, e.g EXP_REMOVE
Now that HPO retains state, and has functions to generate charts after the trials have been completed. What is the expected behavior if a user wants to start a new set of trials, i.e using the same configuration? i.e. what is the expected lifecycle of a set of results? if someone starts a new experiment, using the same configuration, is the old data erased? if not, how would the experiments be uniquly identified?
from hpo.
@johnaohara
experiment_name
is the primary key for experiments. If we want to run a new set of trials using the same configuration,
- user can change the experiment_name in the search space and start the experiment. The configuration of the experiment remains the same.
- delete the experiment and start the same experiment again (experiment_name can be the same as previous). All the previous data available for that experiment gets inaccessible.
The data for an experiment is available until that experiment exists in HPO.
from hpo.
@johnaohara I updated the operation from EXP_STOP to EXP_DELETE and also updated the function names to use delete instead of stop. I did update the function names for gRPC as well.
All the changes are part of the PR #117. Can you please review the same?
from hpo.
Closing this as the changes required went in as part of PR #117.
from hpo.
Related Issues (20)
- secret in minikube needs to deleted while terminating HPO HOT 1
- Update index.html with experiment plot URLs to access. HOT 1
- Terminating HPO in minikube doesn't require to set environment variables for secret creation.
- Subsequent trial is not generated at times when running experiments in parallel
- Posting experiment result for an ongoing trial failed HOT 2
- plot API fails to generate plots and defaults to optimization_history when only exp_name is specified HOT 2
- HPO doesn't start in native with only rest service as it still looks for grpc.
- Remove SCC dependency for Openshift deployment
- Fix package versions to avoid compatibility issue
- CrashLoopBackoff when using ./deploy_hpo.sh -c openshift HOT 2
- Incorrect validation error for property data type mismatch in the search space JSON
- Replace system:build-strategy-docker role from the hpo-operator-rolebinding.yml to a more restrictive one HOT 2
- GRPC sanity tests fail on minikube / openshift
- Add visualization plots for HPO experiment HOT 1
- HPO negative tests failures on Openshift HOT 2
- HPO service needs to be exposed HOT 1
- Add an API to generate hyperparameter importance values from optuna HOT 1
- Create SCC for running HPO on Openshift in restricted manner HOT 1
- Ownership of app/src folder is not hpo user.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpo.