Comments (5)
In that example (https://ax.dev/tutorials/gpei_hartmann_developer.html) there are two main sources of randomness: The scrambling in the Sobol generator used for the initialization, and the randomness in the (quasi) Monte Carlo approximation used in the acquisition function.
Like @kkashin says, the first can easily be handled by making the change to
sobol = Models.SOBOL(search_space=exp.search_space, seed=1000)
And yes, this will require using the more explicit API rather than optimize
, whose purpose is to do a bunch of stuff under the hood for you.
Most of the randomness will come from that (the initialization). The actually Bayesian optimization phase will still have some randomness in that the acquisition function is evaluated with a Monte Carlo approximation. The seed for that is set here:
Ax/ax/models/torch/botorch_defaults.py
Line 162 in eebb8c7
So, you can just run
torch.manual_seed(1000)
before every call to gpei.gen
.
As a side note, for the purposes of benchmarking / performance evaluation I'd recommend setting the number of reps high enough that the seed doesn't matter. But of course there are other settings where reproducibility of a single run would be useful, and this should give that.
from ax.
I'm going to close out this issue for now, given that you're unblocked. We will look into making it possible to pass a seed directly to the optimize
function so that reproducibility is easier to achieve in the future, and will follow up with developments.
from ax.
@Irynei - what level are you looking for reproducibility at? The whole optimization loop or a specific generator?
You can set the seed for (quasi)-random generators, such as the Sobol generator (see https://ax.dev/api/models.html#ax.models.random.sobol.SobolGenerator.seed).
from ax.
I was looking for the whole optimization loop, in the optimize
method, ideally.
As far as I understand a (quasi)-random generator is chosen somehow under the hood?
So I can only set seed when I am using something similar to what explained here https://ax.dev/tutorials/gpei_hartmann_developer.html, right?
from ax.
@bletham, thank you for a great explanation! It's clear to me now.
from ax.
Related Issues (20)
- Can't control arguments in fit_gpytorch_mll under the hood. Getting ABNORMAL_TERMINATION_IN_LNSRCH warning HOT 1
- Different Errors when initializing my loop with Service API and Developer API HOT 7
- `_random_seed` not retained when using `ax_client.save_to_json_file()` and `AxClient.load_from_json_file()` HOT 2
- Question: SEBO optimization with parameter dependency | logistic parameter constrains HOT 4
- Tutorial Request: Deploying Runners on Clusters, Debugging Runners/Schedulers HOT 4
- Issue with tolerance for floating point and its relevance when using log_scale = True HOT 7
- Question: does Ax support working with Tensorflow models? HOT 2
- Feature Request: Conditional Parameter Constraints HOT 5
- Questions about define how to evaluate HOT 3
- get_countour_plot() not plotting all trials HOT 3
- Error: A list of 'ChoiceParameter' is not iterable HOT 4
- [Bug] Generation Strategy equality check error without call to repr HOT 1
- ax.dev, GitHub readme, and loop tutorial examples are ignoring `minimize` kwarg HOT 2
- Get Started code gives the same result regardless of random seed HOT 3
- Trouble with searching documentation (Algolia, API docs, etc.), for example `get_next_trials` HOT 1
- Issue when starting an AxClient with out-of-design points HOT 2
- cannot import name 'TrainingData' HOT 2
- applying complex constrains HOT 2
- Ax is not not starting as many workers as I'd like to; sometimes, get_next_trials returns 0 new trials
- Evaluating custom candidates HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.