Comments (5)
Can you just reparameterize your problem in this case? Like define integer parameters size_per_head
and log_batch_size
and then in your evaluation code have
hidden_size = size_per_head * num_attention_heads
batch_size = 2**log_batch_size
Or would that mean you need a joint constraint on size_per_head
and num_attention_heads
?
@sdsingh From an optimization/candidate generation perspective (other than difficulty of the problem) there is no issue with imposing non-linear constraints on the parameter space. I assume the linearity assumption is mainly imposing structure for representing the constraints in Ax?
from ax.
All parameter constraints are implemented as linear constraints in the modeling layer. When we pass points into our GPs, they are all normalized to [0,1]^d, where then our constraints are applied. You can see how these are implemented w/ a simple matrix multiply in the Botorch model. As a result, we can only support constraints that can be mapped into a linear constraint, however creatively that may be. Unfortunately, I don't see a way of doing that for either of these constraint types.
Both of these constraint types seem pretty useful, though. Let me think about what we can do here, but unfortunately I don't think this will be a quick fix.
from ax.
Thank you, that's indeed a great alternative way.
I just thought we can make it elegant, but after I read the explanation, I knew that's not easy as I thought.
But stiil a very useful library to tune, thanks for your works !!
from ax.
Closing this issue for now as there is an easy workaround and adding modulo constraints is not on our roadmap for the foreseeable future.
from ax.
Can you just reparameterize your problem in this case? Like define integer parameters
size_per_head
andlog_batch_size
and then in your evaluation code have
hidden_size = size_per_head * num_attention_heads
batch_size = 2**log_batch_size
Or would that mean you need a joint constraint on
size_per_head
andnum_attention_heads
?@sdsingh From an optimization/candidate generation perspective (other than difficulty of the problem) there is no issue with imposing non-linear constraints on the parameter space. I assume the linearity assumption is mainly imposing structure for representing the constraints in Ax?
Passing comment on an old thread: similarly, could reparameterize as hidden_size_multiplier
which reflects the same constraint and search space if I'm not mistaken. Also, there is some support for non-linear constraints in Ax now #769, though not sure if modulus is a differentiable function.
from ax.
Related Issues (20)
- Safe optimization in the Service API HOT 5
- The same point is evaluated multiple times during Integer Optimization with BO. HOT 5
- ax_client.generation_strategy.trials_as_df HOT 9
- Managing Objective Function Evaluation Failures in Ax for MOO HOT 6
- [GENERAL SUPPORT]: Managing Objective Function Evaluation Failures in Ax for MOO HOT 3
- [GENERAL SUPPORT]: Using qNegIntegratedPosteriorVariance HOT 3
- [GENERAL SUPPORT]: Reference point for multi-objective bayesian optimization HOT 4
- [GENERAL SUPPORT]: Adjusting search space or accommodating out-of-bounds initial data HOT 19
- [GENERAL SUPPORT]: Manual configuration, HOT 1
- [Bug]: Custom metric issue HOT 4
- [GENERAL SUPPORT]: CI_Level Paretofrontier
- [Bug]: Large sample time increase in ax-platform >= version 0.3.5 HOT 6
- [GENERAL SUPPORT]: Reference Point for Multi-Objective Bayesian Optimization HOT 5
- [GENERAL SUPPORT]: Plotting Pareto fronts / Posterior mean model HOT 6
- [GENERAL SUPPORT]: Getting best predicted point of a botorch model HOT 10
- [GENERAL SUPPORT]: Logical-or in outcome constraints HOT 4
- [GENERAL SUPPORT]: Parallelism and arbitrary parameter type support. HOT 5
- [GENERAL SUPPORT]: How to Standardize and Normalize Data for Service API HOT 6
- [FEATURE REQUEST]: Implement ImprovementGlobalStoppingStrategy for batch trials HOT 2
- [GENERAL SUPPORT]: Plot observed pareto frontier alongside remaining Arms HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.