Comments (16)
Effectively what me and @JaimeRZP are disucssing is the following two approaches.
The fully flattened approach, i.e. something like
Base.@kwdef struct HMC <: AbstractMCMC.AbstractSampler
termination_criterion=:fixed_steps
# Sampling of the trajectory.
trajectory_sampling=:endpoint
integration_time=1.0
num_integration_steps=16
# NUTS related
max_depth=10
Δ_max=1000.0
metric=:diagonal
stepsize=:auto
integrator=:leapfrog
# Adaptation.
adapt=true
adapt_mass_matrix=true
adapt_stepsize=true
num_adapts=0.5
target_acceptance_rate=0.8
end
NUTS(; num_adapts, δ, max_depth, Δ_max, ϵ) = HMC(; num_adapts, δ, max_depth, Δ_max, ϵ)
and then just construct HMCSampler
in the step
, and defer to the existing implementation.
And the alternative the Jaime proposes is more like
# Replaces `HMCSampler`
Base.@kwdef struct HMCSampler{I,K,M,A} <: AbstractMCMC.AbstractSampler
alg::HMCAlgorithm=Custom_alg
"[`integrator`](@ref)."
integrator::I=nothing
"[`AbstractMCMCKernel`](@ref)."
kernel::K=nothing
"[`AbstractMetric`](@ref)."
metric::M=nothing
"[`AbstractAdaptor`](@ref)."
adaptor::A=nothing
end
# Example alg
struct NUTS_alg <: AdaptiveHamiltonian
n_adapts::Int # number of samples with adaption for ϵ
δ::Float64 # target accept rate
max_depth::Int # maximum tree depth
Δ_max::Float64 # maximum error
ϵ::Float64 # (initial) step size
end
where the HMCAlgorithm
would then define the different recipes, e.g. NUTS
, and completely replace HMCSampler
.
I'm personally slightly in favour of approach (1) because I think that
- a power-user who wants to construct their own integrator can just use the "full" interface rather than this simple keyword-approach
- for "simpler" users I think seeing the algorithm stuff can be confusing (and this is the user group we want to target with this change)
Jaime prefers (2) because
- it allows arbitrary customization
- can completely replace
HMCSampler
- closer to current Turing approach
from advancedhmc.jl.
@yebai @devmotion @sethaxen Thoughts?
from advancedhmc.jl.
When a user calls the convinience constructor AHMC.NUTS a HMCSampler instance is created
I think generally constructors have to be faithful about the types they create - IMO a NUTS
function has to return something of type NUTS
.
My first thought would be: Maybe we can just use one sampler type (possibly with hierarchically structured fields) but provide different convenience constructors with default metrics, algorithms etc?
from advancedhmc.jl.
I think generally constructors have to be faithful about the types they create - IMO a NUTS function has to return something of type NUTS.
Fair point and I do agree with this. But then the alternative is to just create separate samplers for each of the different "main" configurations (as is currently done in Turing), but then we share the step
somehow.
My first thought would be: Maybe we can just use one sampler type (possibly with hierarchically structured fields) but provide different convenience constructors with default metrics, algorithms etc?
Not a big fan of this because we really want something for users who just want to do NUTS()
or HMCDA()
, and not have to muck about with HMC(nuts=...)
or something.
from advancedhmc.jl.
Not a big fan of this because we really want something for users who just want to do NUTS() or HMCDA(), and not have to muck about with HMC(nuts=...) or something.
But can't we have a single sampler type AND such convenience constructors? I imagine we could maybe make NUTS and HMCDA type aliases of some parametric version of a generic HMC sampler type, and then add convenience constructors for those that don't require to specify anything.
from advancedhmc.jl.
But can't we have a single sampler type AND such convenience constructors?
Exactly, so in that case I suggest just using different samplers for the different types.
I imagine we could maybe make NUTS and HMCDA type aliases of some parametric version of a generic HMC sampler type, and then add convenience constructors for those that don't require to specify anything.
This is very similar to what @harisorgn wants, right? (Though I'm guessing without the integrator
field, etc.) With the type alias, I'm much more of a fan of that approach 👍
from advancedhmc.jl.
This is very similar
Yes, I guess it's quite similar - but conceptually I wouldn't make a difference between sampler and algorithm, for me the sampler is the algorithm (and hence also include other required things such as metric, kernel, integrator, ...). The fields could still be organized in a similar hierarchical way as proposed above by bundling NUTS-specific options in a NUTSOptions
struct etc - which would also make it possible to define the type aliases of the sampler/algorithm.
from advancedhmc.jl.
Makes sense.
One of my criticisms of including the integrator
, etc. is that we end up with cases where the combinations might not make sense. E.g. if I do Sampler(NUTSConfig(), termination_criterion=FixedNSteps(...))
. In that scenario, I'm clearly no longer doing NUTS
, and so what's the point?
That's why I'd rather have something like
struct Sampler{C}
config::C
end
const NUTS = Sampler{<:NUTSConfig}
and then the integrator, etc. are only constructed in the initialstep
and then passed down along with the state, instead of
Base.@kwdef struct Sampler{C,I}
config::C
integrator::I=nothing
# ...
end
const NUTS = Sampler{<:NUTSConfig}
from advancedhmc.jl.
I agree regarding the last point, I'd say anything that's always computed when sample
is called rather than specified a-priori seems to belong in the state, not the sampler. But I assumed that some of the fields suggested above are fixed upfront.
If that's not the case for any of the other fields apart from NUTSOptions, then I'd suggest a different design: Extract common options to the sampler struct, and if no such options exist, rather have an abstract supertype with NUTS etc subtypes.
from advancedhmc.jl.
If the main concern are inconsistent or unreasonable combinations of fields/settings, then one could add correctness checks in the constructors. And/Or restrict some additional type parameters in the aliases.
from advancedhmc.jl.
But I assumed that some of the fields suggested above are fixed upfront.
They could be, yeah. But then we should just have one separate, a la the current HMCSampler
, which is for power-users, where everything can be specified beforehand, instead of including this + all the other config options into a single type.
Extract common options to the sampler struct, and if no such options exist, rather have an abstract supertype with NUTS etc subtypes.
This to me seems the way to go to me. It's a) very simple, and b) easy for the user.
If the main concern are inconsistent or unreasonable combinations of fields/settings, then one could add correctness checks in the constructors.
My concern is more that we're just adding complexity that no one will ever take advantage of. If you're an advanced user who is willing to construct their own adaptor or integrator, then adding two more lines to construct the remainding objects is really not a big deal. Hence I don't think there's much use in making this alternative interface, whose main target is the layman user, sufficiently general to also cover the the scenario where the user wants to construct everything by hand; that already exists. So, IMO, this new user-interface should be dead-simple, e.g. different types for the different "classical" HMC types.
from advancedhmc.jl.
@devmotion Just to clarify, you mean we should do something like
struct HMC{C}
config::C
end
const NUTS = HMC{<:NUTSConfig}
right?
But, if we don't find any good "shared" parameters, you agree we should just do
abstract type AbstractHMCSampler end
struct NUTS <: AbstractHMCSampler
# ...
end
?
from advancedhmc.jl.
Some related discussions inspired by DynamicHMC interface #313 (comment)
from advancedhmc.jl.
I don't think we actually need:
abstract type AbstractHMCSampler end
struct NUTS <: AbstractHMCSampler
# ...
end
but just
struct NUTS <: AbstractMCMC.AbstractSampler
# ...
end
See this PR for the consensus solution we have found!
from advancedhmc.jl.
@devmotion Just to clarify, you mean we should do something like
Sorry, missed your comment 😬 My idea was slightly different - I thought your first example would motivate the second design, but as soon as there are common fields I'd use a single type.
But before commenting any further I'll have a look at the PR 🙂
from advancedhmc.jl.
This is now merged!
from advancedhmc.jl.
Related Issues (20)
- README example fails HOT 3
- Step size initialization uses GLOBAL RNG instead of reproducible RNG from caller
- Forecasting
- Feature request: HMC sampling from convex polytopes defined by linear programs
- Add nuts-rs's metric adaptation HOT 15
- More friendly default `sample` interface HOT 9
- Make `HMCState` stores `rng` in `AbstractMCMC` interface. HOT 5
- Weighted Log Likelihood #featurerequest
- Objects of type LogTargetDensity are not callable HOT 2
- Removing support for AbstractMCMC.Sample piracy HOT 4
- Retest is broken HOT 2
- Moving ReadMe to Docs
- Component Arrays with DenseEuclideanMetric fails HOT 1
- Doc string error for NUTS HOT 2
- `AbstractMCMC.step` interface broken HOT 3
- Define a model HOT 1
- In sample progress=true prints more than a progress-bar HOT 1
- Accessing samples?
- Progress message `ratio_divergent_transitions_during_adaption` bug HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from advancedhmc.jl.