sbi-benchmark / sbibm Goto Github PK
View Code? Open in Web Editor NEWSimulation-based inference benchmark
Home Page: https://sbi-benchmark.github.io
License: MIT License
Simulation-based inference benchmark
Home Page: https://sbi-benchmark.github.io
License: MIT License
Hello! Thanks for building out and maintaining this package :)
Because of how the name of bernoulli_glm_raw
is initialised, it always gets the same name as bernoulli_glm
.
In [26]: sbibm.tasks.get_task('bernoulli_glm').name
Out[26]: 'bernoulli_glm'
In [27]: sbibm.tasks.get_task('bernoulli_glm_raw').name
Out[27]: 'bernoulli_glm'
Is this intended? In the results dataframe, I can see it referred to as bernoulli_glm_raw
.
In [30]: sbibm.get_results()
Out[30]:
task ... folder
0 bernoulli_glm ... 09ce89a2-3a5b-49d8-8942-f2a5f925674d
1 bernoulli_glm ... 6d4b9630-51ae-497d-b74f-581b108910b6
2 gaussian_mixture ... 6b3342b4-10a4-4933-8ad5-781e97eff7be
3 two_moons ... 0d71ce91-5f5e-4d14-b68d-53cd6710ec17
4 bernoulli_glm_raw ... be9320d9-53d5-44c1-9f25-6889222b5a61
I've noticed that the prior object from task.get_prior()
is not immediately usable with the sbi
package since there are no .sample()
or .log_prob()
methods. Specifically attempting something like this fails:
prior = task.get_prior()
inference = sbi.inference.SNPE(prior=prior, ...)
Looking at the code, I imagine this could be implemented by having task.get_prior()
return a class instead of a function. Then the class could have a __call__()
method to maintain compatibility with the current API. Happy to give this a shot if you guys agree with the change.
Edit: it would actually just suffice to expose the prior_dist
:
sbibm/sbibm/tasks/slcp/task.py
Line 60 in 15f068a
when running the sbibm
demo code based on commit 074e06a, I get
import sbibm
task = sbibm.get_task("two_moons") # See sbibm.get_available_tasks() for all tasks
prior = task.get_prior()
simulator = task.get_simulator()
observation = task.get_observation(num_observation=1) # 10 per task
# These objects can then be used for custom inference algorithms, e.g.
# we might want to generate simulations by sampling from prior:
thetas = prior(num_samples=10_000)
xs = simulator(thetas)
# Alternatively, we can import existing algorithms, e.g:
from sbibm.algorithms import rej_abc # See help(rej_abc) for keywords
posterior_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_observation=1, num_simulations=100_000)
I get
task = <sbibm.tasks.two_moons.task.TwoMoons object at 0x7ff456f40f10>, num_samples = 50, num_simulations = 500, num_observation = 1
observation = tensor([[-0.6397, 0.1623]]), num_top_samples = 100, quantile = 0.2, eps = None, distance = 'l2', batch_size = 1000, save_distances = False
kde_bandwidth = 'cv', sass = False, sass_fraction = 0.5, sass_feature_expansion_degree = 3, lra = False
def run(
task: Task,
num_samples: int,
num_simulations: int,
num_observation: Optional[int] = None,
observation: Optional[torch.Tensor] = None,
num_top_samples: Optional[int] = 100,
quantile: Optional[float] = None,
eps: Optional[float] = None,
distance: str = "l2",
batch_size: int = 1000,
save_distances: bool = False,
kde_bandwidth: Optional[str] = "cv",
sass: bool = False,
sass_fraction: float = 0.5,
sass_feature_expansion_degree: int = 3,
lra: bool = False,
) -> Tuple[torch.Tensor, int, Optional[torch.Tensor]]:
"""Runs REJ-ABC from `sbi`
Choose one of `num_top_samples`, `quantile`, `eps`.
Args:
task: Task instance
num_samples: Number of samples to generate from posterior
num_simulations: Simulation budget
num_observation: Observation number to load, alternative to `observation`
observation: Observation, alternative to `num_observation`
num_top_samples: If given, will use `top=True` with num_top_samples
quantile: Quantile to use
eps: Epsilon threshold to use
distance: Distance to use
batch_size: Batch size for simulator
save_distances: If True, stores distances of samples to disk
kde_bandwidth: If not None, will resample using KDE when necessary, set
e.g. to "cv" for cross-validated bandwidth selection
sass: If True, summary statistics are learned as in
Fearnhead & Prangle 2012.
sass_fraction: Fraction of simulation budget to use for sass.
sass_feature_expansion_degree: Degree of polynomial expansion of the summary
statistics.
lra: If True, posterior samples are adjusted with
linear regression as in Beaumont et al. 2002.
Returns:
Samples from posterior, number of simulator calls, log probability of true params if computable
"""
assert not (num_observation is None and observation is None)
assert not (num_observation is not None and observation is not None)
assert not (num_top_samples is None and quantile is None and eps is None)
log = sbibm.get_logger(__name__)
log.info(f"Running REJ-ABC")
prior = task.get_prior_dist()
simulator = task.get_simulator(max_calls=num_simulations)
if observation is None:
observation = task.get_observation(num_observation)
if num_top_samples is not None and quantile is None:
if sass:
quantile = num_top_samples / (
num_simulations - int(sass_fraction * num_simulations)
)
else:
quantile = num_top_samples / num_simulations
inference_method = MCABC(
simulator=simulator,
prior=prior,
simulation_batch_size=batch_size,
distance=distance,
show_progress_bars=True,
)
> posterior, distances = inference_method(
x_o=observation,
num_simulations=num_simulations,
eps=eps,
quantile=quantile,
return_distances=True,
lra=lra,
sass=sass,
sass_expansion_degree=sass_feature_expansion_degree,
sass_fraction=sass_fraction,
)
E TypeError: __call__() got an unexpected keyword argument 'return_distances'
Hello, I want to ask when I run
from sbibm.visualisation import fig_metric results_df = sbibm.get_results(dataset="main_paper.csv") results_subset = results_df.query("task == 'two_moons'") fig = fig_metric(results_subset, metric="C2ST") # Use fig.show() or fig.save() to show or save the figure
for visualization, it will produce the error that
AttributeError: module 'altair.vegalite' has no attribute 'v4'
from ----> [4] import deneb as den
Can you help me to solve that, thank you!
I was running the benchmark and found that no method was producing accurate posteriors (according to C2ST) for the gaussian_mixture task. I wondered if the simulator has somehow changed, thereby introducing a different ground truth posterior for each saved observation.
Indeed, this simple check shows that there has been some drift in the simulator
task = sbibm.get_task("gaussian_mixture")
num_observation = 5
true_theta = task.get_true_parameters(num_observation)
sbibm_obs = task.get_observation(num_observation)
new_obs = task.get_simulator()(true_theta)
obss = torch.concat([task.get_simulator()(true_theta) for _ in range(100)])
print(
(torch.linalg.norm(sbibm_obs - obss)).mean(),
(torch.linalg.norm(new_obs - obss)).mean(),
)
This typically returns tensor(115.6793) tensor(16.9946)
.
To fix the issue, either the simulator can be returned to its previous state or we could generate new ground truth parameters and observations; however, this runs the issue of not being backwards compatible with previous versions of sbibm
.
Hello, do you have plan to re-run the benchmark for all the 1/2-dim marginals of the tasks, at least for (S)NLE and (S)NPE?
There are some works on 1/2-dim marginal-only sbi, e.g. https://arxiv.org/abs/2107.01214. However, in Fig 1 they are comparing their method trained on marginals vs other methods trained on full distributions, which is not really an apple-to-apple comparison. It'd be useful if you could also provide the baseline for marginal-only sbi. Thanks.
Hi
The pip install fails currently with the following error.
ERROR: Could not find a version that satisfies the requirement sbibm
ERROR: No matching distribution found for sbibm
I tried it in a conda env and also just python3 virutal env
sbi 0.18.0
brought in tons of changes. I was wondering if there are any plans to adopt those? If so, it might be useful to reflect performance changes in the rendered results.
For example, it might be worth considering to make the sbi version an additional field switch, e.g. like the Task
currently.
While running into sbi
API problems (see #21) I ran some unit tests locally on my laptop (mostly as a sanity check). This revealed very quickly that the unit tests, e.g. under algorithms
, are geared towards correctness of results rather than correctness of API usage. I suspect that this is also the reason why this repo doesn't use CI facilities like github-actions.
While I value correctness of results very highly, the correctness of API is a precondition. If the code breaks before a result can be obtained, there will be no result to test for correctness. ;-) It might be worthwhile to introduce faster and slower tests (similar to what sbi
does).
Hey @jan-matthis -
this is a really great project. Thanks again for making it publicly available. I really like the publication that it comes with.
For me, it would be really helpful to test drive the benchmarks in order to see how much data I need to simulate for (S)NPE. However, I only have the forward simulation - no analytical posterior. I was wondering, if it'd be helpful if I contribute such an example to this repo. In this fashion, more people can profit from this exercise. Would that be OK?
if you can, feel free to provide me some pointers on where to start. The README.md
currently says, that I should start from
https://github.com/sbi-benchmark/sbibm/blob/main/sbibm/tasks/slcp/task.py
but which functions are required?
Also (and likely later on) I'd love to know which environment on a cluster would be suitable to run the tests.
After the previously opened issue #53 got closed, when running the same code, a new bug shows up:
AttributeError: 'MCMCPosterior' object has no attribute '_mcmc_init_params'
, but happens only at the 3rd round. Which is weird because initially _mcmc_init_params is present, and then seems to disappear. The code to reproduce is the same as issue #53.
Hi
As far i can tell, this package is built using the sbi package link
The sbi library currently does not seem to support multiple observations, i.e the simulator output should have batch size of 1. So generating time series data shouldn't be possible.
This is enforced in function check_for_possibly_batched_x_shape
in
user_input_checks
In sbibm package, you have the example code with number of observations as an argument.
observation = task.get_observation(num_observation=1) # 10 per task
According to the sbi package, this shouldn't be possible. Did you use some workaround or am i misinterpreting something ?
after installing the sbibm using pip (version 1.0.8) and trying to run the code from mnle repo
when calling the line task = sbibm.get_task("ddm")
it raise a not implemented error
Just wanted to log this here, in case sbibm
will make the move to be python 3.10
compatible. Currently, pyabcranger
is not compatible with python 3.10
, see also
diyabc/abcranger#92
When trying to run snle as in the documentation example, the code runs into an error when calling posterior.copy_hyperparameters_from(posteriors[-1])
in snle.py
, as it seems like MCMCPosterir does not have any function called copy_hyperparameters_from
. What should be the correct behaviour? Would it make sense to just remove the line, or does this require a proper fix?
Here is my code to reproduce the error:
import sbibm
task = sbibm.get_task("two_moons") # See sbibm.get_available_tasks() for all tasks
prior = task.get_prior()
simulator = task.get_simulator()
observation = task.get_observation(num_observation=1) # 10 per task
# These objects can then be used for custom inference algorithms, e.g.
# we might want to generate simulations by sampling from prior:
thetas = prior(num_samples=10_000)
xs = simulator(thetas)
# Alternatively, we can import existing algorithms, e.g:
from sbibm.algorithms import snle # See help(rej_abc) for keywords
posterior_samples, _, _ = snle(task=task, num_samples=10_000, num_observation=1, num_simulations=100_000, neural_net='maf')
# Once we got samples from an approximate posterior, compare them to the reference:
from sbibm.metrics import c2st
reference_samples = task.get_reference_posterior_samples(num_observation=1)
c2st_accuracy = c2st(reference_samples, posterior_samples)
# Visualise both posteriors:
from sbibm.visualisation import fig_posterior
fig = fig_posterior(task_name="two_moons", observation=1, samples=[posterior_samples])
# Note: Use fig.show() or fig.save() to show or save the figure
# Get results from other algorithms for comparison:
from sbibm.visualisation import fig_metric
results_df = sbibm.get_results(dataset="main_paper.csv")
fig = fig_metric(results_df.query("task == 'two_moons'"), metric="C2ST")
Hello,
as I mentioned in my PR #3, there seems to be some UserWarnings raised when KDE is fit with a small number of samples. I put here a small chunk of code which reproduces the warning; that is using my code from #3, so using the ABCpy inference scheme. I have not tried with the other algorithms yet.
I realize there is not much you can do about this as it is due to KDE, but maybe it can be helpful to provide a more explicit warning message saying that the number of samples for KDE are small? Not sure, I realize also this is not super important.
import sbibm
task_name = "two_moons"
task = sbibm.get_task(task_name) # See sbibm.get_available_tasks() for all tasks
prior = task.get_prior()
simulator = task.get_simulator()
observation = task.get_observation(num_observation=1) # 10 per task
from sbibm.algorithms.abcpy.rejection_abc import (
run as rej_abc,
)
num_simulations = 1000
num_samples = 10000
posterior_samples, _, _ = rej_abc(
task=task,
num_samples=num_samples,
num_observation=1,
num_simulations=num_simulations,
num_top_samples=30,
kde_bandwidth="cv",
)
Hello!
I am trying to use your benchmarking framework for some experiments and wanted to include the lotka_volterra
example. I've managed to install julia
and diffeqtorch
as specified in your Readme.md
, however this becomes a bit of a pain if I want to run my experiments on slurm
.
I am using submitit
to run jobs on slurm
. Everytime I want to run an experiment for lotka_volterra
, I have to recreate an image to use diffeqtorch
with julia
within the slurm_setup
as follows:
executor = submitit.AutoExecutor(job_name)
executor.update_parameters(
timeout_min=180,
slurm_job_name=job_name,
slurm_time=f"{timeout_hour}:00:00",
slurm_additional_parameters={
"ntasks": 1,
"cpus-per-task": n_cpus,
"distribution": "block:block",
},
slurm_setup=[
"module purge",
"export JULIA_SYSIMAGE_DIFFEQTORCH='$HOME/.julia_sysimage_diffeqtorch.so'",
"python -c 'from diffeqtorch.install import install_and_test; install_and_test()'",
],
)
This takes forever! I was wondering if you use slurm
for your experiments and if yes, how?
More generally, sampling from the posterior is very time expensive. How do you handle this constraint? (I am talking about sampling for the lotka_volterra
but also for slcp
posteriors...)
When trying to access https://sbi-benchmark-streamlit-metrics-5h5b07.streamlit.app/, I'm getting the following error:
AttributeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).
Traceback:
File "/home/appuser/venv/lib/python3.7/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
exec(code, module.__dict__)
File "/app/streamlit/metrics.py", line 4, in <module>
from app.metrics import page_metrics
File "/app/streamlit/app/metrics.py", line 4, in <module>
from sbibm.visualisation import fig_metric
File "/home/appuser/venv/lib/python3.7/site-packages/sbibm/visualisation/__init__.py", line 1, in <module>
from .correlation import fig_correlation
File "/home/appuser/venv/lib/python3.7/site-packages/sbibm/visualisation/correlation.py", line 4, in <module>
import deneb as den
File "/home/appuser/venv/lib/python3.7/site-packages/deneb/__init__.py", line 4, in <module>
from .correlation_matrix import correlation_matrix
File "/home/appuser/venv/lib/python3.7/site-packages/deneb/correlation_matrix.py", line 6, in <module>
from .utils import Chart
File "/home/appuser/venv/lib/python3.7/site-packages/deneb/utils.py", line 11, in <module>
Chart = alt.vegalite.v4.api.Chart
Hi,
I seem to have an issue on a MacOS 14.1 with Apple M3 Max.
If I run the installation of this package:
python -m pip install sbibm
This error originates from a subprocess and is similar to this issue: diyabc/abcranger#96, opened last week.
My current Python version is 3.11.5.
However, downgrading to Python 3.9.18 and 3.10.13 does not resolve the issue.
Collecting pyabcranger>=0.0.48 (from sbibm)
Using cached pyabcranger-0.0.70.tar.gz (58 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [106 lines of output]
PYTHON VERSION : 3.11
running egg_info
egg_info (cmake)
writing dist/pyabcranger.egg-info/PKG-INFO
writing dependency_links to dist/pyabcranger.egg-info/dependency_links.txt
writing top-level names to dist/pyabcranger.egg-info/top_level.txt
Cloning into 'ranger'...
Cloning into 'wingetopt'...
-- The CXX compiler identification is AppleClang 15.0.0.15000100
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Compiler ID : AppleClang
-- The C compiler identification is AppleClang 15.0.0.15000100
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /Users/nastyakrouglova/anaconda3/bin/python (found version "3.11.5") found components: Interpreter Development.Module
-- [pmm] Using vcpkg executable: /Users/nastyakrouglova/.local/share/pmm/2.1.0/vcpkg-6f7ffeb18f99796233b958aaaf14ec7bd4fb64b2/vcpkg
CMake Error at build/_pmm/2.1.0/vcpkg.cmake:168 (message):
Failed find portfile in
/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-install-q7br6124/pyabcranger_63824d441ed449369860b427234ea34f/pybind11-nodep!
Call Stack (most recent call first):
build/_pmm/2.1.0/vcpkg.cmake:224 (_pmm_vcpkg_copy_custom_ports)
build/_pmm/2.1.0/main.cmake:46 (_pmm_vcpkg)
build/_pmm/2.1.0/main.cmake:66 (_pmm_project_fn)
CMakeLists.txt:83 (pmm)
-- Configuring incomplete, errors occurred!
[git] .gitmodules found. Cloning the submodules if necessary
[git] cloning https://github.com/fradav/ranger.git to ranger...
[git] cloning complete.
[git] cloning https://github.com/alex85k/wingetopt.git to wingetopt...
[git] cloning complete.
[cmake] configuring CMake project...
Traceback (most recent call last):
File "/Users/nastyakrouglova/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/nastyakrouglova/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/nastyakrouglova/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 480, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 23, in <module>
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/cmaketools/__init__.py", line 98, in setup
_setup(**setup_args)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
self.find_sources()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/cmaketools/cmakecommands.py", line 72, in find_sources
mm.run()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
self.add_defaults()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
sdist.add_defaults(self)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
super().add_defaults()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
self._add_defaults_ext()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 305, in get_finalized_command
cmd_obj.ensure_finalized()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
self.finalize_options()
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/cmaketools/cmakecommands.py", line 452, in finalize_options
self.cmake.configure(self.build_base, **cmake_settings)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/cmaketools/cmakebuilder.py", line 321, in configure
cmakeutil.configure(".", build_dir, *args, **kwargs)
File "/private/var/folders/ld/flvgc_x56cqgh3w4r74wm0140000gn/T/pip-build-env-90ml_iw6/overlay/lib/python3.11/site-packages/cmaketools/cmakeutil.py", line 120, in configure
sp.run(args, env=env, check=True).check_returncode()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/nastyakrouglova/anaconda3/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cmake', '.', '-B', 'build', '-DPython_EXECUTABLE=/Users/nastyakrouglova/anaconda3/bin/python', '-DPYABCRANGER=TRUE', '-DUSE_MKL:BOOL=FALSE', '-DCMAKE_BUILD_TYPE:STRING=Release', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE:STRING=Release']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Hi, i'm implementing a new task for a university project but i cannot seem to find the way to input my epxerimental data/observations. Do i add the csv file manually? Or is it meant to be created by the simulator? thank you
When I try to run the example from the first page, namely the one below, I get the following error.
Running REJ-ABC
Running 100000 simulations.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100000/100000 [00:00<00:00, 3258243.29it/s]KDE on 100 samples with bandwidth option cv
Traceback (most recent call last):
File "experiment.py", line 18, in <module>
from sbibm.metrics import c2st
File "/home/ben/sci/sbibm/sbibm/metrics/__init__.py", line 2, in <module>
from sbibm.metrics.ksd import ksd # noqa: F401
File "/home/ben/sci/sbibm/sbibm/metrics/ksd.py", line 7, in <module>
from sbibm.third_party.kgof.kernel import KGauss
File "/home/ben/sci/sbibm/sbibm/third_party/kgof/kernel.py", line 15, in <module>
import kgof.config as config
ModuleNotFoundError: No module named 'kgof'
import sbibm
task = sbibm.get_task("two_moons") # See sbibm.get_available_tasks() for all tasks
prior = task.get_prior()
simulator = task.get_simulator()
observation = task.get_observation(num_observation=1) # 10 per task
# These objects can then be used for custom inference algorithms, e.g.
# we might want to generate simulations by sampling from prior:
thetas = prior(num_samples=10_000)
xs = simulator(thetas)
# Alternatively, we can import existing algorithms, e.g:
from sbibm.algorithms import rej_abc # See help(rej_abc) for keywords
posterior_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_observation=1, num_simulations=100_000)
# Once we got samples from an approximate posterior, compare them to the reference:
from sbibm.metrics import c2st
reference_samples = task.get_reference_posterior_samples(num_observation=1)
c2st_accuracy = c2st(reference_samples, posterior_samples)
# Visualise both posteriors:
from sbibm.visualisation import fig_posterior
fig = fig_posterior(task_name="two_moons", observation=1, samples=[posterior_samples])
# Note: Use fig.show() or fig.save() to show or save the figure
# Get results from other algorithms for comparison:
from sbibm.visualisation import fig_metric
results_df = sbibm.get_results(dataset="main_paper.csv")
fig = fig_metric(results_df.query("task == 'two_moons'"), metric="C2ST")
Hello, thank you so much for creating this useful framework!
I am one of the mantainers of the ABCpy package (https://github.com/eth-cscs/abcpy); I would like to contribute by wrapping it for this benchmarking tool; in this way, some ABC algorithms which are not presently included in your analysis and are implemented in ABCpy could be included (see the README.md in the link above).
I am just opening this issue to let you know I'll try doing this, and asking if you have any suggestions for how to proceed in the best possible way. I think from your guidelines I need to create wrapping functions, so I'll try doing that and provide some updates if I encounter any problem.
Not sure I am overseeing something, but the run
methods in the algorithms
only return the predicted samples - nothing else.
It might be worthwhile to consider refactoring this, so that each python module in the algorithms
directory offers to return the obtained posterior. This would entail in pseudo code:
def train(...):
return trained_objects
def infer(...)
return predicted_objects
def run(...):
trained_objects = train(...)
predicted_objects = infer(trained_objects, ...)
return predicted_objects
This refactoring should/would not change the API which is used downstream. It would however allow more analyses on the obtained posterior (mean/median map estimation versus SGD based map estimation etc).
Thanks for building out and maintaining this package! There was definitely a need for something like this in the ABC/Likelihood Free community.
I'm hitting a seemingly stray dimension in mcabc.py
:
from sbibm.algorithms import rej_abc
task = sbibm.get_task("two_moons")
posterior_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_observation=1, num_simulations=100_000)
which is returning a stacktrace like:
ValueError Traceback (most recent call last)
<ipython-input-128-10fe8b131cec> in <module>
1 from sbibm.algorithms import rej_abc
2 task = sbibm.get_task("two_moons")
----> 3 posterior_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_observation=1, num_simul
ations=100_000)
~/.pyenv/versions/miniforge3-4.9.2/lib/python3.8/site-packages/sbibm/algorithms/sbi/mcabc.py in run(t
ask, num_samples, num_simulations, num_observation, observation, num_top_samples, quantile, eps, dist
ance, batch_size, save_distances, kde_bandwidth, sass, sass_fraction, sass_feature_expansion_degree,
lra)
118 if num_observation is not None:
119 true_parameters = task.get_true_parameters(num_observation=num_observation)
--> 120 log_prob_true_parameters = posterior.log_prob(true_parameters)
121 return samples, simulator.num_simulations, log_prob_true_parameters
122 else:
~/.pyenv/versions/miniforge3-4.9.2/lib/python3.8/site-packages/pyro/distributions/empirical.py in log
_prob(self, value)
94 if self._validate_args:
95 if value.shape != self.batch_shape + self.event_shape:
---> 96 raise ValueError("``value.shape`` must be {}".format(self.batch_shape + self.
event_shape))
97 if self.batch_shape:
98 value = value.unsqueeze(self._aggregation_dim)
ValueError: ``value.shape`` must be torch.Size([2])
A bit of digging shows that the shape of true_parameters
in this is coming out at [1,2]
. Changing this line to log_prob_true_parameters = posterior.log_prob(true_parameters.squeeze())
does indeed make this run.
However, I'm not sure if the correct fix involves squeezing the tensor further upstream?
Thanks for any help!
When installing sbibm
via pip install sbibm
, recent updates like changing sbi.mcmc.mcmc
to sbi.samplers.mcmc.mcmc
(#59) are not included.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.