Comments (13)
taqtiqa-mark
commented on August 20, 2024
I wonder if it isn't worth breaking this issue into three? To be completed in this order.
- Vanilla Test suite.
- Parallel test suite.
- CI build that executes the above.
@justinalsing, just a suggestion based on personal experience:
Each of these can be non-trivial to do well, and all the moving parts can be overwhelming if you try to tackle them together.
Also be aware that CI build centric iterations can be much slower than running tests on your local machine. Generally the CI build of a branch involves all tests and is triggered by a commit that merges several changes from a scratch branch.... on your localhost you'll likely be running subsets of tests with regularity.
from dlmmc.
justinalsing
commented on August 20, 2024
Following the discussion on the main review thread about the JOSS requirements, I’ve added a new notebook (dlm_validation_tests.ipynb) with a suite of validation tests, for each of the DLM models included in the package. For each model, I (1) Generate mock data from the model (for user specified input parameters), (2) Run the DLM model on the mock data, (3) Plot the inferred model parameters against the inputs, showing that the inputs are recovered (within posterior uncertainties).
I’ve added a section to the README pointing the user to this notebook as a set of tests/validation for the full set of implemented models. I’ve also added comments in the CONTRIBUTING file, guiding contributors to include a similar test/demo of any new models implemented in this notebook.
Regarding testing the MPI script, as I discussed on the main review thread I really included this to make non MPI-savvy users lives easier rather than as an integral/critical part of the package. But I’ve emphasized in the README that the MPI script can be tested by running it with certain parameters set so such that it runs very short MCMC chains (and hence runs through very quickly).
Look forward to hearing whether you think this is all OK!
Best,
Justin
from dlmmc.
Chilipp
commented on August 20, 2024
Thanks @justinalsing ! I just started and apparently there is a file missing: utils/dlm.py
# Import required modules
import pystan
import matplotlib.pyplot as plt
import numpy as np
import time
import scipy.interpolate as interpolate
import netCDF4
import pickle
import scipy.stats as stats
from utils.utils import *
from utils.dlm import *
%matplotlib inline---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-890c5b3f6cfd> in <module>
9 import scipy.stats as stats
10 from utils.utils import *
---> 11 from utils.dlm import *
12 get_ipython().run_line_magic('matplotlib', 'inline')
ModuleNotFoundError: No module named 'utils.dlm'
from dlmmc.
justinalsing
commented on August 20, 2024
@Chilipp ah, thanks! I forgot to add it - I just added it now and pushed so it should be there now (sorry for that!)
from dlmmc.
Chilipp
commented on August 20, 2024
Thank you @justinalsing. Unfortunately it it seems like I cannot run the test 2 for dlm_vanilla_ar2 because of the following error. Can you confirm this error?
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-4-c33d4d85b5cc> in <module>
64
65 # Ok, let's run it
---> 66 fit = dlm_model.sampling(data=input_data, iter=4000, warmup=1000, chains=1, init = [initial_state], verbose=True, pars=('sigma_trend', 'sigma_seas', 'sigma_AR', 'rhoAR1', 'rhoAR2', 'trend', 'slope', 'beta', 'seasonal', 'ar'))
67
68 # Plot the results:
~/miniconda/envs/dlmmc/lib/python3.7/site-packages/pystan/model.py in sampling(self, data, pars, chains, iter, warmup, thin, seed, init, sample_file, diagnostic_file, verbose, algorithm, control, n_jobs, **kwargs)
670
671 seed = pystan.misc._check_seed(seed)
--> 672 fit = self.fit_class(data, seed)
673
674 m_pars = fit._get_param_names()
stanfit4anon_model_3780e43ed9ee465806912a94b90d6e06_7690838862458284755.pyx in stanfit4anon_model_3780e43ed9ee465806912a94b90d6e06_7690838862458284755.StanFit4Model.__cinit__()
RuntimeError: Exception: variable does not exist; processing stage=data initialization; variable name=nx; base type=int (in 'unknown file name' at line 6)
Here is some information about the working environment:
conda list
# packages in environment at /Users/psommer/miniconda/envs/dlmmc:
#
# Name Version Build Channel
appnope 0.1.0 py37_1000 conda-forge
attrs 19.1.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
blas 1.1 openblas conda-forge
bleach 3.1.0 py_0 conda-forge
bzip2 1.0.6 h1de35cc_1002 conda-forge
ca-certificates 2019.3.9 hecc5488_0 conda-forge
certifi 2019.3.9 py37_0 conda-forge
cftime 1.0.3.4 py37h917ab60_1000 conda-forge
curl 7.64.0 heae2a1f_2 conda-forge
cycler 0.10.0 py_1 conda-forge
cython 0.29.6 py37h0a44026_0 conda-forge
decorator 4.3.2 py_0 conda-forge
defusedxml 0.5.0 py_1 conda-forge
entrypoints 0.3 py37_1000 conda-forge
freetype 2.9.1 h597ad8a_1005 conda-forge
hdf4 4.2.13 hf3c6af0_1002 conda-forge
hdf5 1.10.4 nompi_h646315f_1105 conda-forge
icu 58.2 h0a44026_1000 conda-forge
ipykernel 5.1.0 py37h24bf2e0_1002 conda-forge
ipython 7.3.0 py37h24bf2e0_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.4.2 py_0 conda-forge
jedi 0.13.3 py37_0 conda-forge
jinja2 2.10 py_1 conda-forge
jpeg 9c h1de35cc_1001 conda-forge
jsonschema 3.0.1 py37_0 conda-forge
jupyter 1.0.0 py_1 conda-forge
jupyter_client 5.2.4 py_3 conda-forge
jupyter_console 6.0.0 py_0 conda-forge
jupyter_core 4.4.0 py_0 conda-forge
kiwisolver 1.0.1 py37h04f5b5a_1002 conda-forge
krb5 1.16.3 hcfa6398_1001 conda-forge
libcurl 7.64.0 he376013_2 conda-forge
libcxx 7.0.0 h2d50403_1 conda-forge
libedit 3.1.20170329 hcfe32e1_1001 conda-forge
libffi 3.2.1 h0a44026_1005 conda-forge
libgfortran 3.0.1 0 conda-forge
libnetcdf 4.6.2 h6b88ef6_1001 conda-forge
libpng 1.6.36 ha441bb4_1000 conda-forge
libsodium 1.0.16 h1de35cc_1001 conda-forge
libssh2 1.8.0 hb1dc21d_1004 conda-forge
llvm-meta 7.0.0 0 conda-forge
markupsafe 1.1.1 py37h1de35cc_0 conda-forge
matplotlib 3.0.3 py37_0 conda-forge
matplotlib-base 3.0.3 py37hf043ca5_0 conda-forge
mistune 0.8.4 py37h1de35cc_1000 conda-forge
mpi 1.0 openmpi conda-forge
mpi4py 3.0.1 py37h27a7d74_0 conda-forge
nbconvert 5.4.1 py_2 conda-forge
nbformat 4.4.0 py_1 conda-forge
ncurses 6.1 h0a44026_1002 conda-forge
netcdf4 1.4.3.2 py37hf8bc7f3_0 conda-forge
notebook 5.7.6 py37_0 conda-forge
numpy 1.16.2 py37_blas_openblash486cb9f_0 [blas_openblas] conda-forge
openblas 0.3.3 hdc02c5d_1001 conda-forge
openmpi 3.1.3 h71abe9c_1001 conda-forge
openssl 1.1.1b h1de35cc_1 conda-forge
pandoc 2.6 1 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parso 0.3.4 py_0 conda-forge
pexpect 4.6.0 py37_1000 conda-forge
pickleshare 0.7.5 py37_1000 conda-forge
prometheus_client 0.6.0 py_0 conda-forge
prompt_toolkit 2.0.9 py_0 conda-forge
ptyprocess 0.6.0 py37_1000 conda-forge
pygments 2.3.1 py_0 conda-forge
pyparsing 2.3.1 py_0 conda-forge
pyqt 5.6.0 py37hc26a216_1008 conda-forge
pyrsistent 0.14.11 py37h1de35cc_0 conda-forge
pystan 2.17.1.0 py37hf8a1672_2 conda-forge
python 3.7.1 hbdd33cc_1002 conda-forge
python-dateutil 2.8.0 py_0 conda-forge
pyzmq 18.0.1 py37h4cc6ddd_0 conda-forge
qt 5.6.2 h822fa55_1013 conda-forge
qtconsole 4.4.3 py_0 conda-forge
readline 7.0 hcfe32e1_1001 conda-forge
scipy 1.2.1 py37_blas_openblash486cb9f_0 [blas_openblas] conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 40.8.0 py37_0 conda-forge
sip 4.18.1 py37h0a44026_1000 conda-forge
six 1.12.0 py37_1000 conda-forge
sqlite 3.26.0 h1765d9f_1001 conda-forge
terminado 0.8.1 py37_1001 conda-forge
testpath 0.4.2 py37_1000 conda-forge
tk 8.6.9 ha441bb4_1000 conda-forge
tornado 6.0.1 py37h1de35cc_0 conda-forge
tqdm 4.31.1 py_0 conda-forge
traitlets 4.3.2 py37_1000 conda-forge
wcwidth 0.1.7 py_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
widgetsnbextension 3.4.2 py37_1000 conda-forge
xz 5.2.4 h1de35cc_1001 conda-forge
zeromq 4.2.5 h0a44026_1006 conda-forge
zlib 1.2.11 h1de35cc_1004 conda-forge
conda info
active environment : dlmmc
active env location : /Users/psommer/miniconda/envs/dlmmc
shell level : 2
user config file : /Users/psommer/.condarc
populated config files : /Users/psommer/.condarc
conda version : 4.6.7
conda-build version : 3.17.8
python version : 3.7.1.final.0
base environment : /Users/psommer/miniconda (writable)
channel URLs : https://conda.anaconda.org/conda-forge/osx-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/osx-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/psommer/miniconda/pkgs
/Users/psommer/.conda/pkgs
envs directories : /Users/psommer/miniconda/envs
/Users/psommer/.conda/envs
platform : osx-64
user-agent : conda/4.6.7 requests/2.21.0 CPython/3.7.1 Darwin/17.7.0 OSX/10.13.6
UID:GID : 502:20
netrc file : /Users/psommer/.netrc
offline mode : False
from dlmmc.
justinalsing
commented on August 20, 2024
Hi @Chilipp - sorry about that (I forgot to add the updated stan models on my last commit). I've added and pushed it, and tested end-to-end by re-cloning, re-compiling the models and running through the dlm validation notebook. Apologies for not doing this check beforehand - as far as I am aware it is fully functional now, just make sure you re-compile the models after you pull the latest, thanks!
from dlmmc.
taqtiqa-mark
commented on August 20, 2024
Hi @justinalsing, I'll likely get time some evening to look at this in the coming week. Do I understand correctly that you've done something like:
- Started a new VM (or container)
- Followed your instructions to setup the software stack.
- Followed your instructions to run your examples.
- Observed no errors or missing steps.
Correct?
from dlmmc.
justinalsing
commented on August 20, 2024
@taqtiqa-mark great, thanks! Yes that's correct
from dlmmc.
Chilipp
commented on August 20, 2024
Dear @justinalsing,
sorry for being so slow in this. I ran through your tests and they run through. However, there was another issue with the filenames of your regressors in the notebooks. Please note that UNIX filesystems are case-sensitive, there either please change the filenames of your regressors or change them in the notebook.
As an example: your file name is regressors/ENSO_MEI_1950_201802.txt but in your notebook you use regressors/enso_mei_1950_201802.txt. The same happened in the dlm_tutorial notebook.
But I have to say that I like your tests and they are really useful to judge the quality of dlmmc.
from dlmmc.
justinalsing
commented on August 20, 2024
Hi @Chilipp - not a problem at all, and thanks for the comment. I've now made the regressor filenames case consistent with the actual filenames in the notebook examples (and pushed the changes)
Thanks
from dlmmc.
taqtiqa-mark
commented on August 20, 2024
CI builds are not a blocking issue for me.
from dlmmc.
Chilipp
commented on August 20, 2024
Hi @justinalsing. Strictly speaking, this issue is not yet solved (although it does not affect the acceptance of openjournals/joss-reviews#1157). But if you close it, I assume this means that you do not plan to add CI builds within the near future?
from dlmmc.
justinalsing
commented on August 20, 2024
Hi @Chilipp - that's right, I don't envisage adding CI builds in the near future (although it is on my to do list)
from dlmmc.
Related Issues (7)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dlmmc.