justinalsing / dlmmc Goto Github PK
View Code? Open in Web Editor NEWDynamical linear modeling (DLM) regression code for analysis of atmospheric time-series data.
License: MIT License
Dynamical linear modeling (DLM) regression code for analysis of atmospheric time-series data.
License: MIT License
hey @justinalsing! As part of the software review in openjournals/joss-reviews#1157 I create this issue to discuss the installation of automated tests using Continuous integration services (such as Travis) (see my comment (openjournals/joss-reviews#1157 (comment)) and the one of @taqtiqa-mark (openjournals/joss-reviews#1157 (comment)).
In your README file you state that the tutorial serves as a test for your suite (see ccf6b83). Nevertheless, this is not an automated test and (in terms of transparency) you should use any of the available CI services that runs your tests on every commit (e.g. Travis CI). I know, this is tedious but all open source projects that you are relying on went through this process, so why shouldn't yours as well?
Minimal requirements for your automated tests are:
jupyter nbconvert --execute
on your notebooksAdditionally, citing openjournals/joss-reviews#1157 (comment)
You should definitely think about tests to check the mathematical correctness of your models since this is the central part of your code.
Fits null and alternate specifications to simulated and empirical data. The tests would be simply checking parameters vales, and goodness of fit metrics. The 'true' or expected values should come from a reference implementation that is clearly identified, e.g. some R-project, etc. library result set, and the script used to generate the reference values would be part of the project.
Comparing to some reference R-project, etc. would be ideal, of course. Alternatively you could just use the trend of your tutorial dataset (here you know, that this one is correct) and implement a unittest that runs your four models and compares it to this reference trend. You are the expert of your software so it's up to you which metric and which accuracy you are expecting of your model (but nevertheless, please justify your choice).
Parallel test suite that exercises the parallel calculations. You could for example run your dlm_lat_alt_mpi_run.py
script in your test suite
If you address this issue in a commit or pull request, please include the link to this issue (#5) in the commit/PR-message. In that way, it will be easier for us to follow the implementation.
Hello justin,
I was going through the model description document and therein you have mentioned that AR1 model was used for Ball et al. (2017) and Ball et al. (2018) which doesn't match with the description given in those papers. In Ball et al. (2017), the model used was AR1, while AR2 was used for Ball et al. (2018). It'll be helpful for users like me if you rectify this discrepancy.
Blocking openjournals/joss-reviews/issues/1157
Likely related to #3 - in that the current instructions appear to assume alot, and those assumptions are undisclosed.
The project readme claims Linux is supported.
No instructions are provided.
This assumes Ubuntu 16.04.6 (LTS - Xenial).
Instructions, and terminal commands that worked-for-me(TM) on a DigitialOcean droplet....
You can skip step (1) if you have a base VM installed with Ubuntu 16.04.6 (LTS - Xenial)
Vagrantfile
:
root@default:~# sudo apt -y install apt-transport-https ca-certificates curl software-properties-common
root@default:~# sudo apt -y install python3=3.5.1-3
root@default:~# sudo apt -y install python3-pip=8.1.1-2ubuntu0.4
root@default:~# pip3 install --upgrade pip
root@default:~# sudo apt -y install python3-dev=3.5.1-3
root@default:~# sudo adduser dlmmc
Now, switch to use the newly created user account dlmmc
:
root@default:~# su - dlmmc
dlmmc@default:~$ pip3 install numpy scipy matplotlib netCDF4 pystan ipython[all]
dlmmc@default:~$ git clone https://github.com/justinalsing/dlmmc.git
dlmmc@default:~$ pushd dlmmc
dlmmc@default:~$ python3 compile_stan_models.py >compile_stan_models.log 2>&1
dlmmc@default:~$ jupyter-nbconvert --to notebook --execute --ExecutePreprocessor.timeout=100000 dlm_tutorial.ipynb >dlm_tutorial.log 2>&1
Given you are not using std test suites, please also add, to the installation validation instructions, what a successful run looks like.
For the above instructions:
dlmmc@default:~$ cat compile_stan_models.log
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1769d29906593e8f6fa11e816b642cff NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_323f0530039bc4ac2c22bb5250e1d6c1 NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_c3ff00cf2253f51bed2b150f31119693 NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_b9cb9e0eb2389c8a6e3078345a6a1dd4 NOW.
dlmmc@default:~$ cat dlm_tutorial.log
[NbConvertApp] Converting notebook dlm_tutorial.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3
Gradient evaluation took 0.023274 seconds
1000 transitions using 10 leapfrog steps per transition would take 232.74 seconds.
Adjust your expectations accordingly!
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan! (in 'unknown file name' at line 159)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan! (in 'unknown file name' at line 159)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Iteration: 1 / 3000 [ 0%] (Warmup)
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan! (in 'unknown file name' at line 159)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan! (in 'unknown file name' at line 159)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Iteration: 300 / 3000 [ 10%] (Warmup)
Iteration: 600 / 3000 [ 20%] (Warmup)
Iteration: 900 / 3000 [ 30%] (Warmup)
Iteration: 1001 / 3000 [ 33%] (Sampling)
Iteration: 1300 / 3000 [ 43%] (Sampling)
Iteration: 1600 / 3000 [ 53%] (Sampling)
Iteration: 1900 / 3000 [ 63%] (Sampling)
Iteration: 2200 / 3000 [ 73%] (Sampling)
Iteration: 2500 / 3000 [ 83%] (Sampling)
Iteration: 2800 / 3000 [ 93%] (Sampling)
Iteration: 3000 / 3000 [100%] (Sampling)
Elapsed Time: 267.023 seconds (Warm-up)
420.726 seconds (Sampling)
687.749 seconds (Total)
[NbConvertApp] Writing 688383 bytes to dlm_tutorial.nbconvert.ipynb
Dear @justinalsing,
I use a combination of dynamical linear models with shrinkage priors (Arnold, 2015) or repo here. The approach has several advantages:
However, I would like to use it in combination with seasonality and external regressors as in the case of dlmmc. Tests on temperature time-series with discontinuities (see below)
showed that both packages reach similar temperature trends (compare below trends using dlm-shrinkage and dlmmc).
Nevertheless, I am not much proficient in Stan. Would you be interested in helping me to modify F
and G
matrices in dlmmc to include shrinkage priors?
Thanks for your response
Ales
Blocking openjournals/joss-reviews/issues/1157
Either remove the reference to MPI support, or provide instructions and test cases to validate MPI setup.
Relevant discussion here.
Blocking openjournals/joss-reviews/issues/1157
I think @Chilipp's comment is reasonable:
As a user, I do not want to test whether it works with pip or not, I simply want to install it and it should work
There is an apparent issue with the install instructions working with conda
for some users (see #3) and sometimes with pip
with others (see #6). Also see here:
One point of contention though: I used pip to install dependencies (which works fine for me and other colleagues who use pystan extensively). However, @Chilipp did not get a working pystan set up when he did pip install, but he did with conda, so suggested that I change the install instructions to conda (which I duly did). But I don't use conda personally or have it set up
There are three OS that the application claims to work on:
Yet there is ambiguity about which installation method should be used conda
or pip
: Please clarify
hey @justinalsing! As part of the software review in openjournals/joss-reviews#1157 I report the following bug when compiling the stan models on my mac.
As outlined in the documentation, I installed all the dependencies through pip (I don't think that openmpi is available through pip)
conda create -n dlmmc python=3.7
conda activate dlmmc
pip install numpy scipy matplotlib netCDF4 pystan openmpi mpi4py
python compile_stan_models.py
The last command fails with
DIAGNOSTIC(S) FROM PARSER:
Warning: left-hand side variable (name=x_realization) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=r) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=r_realization) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=M) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_386ef73bc045cd2ec5048e789ab86d0f NOW.
Traceback (most recent call last):
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "compile_stan_models.py", line 5, in <module>
model_vanilla_ar1 = pystan.StanModel(model_code=dlm_vanilla_ar1)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/site-packages/pystan/model.py", line 349, in __init__
build_extension.run()
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 533, in build_extension
depends=ext.depends)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit status 1
here is the info on the environment:
active environment : dlmmc
active env location : /Users/psommer/miniconda3/envs/dlmmc
shell level : 2
user config file : /Users/psommer/.condarc
populated config files : /Users/psommer/.condarc
conda version : 4.5.12
conda-build version : 3.11.0
python version : 3.6.4.final.0
base environment : /Users/psommer/miniconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/osx-64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/chilipp/label/dev/osx-64
https://conda.anaconda.org/chilipp/label/dev/noarch
https://conda.anaconda.org/chilipp/label/master/osx-64
https://conda.anaconda.org/chilipp/label/master/noarch
https://conda.anaconda.org/chilipp/osx-64
https://conda.anaconda.org/chilipp/noarch
https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/osx-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/osx-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /Users/psommer/miniconda3/pkgs
/Users/psommer/.conda/pkgs
envs directories : /Users/psommer/miniconda3/envs
/Users/psommer/.conda/envs
platform : osx-64
user-agent : conda/4.5.12 requests/2.18.4 CPython/3.6.4 Darwin/17.7.0 OSX/10.13.6
UID:GID : 502:20
netrc file : None
offline mode : False
# conda environments:
#
/Users/psommer/miniconda
base /Users/psommer/miniconda3
dlmmc * /Users/psommer/miniconda3/envs/dlmmc
sys.version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 ...
sys.prefix: /Users/psommer/miniconda3
sys.executable: /Users/psommer/miniconda3/bin/python
conda location: /Users/psommer/miniconda3/lib/python3.6/site-packages/conda
conda-build: /Users/psommer/miniconda3/bin/conda-build
conda-convert: /Users/psommer/miniconda3/bin/conda-convert
conda-develop: /Users/psommer/miniconda3/bin/conda-develop
conda-env: /Users/psommer/miniconda3/bin/conda-env
conda-index: /Users/psommer/miniconda3/bin/conda-index
conda-inspect: /Users/psommer/miniconda3/bin/conda-inspect
conda-metapackage: /Users/psommer/miniconda3/bin/conda-metapackage
conda-render: /Users/psommer/miniconda3/bin/conda-render
conda-server: /Users/psommer/miniconda3/bin/conda-server
conda-skeleton: /Users/psommer/miniconda3/bin/conda-skeleton
conda-smithy: /Users/psommer/miniconda3/bin/conda-smithy
conda-verify: /Users/psommer/miniconda3/bin/conda-verify
user site dirs:
CIO_TEST: <not set>
CONDA_DEFAULT_ENV: dlmmc
CONDA_EXE: /Users/psommer/miniconda3/bin/conda
CONDA_PREFIX: /Users/psommer/miniconda3/envs/dlmmc
CONDA_PREFIX_1: /Users/psommer/miniconda3
CONDA_PROMPT_MODIFIER: (dlmmc)
CONDA_PYTHON_EXE: /Users/psommer/miniconda3/bin/python
CONDA_ROOT: /Users/psommer/miniconda3
CONDA_SHLVL: 2
PATH: /Users/psommer/miniconda3/envs/dlmmc/bin:/Users/psommer/miniconda3/bin:/usr/local/netcdf/bin:/Users/psommer/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin
PYTHONWARNINGS: ignore:mode:DeprecationWarning:docutils.io:245
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
# packages in environment at /Users/psommer/miniconda3/envs/dlmmc:
#
# Name Version Build Channel
appnope 0.1.0 <pip>
backcall 0.1.0 <pip>
bzip2 1.0.6 h1de35cc_1002 conda-forge
ca-certificates 2018.11.29 ha4d7672_0 conda-forge
certifi 2018.11.29 py37_1000 conda-forge
cftime 1.0.3.4 <pip>
cycler 0.10.0 <pip>
Cython 0.29.2 <pip>
decorator 4.3.0 <pip>
ipykernel 5.1.0 <pip>
ipython 7.2.0 <pip>
ipython-genutils 0.2.0 <pip>
jedi 0.13.2 <pip>
jupyter-client 5.2.4 <pip>
jupyter-core 4.4.0 <pip>
kiwisolver 1.0.1 <pip>
libcxx 7.0.0 h2d50403_2 conda-forge
libffi 3.2.1 h0a44026_1005 conda-forge
llvm-meta 7.0.0 0 conda-forge
matplotlib 3.0.2 <pip>
ncurses 6.1 h0a44026_1002 conda-forge
netCDF4 1.4.2 <pip>
numpy 1.16.0 <pip>
openssl 1.0.2p h1de35cc_1002 conda-forge
parso 0.3.1 <pip>
pexpect 4.6.0 <pip>
pickleshare 0.7.5 <pip>
pip 18.1 py37_1000 conda-forge
prompt-toolkit 2.0.7 <pip>
ptyprocess 0.6.0 <pip>
Pygments 2.3.1 <pip>
pyparsing 2.3.1 <pip>
pystan 2.18.1.0 <pip>
python 3.7.1 h145921a_1000 conda-forge
python-dateutil 2.7.5 <pip>
pyzmq 17.1.2 <pip>
readline 7.0 hcfe32e1_1001 conda-forge
scipy 1.2.0 <pip>
setuptools 40.6.3 py37_0 conda-forge
six 1.12.0 <pip>
sqlite 3.26.0 h1765d9f_1000 conda-forge
tk 8.6.9 ha441bb4_1000 conda-forge
tornado 5.1.1 <pip>
traitlets 4.3.2 <pip>
wcwidth 0.1.7 <pip>
wheel 0.32.3 py37_0 conda-forge
xz 5.2.4 h1de35cc_1001 conda-forge
zlib 1.2.11 h1de35cc_1004 conda-forge
please note that it works without issues if I use the conda packages (which I personally would recommend anyway). In other words, the following works without any issues:
conda create -c conda-forge -n dlmmc2 python=3.7 numpy scipy matplotlib netCDF4 pystan ipykernel mpi4py -y
conda activate dlmmc2
python compile_stan_models.py
I strongly recommend to move away from pip in your installation instructions or to be more explicit in how to install the necessary dependencies.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.