justinalsing / dlmmc Goto Github PK

View Code? Open in Web Editor NEW

23.0 23.0 5.0 28.08 MB

Dynamical linear modeling (DLM) regression code for analysis of atmospheric time-series data.

License: MIT License

Jupyter Notebook 94.31% Python 2.72% TeX 2.97%

dlmmc's People

Contributors

Stargazers

Watchers

Forkers

be557308 mnkmishra02 tommylees112 zoehuang7 mdiallofzj

dlmmc's Issues

Add CI builds

hey @justinalsing! As part of the software review in openjournals/joss-reviews#1157 I create this issue to discuss the installation of automated tests using Continuous integration services (such as Travis) (see my comment (openjournals/joss-reviews#1157 (comment)) and the one of @taqtiqa-mark (openjournals/joss-reviews#1157 (comment)).

In your README file you state that the tutorial serves as a test for your suite (see ccf6b83). Nevertheless, this is not an automated test and (in terms of transparency) you should use any of the available CI services that runs your tests on every commit (e.g. Travis CI). I know, this is tedious but all open source projects that you are relying on went through this process, so why shouldn't yours as well?

Minimal requirements for your automated tests are:

Compile the models
test all models in your package (not only the one provided in the tutorial)
Execute the tutorial notebooks to make sure they run without any exceptions (e.g. by using jupyter nbconvert --execute on your notebooks

Additionally, citing openjournals/joss-reviews#1157 (comment)

You should definitely think about tests to check the mathematical correctness of your models since this is the central part of your code.

Fits null and alternate specifications to simulated and empirical data. The tests would be simply checking parameters vales, and goodness of fit metrics. The 'true' or expected values should come from a reference implementation that is clearly identified, e.g. some R-project, etc. library result set, and the script used to generate the reference values would be part of the project.

Comparing to some reference R-project, etc. would be ideal, of course. Alternatively you could just use the trend of your tutorial dataset (here you know, that this one is correct) and implement a unittest that runs your four models and compares it to this reference trend. You are the expert of your software so it's up to you which metric and which accuracy you are expecting of your model (but nevertheless, please justify your choice).
Parallel test suite that exercises the parallel calculations. You could for example run your dlm_lat_alt_mpi_run.py script in your test suite

If you address this issue in a commit or pull request, please include the link to this issue (#5) in the commit/PR-message. In that way, it will be easier for us to follow the implementation.

Wrong reference for model used in papers

Hello justin,
I was going through the model description document and therein you have mentioned that AR1 model was used for Ball et al. (2017) and Ball et al. (2018) which doesn't match with the description given in those papers. In Ball et al. (2017), the model used was AR1, while AR2 was used for Ball et al. (2018). It'll be helpful for users like me if you rectify this discrepancy.

Linux installation instruction incomplete

Blocking openjournals/joss-reviews/issues/1157

Likely related to #3 - in that the current instructions appear to assume alot, and those assumptions are undisclosed.

The project readme claims Linux is supported.
No instructions are provided.
This assumes Ubuntu 16.04.6 (LTS - Xenial).

Instructions, and terminal commands that worked-for-me(TM) on a DigitialOcean droplet....

You can skip step (1) if you have a base VM installed with Ubuntu 16.04.6 (LTS - Xenial)

Start with a clean installation of Ubuntu 16.04.6, by either:
1. Following these instructions
2. Download Ubuntu 16.04.6 VirtualBox image, extract, import and start the VM.
3. Launch a Ubuntu 16.04.6 VM using DigitalOcean (referal-link) or some such cloud computing service provider. If you do signup to DO please signup using the referal link - its some (small) compensation for preparing these instructions for you ;)
  NOTE: If you use DigitalOcean (referal-link), the attached ZIP file contains a Vagrantfile setup to launch a DO droplet - see the assumtions list at the start of the Vagrantfile:
  1. Vagrant+Chef+DO: dlmmc-digiticalocean-chef_zero.zip
    HTH?
Start the VM, and connect to the VM terminal - or launch a terminal if using the Linux Desktop interface.
Execute the following:

root@default:~# sudo apt -y install apt-transport-https ca-certificates curl software-properties-common
root@default:~# sudo apt -y install python3=3.5.1-3
root@default:~# sudo apt -y install python3-pip=8.1.1-2ubuntu0.4
root@default:~# pip3 install --upgrade pip
root@default:~# sudo apt -y install python3-dev=3.5.1-3
root@default:~# sudo adduser dlmmc

Now, switch to use the newly created user account dlmmc:

root@default:~# su - dlmmc
dlmmc@default:~$ pip3 install numpy scipy matplotlib netCDF4 pystan ipython[all]
dlmmc@default:~$ git clone https://github.com/justinalsing/dlmmc.git
dlmmc@default:~$ pushd dlmmc
dlmmc@default:~$ python3 compile_stan_models.py >compile_stan_models.log 2>&1
dlmmc@default:~$ jupyter-nbconvert --to notebook --execute --ExecutePreprocessor.timeout=100000 dlm_tutorial.ipynb >dlm_tutorial.log 2>&1

Given you are not using std test suites, please also add, to the installation validation instructions, what a successful run looks like.

For the above instructions:

Compile STAN models

dlmmc@default:~$ cat compile_stan_models.log
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1769d29906593e8f6fa11e816b642cff NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_323f0530039bc4ac2c22bb5250e1d6c1 NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_c3ff00cf2253f51bed2b150f31119693 NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_b9cb9e0eb2389c8a6e3078345a6a1dd4 NOW.

DLM tutorial (a.k.a the test-suite)

dlmmc@default:~$ cat dlm_tutorial.log
[NbConvertApp] Converting notebook dlm_tutorial.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3

Gradient evaluation took 0.023274 seconds
1000 transitions using 10 leapfrog steps per transition would take 232.74 seconds.
Adjust your expectations accordingly!


Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan!  (in 'unknown file name' at line 159)

If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan!  (in 'unknown file name' at line 159)

If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

Iteration:    1 / 3000 [  0%]  (Warmup)
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan!  (in 'unknown file name' at line 159)

If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: multiply: B[12] is -nan, but must not be nan!  (in 'unknown file name' at line 159)

If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

Iteration:  300 / 3000 [ 10%]  (Warmup)
Iteration:  600 / 3000 [ 20%]  (Warmup)
Iteration:  900 / 3000 [ 30%]  (Warmup)
Iteration: 1001 / 3000 [ 33%]  (Sampling)
Iteration: 1300 / 3000 [ 43%]  (Sampling)
Iteration: 1600 / 3000 [ 53%]  (Sampling)
Iteration: 1900 / 3000 [ 63%]  (Sampling)
Iteration: 2200 / 3000 [ 73%]  (Sampling)
Iteration: 2500 / 3000 [ 83%]  (Sampling)
Iteration: 2800 / 3000 [ 93%]  (Sampling)
Iteration: 3000 / 3000 [100%]  (Sampling)

 Elapsed Time: 267.023 seconds (Warm-up)
               420.726 seconds (Sampling)
               687.749 seconds (Total)

[NbConvertApp] Writing 688383 bytes to dlm_tutorial.nbconvert.ipynb

Including shrinkage priors

Dear @justinalsing,
I use a combination of dynamical linear models with shrinkage priors (Arnold, 2015) or repo here. The approach has several advantages:

not required to specify a number of change points (discontinuities) in advance, it can be estimated from the data
flexible (non-parametric) model for non-linear backround trend similarly to dlmmc

However, I would like to use it in combination with seasonality and external regressors as in the case of dlmmc. Tests on temperature time-series with discontinuities (see below)

showed that both packages reach similar temperature trends (compare below trends using dlm-shrinkage and dlmmc).

Nevertheless, I am not much proficient in Stan. Would you be interested in helping me to modify F and G matrices in dlmmc to include shrinkage priors?

Thanks for your response
Ales

Clarify MPI status

Blocking openjournals/joss-reviews/issues/1157

Either remove the reference to MPI support, or provide instructions and test cases to validate MPI setup.

Relevant discussion here.

OS versions: Conda and/or Pip

Blocking openjournals/joss-reviews/issues/1157

I think @Chilipp's comment is reasonable:

As a user, I do not want to test whether it works with pip or not, I simply want to install it and it should work

There is an apparent issue with the install instructions working with conda for some users (see #3) and sometimes with pip with others (see #6). Also see here:

One point of contention though: I used pip to install dependencies (which works fine for me and other colleagues who use pystan extensively). However, @Chilipp did not get a working pystan set up when he did pip install, but he did with conda, so suggested that I change the install instructions to conda (which I duly did). But I don't use conda personally or have it set up

There are three OS that the application claims to work on:

Windows (all versions?),
MacOS (all versions?),
Linux (all distributions and all versions?)
a. Draft Ubuntu 16.04.6 instructions are in #6

Yet there is ambiguity about which installation method should be used conda or pip: Please clarify

installation failure with pip

hey @justinalsing! As part of the software review in openjournals/joss-reviews#1157 I report the following bug when compiling the stan models on my mac.

As outlined in the documentation, I installed all the dependencies through pip (I don't think that openmpi is available through pip)

conda create -n dlmmc python=3.7
conda activate dlmmc
pip install numpy scipy matplotlib netCDF4 pystan openmpi mpi4py
python compile_stan_models.py

The last command fails with

distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit status 1

DIAGNOSTIC(S) FROM PARSER:
Warning: left-hand side variable (name=x_realization) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=r) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=r_realization) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
Warning: left-hand side variable (name=M) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.

INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_386ef73bc045cd2ec5048e789ab86d0f NOW.
Traceback (most recent call last):
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
    extra_postargs)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command '/usr/bin/gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "compile_stan_models.py", line 5, in <module>
    model_vanilla_ar1 = pystan.StanModel(model_code=dlm_vanilla_ar1)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/site-packages/pystan/model.py", line 349, in __init__
    build_extension.run()
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/Users/psommer/miniconda3/envs/dlmmc/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit status 1

here is the info on the environment:

conda info -a

     active environment : dlmmc
    active env location : /Users/psommer/miniconda3/envs/dlmmc
            shell level : 2
       user config file : /Users/psommer/.condarc
 populated config files : /Users/psommer/.condarc
          conda version : 4.5.12
    conda-build version : 3.11.0
         python version : 3.6.4.final.0
       base environment : /Users/psommer/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/chilipp/label/dev/osx-64
                          https://conda.anaconda.org/chilipp/label/dev/noarch
                          https://conda.anaconda.org/chilipp/label/master/osx-64
                          https://conda.anaconda.org/chilipp/label/master/noarch
                          https://conda.anaconda.org/chilipp/osx-64
                          https://conda.anaconda.org/chilipp/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/osx-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/osx-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /Users/psommer/miniconda3/pkgs
                          /Users/psommer/.conda/pkgs
       envs directories : /Users/psommer/miniconda3/envs
                          /Users/psommer/.conda/envs
               platform : osx-64
             user-agent : conda/4.5.12 requests/2.18.4 CPython/3.6.4 Darwin/17.7.0 OSX/10.13.6
                UID:GID : 502:20
             netrc file : None
           offline mode : False

# conda environments:
#
                         /Users/psommer/miniconda
base                     /Users/psommer/miniconda3
dlmmc                 *  /Users/psommer/miniconda3/envs/dlmmc

sys.version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 ...
sys.prefix: /Users/psommer/miniconda3
sys.executable: /Users/psommer/miniconda3/bin/python
conda location: /Users/psommer/miniconda3/lib/python3.6/site-packages/conda
conda-build: /Users/psommer/miniconda3/bin/conda-build
conda-convert: /Users/psommer/miniconda3/bin/conda-convert
conda-develop: /Users/psommer/miniconda3/bin/conda-develop
conda-env: /Users/psommer/miniconda3/bin/conda-env
conda-index: /Users/psommer/miniconda3/bin/conda-index
conda-inspect: /Users/psommer/miniconda3/bin/conda-inspect
conda-metapackage: /Users/psommer/miniconda3/bin/conda-metapackage
conda-render: /Users/psommer/miniconda3/bin/conda-render
conda-server: /Users/psommer/miniconda3/bin/conda-server
conda-skeleton: /Users/psommer/miniconda3/bin/conda-skeleton
conda-smithy: /Users/psommer/miniconda3/bin/conda-smithy
conda-verify: /Users/psommer/miniconda3/bin/conda-verify
user site dirs: 

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: dlmmc
CONDA_EXE: /Users/psommer/miniconda3/bin/conda
CONDA_PREFIX: /Users/psommer/miniconda3/envs/dlmmc
CONDA_PREFIX_1: /Users/psommer/miniconda3
CONDA_PROMPT_MODIFIER: (dlmmc) 
CONDA_PYTHON_EXE: /Users/psommer/miniconda3/bin/python
CONDA_ROOT: /Users/psommer/miniconda3
CONDA_SHLVL: 2
PATH: /Users/psommer/miniconda3/envs/dlmmc/bin:/Users/psommer/miniconda3/bin:/usr/local/netcdf/bin:/Users/psommer/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin
PYTHONWARNINGS: ignore:mode:DeprecationWarning:docutils.io:245
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

conda list

# packages in environment at /Users/psommer/miniconda3/envs/dlmmc:
#
# Name                    Version                   Build  Channel
appnope                   0.1.0                     <pip>
backcall                  0.1.0                     <pip>
bzip2                     1.0.6             h1de35cc_1002    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py37_1000    conda-forge
cftime                    1.0.3.4                   <pip>
cycler                    0.10.0                    <pip>
Cython                    0.29.2                    <pip>
decorator                 4.3.0                     <pip>
ipykernel                 5.1.0                     <pip>
ipython                   7.2.0                     <pip>
ipython-genutils          0.2.0                     <pip>
jedi                      0.13.2                    <pip>
jupyter-client            5.2.4                     <pip>
jupyter-core              4.4.0                     <pip>
kiwisolver                1.0.1                     <pip>
libcxx                    7.0.0                h2d50403_2    conda-forge
libffi                    3.2.1             h0a44026_1005    conda-forge
llvm-meta                 7.0.0                         0    conda-forge
matplotlib                3.0.2                     <pip>
ncurses                   6.1               h0a44026_1002    conda-forge
netCDF4                   1.4.2                     <pip>
numpy                     1.16.0                    <pip>
openssl                   1.0.2p            h1de35cc_1002    conda-forge
parso                     0.3.1                     <pip>
pexpect                   4.6.0                     <pip>
pickleshare               0.7.5                     <pip>
pip                       18.1                  py37_1000    conda-forge
prompt-toolkit            2.0.7                     <pip>
ptyprocess                0.6.0                     <pip>
Pygments                  2.3.1                     <pip>
pyparsing                 2.3.1                     <pip>
pystan                    2.18.1.0                  <pip>
python                    3.7.1             h145921a_1000    conda-forge
python-dateutil           2.7.5                     <pip>
pyzmq                     17.1.2                    <pip>
readline                  7.0               hcfe32e1_1001    conda-forge
scipy                     1.2.0                     <pip>
setuptools                40.6.3                   py37_0    conda-forge
six                       1.12.0                    <pip>
sqlite                    3.26.0            h1765d9f_1000    conda-forge
tk                        8.6.9             ha441bb4_1000    conda-forge
tornado                   5.1.1                     <pip>
traitlets                 4.3.2                     <pip>
wcwidth                   0.1.7                     <pip>
wheel                     0.32.3                   py37_0    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
zlib                      1.2.11            h1de35cc_1004    conda-forge

please note that it works without issues if I use the conda packages (which I personally would recommend anyway). In other words, the following works without any issues:

conda create -c conda-forge -n dlmmc2 python=3.7 numpy scipy matplotlib netCDF4 pystan ipykernel mpi4py -y
conda activate dlmmc2
python compile_stan_models.py

I strongly recommend to move away from pip in your installation instructions or to be more explicit in how to install the necessary dependencies.

justinalsing / dlmmc Goto Github PK

dlmmc's People

Contributors

Stargazers

Watchers

Forkers

dlmmc's Issues

Add CI builds

Wrong reference for model used in papers

Linux installation instruction incomplete

Including shrinkage priors

Clarify MPI status

OS versions: Conda and/or Pip

installation failure with pip

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent