dfm / corner.py

Make some beautiful corner plots

corner.py

Read the documentation

If you make use of this code, please cite the JOSS paper:

@article{corner,
  doi = {10.21105/joss.00024},
  url = {https://doi.org/10.21105/joss.00024},
  year  = {2016},
  month = {jun},
  publisher = {The Open Journal},
  volume = {1},
  number = {2},
  pages = {24},
  author = {Daniel Foreman-Mackey},
  title = {corner.py: Scatterplot matrices in Python},
  journal = {The Journal of Open Source Software}
}

corner.py's Issues

array of title formats?

Would it be easy to change title_fmt to take an array of formats so that each parameter could get its own? On a related note, the default .2f could be changed to give 2 significant figures in the uncertainty, rather than a fixed decimal place.

Documentation for all the options

I've just added a lot of new options (c.f. #41) and the docs were already out of date... need to fix that!

Errors in titles unaffected by quantiles argument

The help text (corner.py, line 65) says

show_titles : bool
Displays a title above each 1-D histogram showing the 0.5 quantile
with the upper and lower errors supplied by the quantiles argument.

but actually the errors in the titles are not affected by the quantiles argument (corner.py, line 269) but instead are hard-coded:

q_16, q_50, q_84 = quantile(x, [0.16, 0.5, 0.84],
weights=weights)
q_m, q_p = q_50-q_16, q_84-q_50

            # Format the quantile display.
            fmt = "{{0:{0}}}".format(title_fmt).format
            title = r"${{{0}}}_{{-{1}}}^{{+{2}}}$"
            title = title.format(fmt(q_50), fmt(q_m), fmt(q_p))

Shouldn't the the array passed to quantile use the quantiles argument for the lower and upper bounds? If not, shouldn't the help text say that the errors are hard-coded to the 68% confidence region?

plot upper triangle instead

I should have an upper=True keyword so I can plot the upper corner instead of the lower corner.

repeated visualization in IPython notebook

In ipython notebook 3 (I forget the exact version), calling triangle.corner without capturing the return value creates two versions of the plot. I have an example of this at http://nbviewer.ipython.org/gist/DougBurke/a13dda72ca41d0a05e7d where the Out [7] cell shows the plot twice. I show a workaround in cell 10.

Image in readme.rst is down

np.zeros () line 535

Hi all,

Besides the last modification on line 213, I think we also have to define the argument of np.zeros function (on line 535) as int.
These are the messages I got:

/numpy/lib/function_base.py:564: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
n = np.zeros(bins, ntype)
/numpy/lib/function_base.py:600: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
n += np.bincount(indices, weights=tmp_w, minlength=bins).astype(ntype)

Thanks in advance.

Best,
Mariana Penna Lima

a version with python3 compatibility on pypi ?

I see a recent fix for python3. Would it be possible to ship a 0.0.7 version on pypi ?

no graphics server and plot to pdf

Hi,

I'm running a script which invokes corner.corner, and everything goes fine in my local machine (python3).

I'm trying to run the same script in a cluster (python2.7) with no graphics device. To save the plots directly in a pdf file, I use the commands

import matplotlib
matplotlib.use('Agg')

at the beginning of my script. After including these lines, I find that corner.corner hangs.

Can someone tell me why, and how I can solve this? Does it depend on the version of python, or on the absence of a graphics device?

Thanks a lot

Apply priors on 1 and 2d histograms

Hi,
Not too much of an issue here, more of an additional feature.
The idea is to multiply the 1d and 2d histograms curves/surfaces with the respective prior of each parameter, so that the full "posterior distribution" is plotted - not just the likelihood of the raw samples.

It seemed that the "weights" arguments served another purpose, so I hacked something together. It actually holds in a tiny amount of lines.
Only one new input argument "priors" is needed. It is a N-list of (callable, dict) (for N parameters). The callable argument being the function to call with the dict as *_kwargs:
prior_prob = callable(x, *_dict).
The only drawback is the loss of the parameter "histtype" (hist_kwargs["histtype"]) for the 1d histograms, since they are now always plotted with ax.plot(...) whether they have smoothing or not.
Cheers,
G. Schworer @ceyzeriat

plot full square

I should be able to set duplicate=True and have it plot both the upper triangle and the lower triangle in a full square with every 2-d plot duplicated.

Tick fontsize control

Hello,
On a corner plot with many variables, the axis tick labels overlap with the axis labels. I was wondering how to control the tick label's font size, or chose not to display them.
Thank you!

multiple sets of points

I have K sets of points, each of which is a large number N_k of objects in D dimensions. I want to plot these all on the same corner plot in different colors. How do I do that? I am happy to write the patch.

Equal contour levels

With the recent bug highlighted in #68 the contour levels have been sorted to ensure they are in an increasing order so the call to contourf() does not fail.

I recently came across another related issue. When you are trying to plot a set of samples which have an extremely skewed (for instance, when the sampling is converging on a hard coded upper/lower limit in some model), it is possible for the contour levels to be equal and will subsequently cause contourf() to fail as there is a difference of 0 between two or more elements of the V list.

A simple fix I have used is to test for equal elements and disable the contour plotting if any are present after the V list in corner.hist2d has been sorted.

if len(V) != len(set(V)):
    logging.warn("Equal contour levels found. Explicitly disabling.")
    plot_contours = False
    no_fill_contours = True

Of course the problem could be because of a bad model or constraint leading to the bad sampling, but that's another problem for another day!

why does max have to be greater than min in range?

max must be larger than min in range parameter

But aren't some of us astronomers??

hist2d shouldn't "pop" kwargs!

can we get the 45-degree labels to pivot better?

They should pivot around their end closest to the plot, not around their centers. Know what I mean? I don't have the foo.

Overplotting

Please write an option for overplotting multiple chains on one corner plot (in different colors and/or line styles), with the largest chain extent setting the plot parameters. While you're at it, make colors/line styles options in triangle.corner.

Large memory consumption

I am using the triangle plots package together with emcee in a very simple minded way. Using emcee I generated a list of about 1,000,000 samples in 8 dimensions (using 1000 walkers and 1050 samples). and then I invoke the corner plot here is a snippet of code:

sampler = emcee.EnsembleSampler(1000, ndims,
                                ln_post_prob, args=(cs_lib, struct, bnds))
sampler.run_mcmc(starts, 1050)
num_samples = (1000 * (1050 - 50))
samples = sampler.chain[:, 50:, :].reshape((num_samples, ndims))
lbls = [r"$a_{%d}$" % i for i in range(ndims)]
ranges = [(-0.001, (0.001 + 1.1*samples[:,i].max())) for i in range(ndims)]
fig = tplot.corner(samples, labels=lbls, extents=ranges)
fig.savefig(fig_file_name)

During the MCMC sampling memory consumption of the python process is about 132MB. When the system attempts to create a corner plot using this package immediately after the sampling, memory consumption jumps to 840MB.

Repeating the exercise with 8,000,000 samples in 8 dimensions (using 4000 walkers and 2050 samples) gives memory consumption of about 624MB during sampling and then, during corner plot creation, climbs to 4.8GB.

I recognize that the growth in consumption (as a function of number of samples) is less than linear, but 4.8GB to make a corner plot for 8,000,000 samples in 8 dimensions seems excessive to me.

If this is a problem in the underlying matplotlib, I apologize for raising the issue here here, if there is some set of options that will reduce memory consumption that I neglected to use, then I apologize for not seeing them.

uncertainties on parameter values on diagonal 1D histograms

I realise playing around with the displaying of the quantiles on the 1D histograms that the uncertainties are not updated accordingly to the extremes of the percentiles values provided (stay as a default 0.16 and 0.84, in the code, line 266 of the corner.py file, Version 2.0.1).

A small hack to recover the quantiles values provided in the quantiles keywords do the trick (provided that the array as an even number).

Legend with Multiple plots

Hello,

I have two corner plots that I overlay using the fig=fig keyword. It is only with two parameters and I wanted to add a legend in the empty corner. One plots the contours blue and the other one plots red. Is there a simple way to do this with corner.py?

Thank you,
Kara

Normalising the y axis of the 1D hist Pdfs diagonals to unity!

1 - Is there any quick way to do this with corner? if not, then its unclear how to use this with weights, what is the shape of weights! because in the documents its written [nsmaples,] isn't should be [nsamples,ndim] as the same shape of x the sample array?

2 - whenever I try using weights, it gives: TypeError: hist() got multiple values for keyword argument 'weights'

Thanks in advance for your help.

Plot two corner plots on top of each other

Hello,

I was wondering, is there was a way to plot two separate corner plots on top of each other? Perhaps in different colors?

Thanks!

why can't I plot when the number of points is less than the number of dimensions?

Cython dependency

pip install triangle gives me error b/c I didn't have Cython.Distutils

after installing Cython, seems to work.

test errors on python3, matplotlib 2.0.2

Hi all,

I ran into the following test errors on a fresh install of corner today:

https://gist.github.com/tomr-stargazer/b2271fda034533d084f6db94f163fdc9

relevant software versions:
nosetests --version
nosetests version 1.3.7

python -c 'import matplotlib; print(matplotlib.version)'
2.0.2

python --version
Python 3.6.0 :: Anaconda custom (x86_64)

Rotate bottom right histogram

Please rotate bottom right histogram 90 degrees and remove axis labels so that x and y axis layouts are the same.

Smooth parameter must be a (not-necessarily integer) number of pixels

Seems to be in physical units at the moment.

bigger axis labels

When the plot gets huge, the labels get (relatively) small. This should either be fixed, or given an option=True to fix it at user's discretion.

Optimal binsize

This must be a solved problem by now. Ivezic had a suggestion in his textbook, we could think about tracking it down.

tests.py example error: Contour levels must be increasing error in

Hi! Above all I want to thank you for this greate library which I discovered while working with PyMC (https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html)

I was trying to learn more from the examples in test.py but I got this error:

Testing 'hist2d'
.. cutoff
.. cutoff2
.. basic
.. color
.. levels1
.. levels2
.. filled
.. smooth1
.. smooth2
.. philsplot
.. lowN
Traceback (most recent call last):
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 110, in
test_hist2d()
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 49, in test_hist2d
_run_hist2d("lowN", N=20)
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 31, in _run_hist2d
corner.hist2d(x, y, ax=ax, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/corner.py", line 561, in hist2d
cmap=white_cmap, antialiased=False)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/init.py", line 1812, in inner
return func(ax, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/axes/_axes.py", line 5652, in contourf
return mcontour.QuadContourSet(self, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1424, in init
ContourSet.init(self, ax, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 863, in init
self._process_args(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1445, in _process_args
x, y, z = self._contour_args(args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1540, in _contour_args
self._contour_level_args(z, args)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1189, in _contour_level_args
raise ValueError("Contour levels must be increasing")
ValueError: Contour levels must be increasing

Thanks for any advice. Also if you have any other examples I should consider first I would love to hear your opinion.

Thanks

Be sane about axis limits when over-plotting

Right now, it hard sets the limits to the range of the last set of samples that are plotted. This isn't very useful. You can hack this by using the extent keyword but it would probably be good to default to something better... like only letting the limits get bigger. Eh? 🍁

passing linestyle type arguments to histograms?

Is there any way to get dashed (or other styles) histograms? I have not been able to get it working with standard matplotlib arguments

how to make the title displays the 2-sigma errors than the 1-sigma error?

Generator for array_like

To allow for much larger data sets, it would be nice if the array_like argument could accept a generator. This would require the "data structure" for the plot to be a "histogram" (a collection of bins, and for each bin a count/frequency) and probably an additional parameter indicating the sample size (which indicates the bin resolution).

Project idea: IPython notebook gallery

@drphilmarshall suggested that a good way to demonstrate and document the features of this project would be to have an IPython notebook gallery. It's a great idea and if anyone feels like taking it on, that would be sweet! If not, I might do it some day :-)

Quantiles keyword has stopped working

When I include, for example, quantiles=[0.16, 0.5, 0.84] in a call to corner.corner I receive the following error:

ValueError                                Traceback (most recent call last)

/astro/mark/casa/casa-stable-4.7.74/lib/python2.7/casapy.py in <module>()
      1 
      2 
----> 3 
      4 
      5 

/astro/mark/.casa/lib/python2.7/site-packages/corner/corner.pyc in corner(xs, bins, range, weights, color, smooth, smooth1d, labels, label_kwargs, show_titles, title_fmt, title_kwargs, truths, truth_color, scale_hist, quantiles, verbose, fig, max_n_ticks, top_ticks, use_math_text, hist_kwargs, **hist2d_kwargs)
    254         # Plot quantiles if wanted.

    255         if len(quantiles) > 0:
--> 256             qvalues = quantile(x, quantiles, weights=weights)
    257             for q in qvalues:
    258                 ax.axvline(q, ls="dashed", color=color)

/astro/mark/.casa/lib/python2.7/site-packages/corner/corner.pyc in quantile(x, q, weights)
    411 
    412     if weights is None:
--> 413         return np.percentile(x, 100.0 * q)
    414     else:
    415         weights = np.atleast_1d(weights)

/astro/mark/casa/casa-stable-4.7.74/lib/python2.7/site-packages/numpy/lib/function_base.pyc in percentile(a, q, axis, out, overwrite_input)
   3077     a = np.asarray(a)
   3078 
-> 3079     if q == 0:
   3080         return a.min(axis=axis, out=out)
   3081     elif q == 100:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I've recently moved institution and so am working on a fresh set-up and initially thought that I hadn't set something up correctly. But then it occurred to me to check if there had been any recent changes to corner. Sure enough, some changes were made 9 days ago including this:

-        return np.percentile(x, [100. * qi for qi in q])
+        return np.percentile(x, 100.0 * q)

I suspect this is the source of the error. Was there a good reason for this change?

Triangle_plot does not deal with maskedarray

And the reason is fairly simple: np.hist* neither (as per: http://stackoverflow.com/questions/3610040/how-to-create-the-histogram-of-an-array-with-masked-values-in-numpy )

The fix is quite simple and I applied it to my fork of the code, as I couldn't commit to a new branch on dfm/triangle.py. Not sure how to merge that back in master if you are interested.

Btw, maskedarrays (np.ma) are quite a powerful tool, especially to exclude (hide) data out of prior range without actually throwing it away, or focus on a specific region, etc.

Cheers,
Guillaume

Time for a new pip package?

Hey @dfm -

AstroHackWeek 2015 is coming up, and it would be good if the kids can corner plot their sampled PDFs with PJM-compliant filled contours etc (nicely nailed by your good self in #41 and #45). What do you think about updating the package so that pip install --upgrade traiangle_plot gives them this new power?

Thanks!

Rename package

I've bugged Dan about this before on twitter. :)

There's considerable confusion caused by names in this module: The module name is triangle, the pypi name is "triangle_plot," the repo name is "triangle.py," and the main entry point within the package is corner(). Results of the confusion: #39 and #28.

I suggest renaming all of these to simply corner, which is miraculously unclaimed on PyPI. You get:

• pip install corner
• from corner import corner or import corner; corner.corner(...)

(I'm not so set on the name, just the consistency.)

Two possible downsides:

• There could still be conflicts with other PyPI packages if any of them provide a corner module or package (despite not being named "corner"). However, I didn't see any obvious candidates for this when searching "corner" on PyPI.
• Dan might not want to rename the repo, since it is so popular. I would think github redirects would take care of this though?

Any thoughts on the color of this bikeshed?

ValueError ("Contour levels must be increasing")

Greetings! I run your few line example exactly as it appears on your web sites, and it crashes as follows:

$ python testCorner0.py
Traceback (most recent call last):
File "testCorner0.py", line 6, in
figure = corner.corner(samples)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/corner.py", line 330, in corner
*_hist2d_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/corner.py", line 519, in hist2d
ax.contour(X2, Y2, H2.T, V, *_contour_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/init.py", line 1846, in inner
return func(ax, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/axes/_axes.py", line 5641, in contour
return mcontour.QuadContourSet(self, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1448, in init
ContourSet.init(self, ax, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 874, in init
self._process_args(_args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1469, in _process_args
x, y, z = self._contour_args(args, kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1564, in _contour_args
self._contour_level_args(z, args)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1200, in _contour_level_args
raise ValueError("Contour levels must be increasing")
ValueError: Contour levels must be increasing

My OS, below. Thanks, and regards to all, David.

$ lsb_release -a
LSB Version: :core-3.1-ia32:core-3.1-noarch:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: ScientificSL
Description: Scientific Linux SL release 5.5 (Boron)
Release: 5.5
Codename: Boron

Plotting a range of truth values

I'm looking for a way to plot a range of truth values, sort of like a confidence interval. Ideally this would take the form of a semi-opaque wide line, just like the standard truths function, but the width of the line would represent the confidence width.
Is there any way to do that? I have been playing around with the axes object, maybe that's a way forward.
fig = corner.corner(data)
fig.axes[3].axhline(value, lw=2)

PDF presentation

OK @dfm, here's a wishlist for those of us using triangle.py to plot samples drawn from PDFs:

• Contour levels that enclose 68 and 95% of the 2D probability
• Arbitrary levels, why not. (See #31)
• Optional Gaussian smoothing of 2D PDFs for presentation purposes
• Ditto for 1D histograms
• Uniformly filled contours to show confidence regions without being distracted by the actual density

This might take you a while, sorry - let us know when you're done! Thanks Dan.

Produce log axis corner plots

Hi there,

I'm relatively new to python and am giving your corner script a go! But I can seem to work out how to get the axes of the plots to display a log scale.

Besides logging the actual data (which doesn't do the right thing to the axes - but does give the right sort of shape to the plots), is there a way to do this?

core dump segmentation fault

I have been trying to use this library to plot my mcmc results. I have 26 free parameters that have been measured by emcee but when I tried to plot them just before the script return fig I got the core dump error message (I tested where the code would crash). How could I solve the problem. Wouldn't it be more convenient and lighter if you use
gs = gridspec.GridSpec(K, K)
instead of
fig, axes = pl.subplots(K, K, figsize=(dim, dim))
to make subplots?

Thanks in advance.

Conflict with another triangle.py (LinearNDInterpolator)

Hi- I noticed that triangle-plot conflicts with another triangle.py which seems to be necessary for the LinearNDInterpolator routine (scipy.interpolate.LinearNDInterpolator())

http://dzhelil.info/triangle/

I'm not exactly sure what to do in this case... I need the LinearNDInterpolator to perform the calculations, but the corner plots would be really nice as well... Do you know of an easy way to alias the package name? (Sorry, my ignorance of the guts of python is showing!)

Filled contours are broken in matplotlib 1.5

This isn't what is supposed to happen!!

Set contour levels

Add argument for setting contour levels

Smoothing of correlated variables

The smoothing implemented in corner.py does not handle strongly correlated variables very well.

The smoothing for 2D histograms is done with the scipy gaussian_filter function, which is isotropic. If your variables are highly correlated this is pretty distorting, since it smooths points in the wrong directions. See the Delta-M versus h subplot in the attached image, where the contour is enlarged a lot in one direction.

The easiest way to solve this in general is to either supply a covariance matrix to use an elliptical smoothing kernel, or transform to variables with unit covariance before smoothing. Neither of those is a simple change here, unfortunately.

Comments from Loredo regarding terminology and plotting parameter tweaks

In an email on 2013-11-25, Tom Loredo (Cornell) suggested:

Two suggestions:

• There does not seem to be a universal terminology for these plots,
  but the terms I see most often are "pairs plots" and "scatterplot
  matrices" (I'd never seen "corner plot" or "triangle plot" before).
  I suggest putting these in your README.rst so people
  will more easily find your solution via a web search.
• One option I have in mine is to plot the posterior samples colored
  according to time in the chain, and transparent. I use a color map
  from blue to green to red, or something like that. It provides a
  graphical 2-D convergence diagnostic: if everything is mixed, the
  predominant color is brownish; but if the chain gets stuck anywhere,
  spots of a specific color stand out. In particular, if you don't
  discard enough "burn in," there is often a blue region sitting away
  from the brownish region. Probably too busy for a final plot, but
  helpful when you're exploring/tuning. My statistician collaborators
  at Duke like this and have started using it.

Thanks Tom!