astropy / photutils

Astropy package for source detection and photometry. Maintainer: @larrybradley

Photutils

Photutils is an Astropy package for detection and photometry of astronomical sources.

Please see the online documentation for installation instructions and usage information.

Citing Photutils

If you use Photutils for a project that leads to a publication, whether directly or as a dependency of another package, please include the following acknowledgment:

This research made use of Photutils, an Astropy package for
detection and photometry of astronomical sources (Bradley et al.
<YEAR>).

where (Bradley et al. <YEAR>) is a citation to the Zenodo record of the Photutils version that was used. We also encourage citations in the main text wherever appropriate. Please see the CITATION file for details and an example BibTeX entry.

License

Photutils is licensed under a 3-clause BSD license. Please see the LICENSE file for details.

photutils's Issues

Integrate findobj in photutils

@larrybradley has developed findobj code, docs and mentioned via email that it should eventually be merged into photutils.

We can use this issue to discuss API (which parts fit where in photutils), timeline, ...

readthedocs plot doesn't work

The plot directive doesn't work for photutils on readthedocs.

I tried to fix it in #58, but the error still is the same.

@astrofrog @eteq Any idea how to fix this?

Example how to run photutils on CCDData

For some aspects of the discussion in astropy/astropy-api#5 I think it would be helpful if someone wrote a short IPython notebook that tries to do photometry on ccdproc.CCDData.

If you don't like the current photutils API (e.g. that gain is a separate parameter instead of a metadata attribute of the CCDData object), please also include a "use case" that shows exactly how you'd like it to work.

I'm not working with CCD data, so there's almost 100% chance I'd do it incorrectly if I tried, but maybe someone has the skills and time to do this?
Once we have agreed on the API and implemented it this could become part of the docs.

cc @mwcraig @crawfordsm @bsipocz

questions about development directions

I just wrote a Region class that stores circular or irregular polygon apertures, knows how to expand them to make annular background regions, and is in sky coords and wcs-aware so you give it images and it returns aperture photometry. So

1. are you intending/hoping to make this package wcs-aware, or is the intent to keep things modular and write other things that call this package (I was thinking about making my code call this to do the actual calculations, once I had done the sky->pixel conversion)
2. are there plans for irregular polygon, and/or objections to me adding it to this (which would make it trivial for me to then have my code call this to do the calculations - my code doesn't do subpixel calculations for example)

my bg: lots of astronomical coding experience, but a lot of it old-school, i.e. OOP comes a little harder to my brain, and although I've included python in my tools for 4-5 years, I might need some wrist-slapping to fit your standards. I list Tom as my character reference since he knows me well :)

thanks
remy

Enabling doctests for photutils

This is a reminder to enable doctest for the photutils' documentation

Introduce PixelCoordinates class

As discussed in today's photutils hangout (see minutes), we propose to introduce a PixCoord class and use it throughout photutils.

This is a first step towards a simple and consistent API. Currently e.g. the aperture_photometry function takes two arguments xc and yc and the psf_photometry function (to be merged today) takes a single argument positions. Soon we also want to support SkyCoord coordinates, at least in some convenience functions ... this will be discussed in a separate issue later.

Once Generalised WCS becomes available in Astropy it will probably contain a class similar to PixCoord and we'll use it, but for now we'll just make our own in photutils.

cc @astrofrog @eteq @bsipocz

Register photutils on PyPI

Could you please register photutils on PyPI to avoid someone else making a Python package with that name?

See astropy/package-template#17 .

aperture_photometry got slower between PRs 118 and 108: find the reason and fix it

Benchmark results are here for #118 #108 and #107:

https://gist.github.com/bsipocz/07398ec587b740c31a5f

Add tests for detection and morphology

This is a reminder for me to work on the tests.

Implement accuracy tests for exact modes

This is a reminder that accuracy tests are needed for the exact mode circular and elliptical circular and annulus photometry.

Make aperture center coordinate an aperture member

As discussed in today's photutils hangout (see minutes), we propose to make the aperture center coordinate a member of the following aperture classes (see here: CircularAnnulus, CircularAperture, EllipticalAnnulus, EllipticalAperture, RectancularAperture.

The motivation for this change is that it leads to a simpler API, e.g. aperture_photometry only accepts an Aperture object instead of separate xc and yc arguments ... e.g. for the planned PolygonAperture there is no need for a center. And further in the future this should make it simpler to use Region classes in photutils as proposed in the Generalised WCS once these become available in the Astropy core.

We think this change should not affect performance (a CircularAperture would have numpy array or Quantity members for xc, yc, radius`), but there have been concerns voiced by some that it will ... please do contribute a benchmark if you need speed!

Create geometry sub-package and expose overlap functions

I find the functions in photutils.circular_overlap useful for implementing analysis methods in gammapy (see #30).

Unfortunately they don't show up in the docs (see here and the don't have helpful docstrings (i.e. mention what the inputs / outputs are):

In [3]: import photutils.circular_overlap as c
In [4]: c.circular_overlap_grid?
Type:       builtin_function_or_method
String Form:<built-in function circular_overlap_grid>
Docstring:
For a circle of radius R, find the area of overlap in each element on
a given grid of pixels, using either an exact overlap method, or by
subsampling a pixel.

@astrofrog Can these functions be exposed to photutils users or do you think they should remain photutils internal implementation that should not be visible in the docs?

EDIT: Based on @astrofrog's feedback I'm changing the issue title from "Document low-level overlap functions" to "Create geometry sub-package and expose overlap functions"

Bug in aperture photometry radius definition

Shouldn't the following example give pi/4?

In [20]: photutils.aperture_circular([[1.]], 0.5, 0.5, 0.5)
Out[20]: 0.1963495408493621

cc @kbarbary @cdeil

I don't have time to investigate today, but if anyone else has a chance, please feel free to!

Dependencies: Cython

I just found this great looking package, but it appears the documentation is missing a dependency. I couldn't install photutils until I had installed Cython. It took me a while to figure that out, since it wasn't clear from the error messages I got when running a sudo ./setup.py build

Make PSF/PRF fitting robust to NaN values

As commented in #26:

It may be that we should make sure the fitting is robust to there being NaN values in the PRF. At the end of the day, if the PRF has NaN values, we should just ignore them with the fitting and fit only where there are valid model and data values?

Failing tests in test_findstars

I've failing findstars tests, e.g. astropy table t from line 214 doesn't seem to be compatible with an np.array.view(np.float).

In [52]: t = Table.read(datafn, format='ascii')

In [53]: np.array(t).view(np.float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-53-ad5b273f4449> in <module>()
----> 1 np.array(t).view(np.float)

ValueError: new type not compatible with array.

Setup output is here:
https://gist.github.com/bsipocz/c298d2f7fb1c88d057a4

Make ``Aperture`` objects support arrays of apertures

This should be simple enough following astropy core examples, e.g. SkyCoord or Time

Import fails when importing from source directory

In a clean install of photutils (i.e. fresh checkout, new environment) import photutils fails in the top-level directory. The import works fine in the other directories I tried.

git clone https://github.com/astropy/photutils.git
cd photutils
python setup.py install
python -c "import photutils"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "photutils/__init__.py", line 15, in <module>
    from .detection.core import *
  File "photutils/detection/core.py", line 5, in <module>
    from ..utils.scale_img import img_stats
  File "photutils/utils/__init__.py", line 9, in <module>
    from .sampling import *
ImportError: No module named sampling

Add more tests for aperture photometry code

As can be seen in the coverage report for aperture.py, there are some important lines that are completely untested.

@bsipocz Can you please add more tests for the aperture photometry code?
(I know you're working on other things already ... I'm just creating this issue as a reminder ... it's completely up to you what you want to do first.)

Implement bilinear interpolation

At the moment, the photometry assumes nearest-neighbor interpolation. It would be nice to implement bilinear interpolation as it is flux-conserving and maybe more realistic.

Add overview of available datasets and fetch_all_remote function

We should generate an overview table summarising the available datasets (image size, dtype, file size) and add a function that fetches all remote data files into the cache on the user's machine so that after calling it she can use them all offline.

Feature request: Access to pixels in aperture

I want to do aperture photometry by subtracting out the median within an annulus around my source. This is a slightly more robust way to compute backgrounds in a crowded field. To do this, I need access to the pixel values in the background annulus.

Move sampling functions to scikit-image

This is a reminder for myself to move the "sampling" functions to `scikit-image as discussed here: #84 (comment)

PSF / PRF extraction method

In this issue I'd like to discuss if the current PRF extraction and application method makes sense or whether it can be replaced by an oversampled PSF estimation.

cc @astrofrog @bsipocz @larrybradley @adonath @eteq

Current method

@adonath implemented as part of his GSoC the create_prf function, which creates a DiscretePRF object which is then used via psf_photometry for PSF fitting photometry to estimate source fluxes (and optionally positions). A full example for Spitzer data analysis using this is here.

If I understand correctly the DiscretePRF class represents the PSF pre-sampled for a 2d grid of sub-pixel positions (i.e. 100 PSF images for an oversampling factor of 10), and the way PRF estimation in create_prf works is that only stars where the position falls into a given sub-pixel are used in psf_photometry when fitting data with that PRF.

The problem

The problem with the current approach is that if the number of stars is limited, then there will be subpixels with few or no star to estimate the PSF for that subpixel, resulting in failing or poor psf_photometry results.

A proposal

How about instead of a function create_prf that estimates a DiscretePRF model, we write a create_psf function that estimates a DiscretePSF model?
But instead of being a 4-dimensional array like DiscretePRF of shape (nx, ny, n_subpix, n_subpix) it's a 2-dimensional array of shape (nx * n_subpix, ny * n_subpix), i.e. simply a high-resolution image of the PSF ... if I'm not mistaken that's also the format psfex uses.
create_psf would re-sample the stars onto this high-resolution image and then psf_photometry would re-sample the high-resolution PSF image onto the actual data grid or pre-compute a PRF object if that is actually needed for speed.

The big difference would be that the PSF used in psf_photometry for every star would contain the info from every star available for PSF estimation, thus largely getting rid of the problem described above.

Am I missing something and there's a good use case for the current method that would not be covered by this alternative proposal? If so, can you please point me to a webpage or paper describing why it's better?

Rename annulus functions

I would like to suggest the following renames:

annulus_circular -> aperture_circular_annulus
annulus_elliptical -> aperture_circular_annulus

I've had comments that the current names are a bit confusing, and it would be better for all the functions that do aperture photometry to start with aperture.

An alternative would be to combine annulus_circular and aperture_circular into a single function that takes an optional r_in argument. If not specified, the aperture is cir cular, otherwise it's a circular annulus.

@kbarbary @cdeil - any thoughts on this?

Remove photutils.utils functions

after astropy/imageutils#11 is merged.

Move gain to ``ccdproc`` package

There was a short discussion whether dealing with gain can/should be moved to the ccdproc package. If moved, it need to be well documented how to work out the photometry errors in aperture_photometry.

cc: @astrofrog, @cdeil, @crawfordsm

weighted aperture photometry / PSF

Hi! Do you have any routines to perform weighted aperture photometry (i.e. multiply aperture by a PSF function), and if not is there any chance you might be able to add this on short timescales? Thanks!

Use the SEP C library in photutils

I'm pleased to see that photutils is getting more attention these days! This can only be good.

This is a suggestion that one could gain considerable benefits by implementing more functionality in pure C, and wrapping it using Cython, particularly in the aperture photometry functionality. Note that this is not a comment on API design. One could always build arbitrarily complex APIs on top of the low-level interface.

There would be three main benefits of writing more in C:

Performance As one example, aperture_photometry, as written, creates many temporary arrays that could be avoided by writing more of the function in C. As a demonstration, I have a nascent C library for detection and photometry here: https://github.com/kbarbary/sep . I wrote a simple wrapper using Cython that does circular aperture photometry: https://github.com/kbarbary/seppy . Here is a comparison between photutils and the wrapper, for 1000 apertures in a 2k x 2k image (various aperture sizes):

$ python bench.py
r = 3.0
  photutils: 130.003929 ms
  sep wrapper: 4.734993 ms
r = 5.0
  photutils: 138.437033 ms
  sep wrapper: 8.326054 ms
r = 10.0
  photutils: 158.399105 ms
  sep wrapper: 18.796921 ms
r = 20.0
  photutils: 206.316948 ms
  sep wrapper: 41.263103 ms

(results from running the file: https://github.com/kbarbary/seppy/blob/master/bench.py)

While this is not exactly a one-to-one comparison (the photutils function has to parse a few more optional parameters) it should give an idea of the magnitude of possible improvement.

Simplicity Some problems are well-suited to array operations. For such problems, writing expressions array-wise makes everything easier to understand. Aperture photometry is not such a problem. The current implementation tries to force the problem into an array-shaped box and as a result, ends up being a lot more complex than need be. I personally find it confusing, and I wrote a lot of it! In contrast, both the C function ( https://github.com/kbarbary/seppy/blob/master/src/aper.c#L53 ) and the wrapper ( https://github.com/kbarbary/seppy/blob/master/sep.pyx#L23 ) for sep are quite straightforward once you get past the fact that you have to read C.

Legacy If we create a pure C library for this purpose, we'll get the added benefit of having a nice C library. This could be used for C programs or wrapped in other languages (we may not be using Python forever...). To my knowledge, a widely used library like this does not yet exist.

Update getting started example

@bsipocz Could you please update the example in the Getting Started section, following the note there

Eventually this will contain an example showing object detection and photometry used in series.

How about this?

1. cut out the central part of the example star image?
2. run one of @larrybradley's detection methods
3. run aperture photometry
4. plot the image and circles to illustrate the detections and apertures as done here and maybe plot a flux histogram to illustrate the result of the photometry?

I realise that steps 1 and 4 are just a distraction in the code example, but having a some nice example images to visually summarise what photutils is about on the front page is maybe worth it? (Or maybe we can define a plot_apertures utility function to keep our examples in the docs short?)

Correct coverage measurement for photutils

In issue #79 @larrybradley mentioned that there are inconsistencies between the coverage as measured on coveralls and the values he gets locally.

Let's use this issue to try to understand what is going on and get "correct" numbers everywhere.

For the latest photutils merge commit 75a97e3 we get:

• coveralls shows 88.49%, 0.88 HITS PER LINE / 1446 OF 1634 RELEVANT LINES COVERED
• On my machine I see 91% coverage and 1587 statement lines, 47 excluded lines and 145 missing lines.

If I'm not mistaken, the difference is that coveralls counts "excluded" lines as non-covered, whereas on my machine those are simply ignored when measuring coverage.

@larrybradley Does this make sense?

Add tests and documentation for downsample and upsample

This is a reminder.

Remove convenience functions

As discussed on the hangout yesterday, we should remove the convenience functions annulus_elliptical, aperture_elliptical, aperture_circular, and annulus_circular. The docs will need to be updated accordingly.

Closes #34

Add more tests for PSF / PRF code

A few of the PSF / PRF functions are currently untested and tests should be added:

• fix_prf_nan
• the NaN-handling part in create_prf
• subtract_psf

cc @adonath

write unit tests for geometry package

as mentioned in #99, unit tests are missing for the geometry package . Also check whether circular_overlap_grid is faster than elliptical_overlap_grid, if not then get rid of it. Checked, and it's indeed significantly faster.

Make the geometry APIs uniform.

Aperture, and psf_photometry, and create_psf to accept non-zipped xc and yc coordinate lists

Currently both the Aperture objects, and psf_photometry, and create_psf accept only zipped x,y coordinate pairs/list of pairs as position. They are expected to accept xc and yc lists, similarly to WCS:

>>> import numpy as np
>>> import photutils
>>> data = np.ones((100, 100))
>>> xc = [10., 20., 30., 40.]
>>> yc = [10., 20., 30., 40.]
>>> apertures = photutils.CircularAperture((xc, yc), 3.)
>>> flux = photutils.aperture_photometry(data, apertures)

Add rotate methods to Apertures

If I add an Aperture.rotate(x, y, angle) method for all apertures, would you want it?

Use case: for gamma-ray analysis we need for a given source region "reflected" regions for background estimation around the observation position:
http://www.aanda.org/index.php?option=com_image&format=raw&url=/articles/aa/full/2007/18/aa6674-06/img41.gif

Update docs now that aperture broadcasting is removed

Aperture broadcasting has been removed in #37.

The docstring and docs need to be updated ... alternative examples should be given of how one can create arrays of apertures and pass those in instead of letting aperture_photometry do the broadcasting.

Add circle-circle intersection area function

I'm currently integrating PyFACT into gammapy (gammapy/gammapy#68).

There's a function that computes the circle-circle overlap area that I think would fit better in photutils, because it already contains similar functionality:
circle_circle_intersection

This could fit in photutils.geometry.

TypeError from photutils unit tests with numpy 1.8

I'm getting this TypeError when running the photutils unit tests:
https://gist.github.com/cdeil/8974062#file-gistfile1-txt-L384

The issue doesn't show up on travis-ci, maybe the problem appeared in numpy 1.8?

Write PSF photometry docs

@adonath added PSF photometry methods to photutils in #26.

At the moment no tutorial-style docs are included in the photutils repo, but there is an external notebook: http://nbviewer.ipython.org/gist/adonath/6550989/PSFPhotometrySpitzer.ipynb

This or something similar should be integrated into the main photutils docs.
(and the link to the external notebook in the psf_photometry docstring needs to be removed)

@astrofrog Are the example files small enough that they can be in the repo or where should they be located?

Use asv to track performance

Once the user-facing API has settled, it would be good to set up a series of performance benchmarks using http://spacetelescope.github.io/asv/index.html since it would be interesting to see if any changes introduce any significant slow-downs.

It is also possible to do it before the API settles, but each benchmark test will need to hard-code the different ways of doing the same operation.

Separate high-level and low-level API?

Currently the photutils API contains high-level functions like aperture_photometry as well as low-level functions like centroid_1dg.

I think there is agreement that more low-level functions and classes should be exposed in the photutils API (see #8 and #31) so that it becomes a toolbox (e.g. I'd like to use these low-level functions in gammapy to implement gamma-ray specific detection methods).

The drawback is that it gets harder for end-users to find the main common high-level functions and classes (see current version of the photutils docs on what the status is).

As discussed previously, the solution to this problem is separate namespaces and docs sections for the high-level (end-user) and low-level (toolbox for reuse) functionality.

I see the following two options where to put the high-level API:

1. A namespace photutils.ui ("ui" for user interface) or photutils.api ("api" for application programming interface". E.g. Sherpa uses sherpa.ui, Skyfield uses skyfield.api and Statsmodels uses statsmodels.api
2. Use the top-level photutils namespace (this is what we have now).

@astrofrog @eteq @bsipocz @larrybradley @kbarbary Agreed that having the high-level API in a separate namespace is a good idea? Any preference for using photutils or photutils.api or photutils.ui?

Sphinx build error with Anaconda Python 3.4 on Ubuntu 12.04

I'm trying out photutils master with Anaconda Python 3.4 on Ubuntu 12.04 and Astropy 0.4.

The unit tests pass, but build_sphinx has a few issues:
https://gist.github.com/cdeil/d7d484b7f331dbc21e49

@bsipocz I think in this case all that's needed is to type list(positions) instead of positions:

/home/deil/software/photutils/docs/photutils/index.rst:85: WARNING: Exception occurred in plotting index-1
from /home/deil/software/photutils/docs/photutils/index.rst:
Traceback (most recent call last):
File "/home/deil/anaconda3/lib/python3.4/site-packages/matplotlib/sphinxext/plot_directive.py", line 515, in run_code
exec(code, ns)
File "<string>", line 12, in <module>
File "/home/deil/software/photutils/build/lib.linux-x86_64-3.4/photutils/aperture.py", line 128, in __init__
"expected got '{0}'.".format(positions))
TypeError: List or array of (x,y) pixel coordinates is expected got '<zip object at 0x7f4d68023ec8>'.

This might be an incorrect LaTeX expression or an issue with my LaTeX or Sphinx installation:
https://gist.github.com/cdeil/d7d484b7f331dbc21e49#file-gistfile1-txt-L152

@astrofrog Finally, this error is either a bug in astropy-helpers or means we have to update our the submodule pointer for photutils?

 File "/home/deil/software/photutils/astropy_helpers/astropy_helpers/sphinx/ext/viewcode.py", line 132, in collect_pages
for modname, entry in env._viewcode_modules.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'

Travis-ci is pretty much the same OS and Python install ... I wonder why these issues show up for me, but not on travis-ci...

Zoom function

Hey @astrofrog, @kbarbary

I'm trying to build an IFUutils packages (or maybe just a function) to extract an aperture in a cube. similar to @keflavich I need to get at the pixels to do the median instead of the sum.
But besides the point I found that you wrote your own downsample and I was wondering why you don't use ndimage.zoom. Does that not do the required operation?

I opened a notebook to play around @ astropy-notebook.cita.utoronto.ca:8888 - the pwd is uoft (it's a virtual machine - if it get's trashed it doesn't matter).

Fix failing tests on Linux

I've set up Travis testing for photutils, and in the process uncovered that some tests are failing there (on Linux) whereas they pass on Mac. I'm investigating.

The link to the latest travis build is https://travis-ci.org/astropy/photutils/builds/3464549 - any future pull requests will get tested by Travis.

Add `make_gauss_catalog` and `make_gauss_image` functions

Once #54 is merged I think we should move the code that generates the example dataset in photutils/detection/tests/test_findstars.py to photutils/datasets/make.py.
This will make it available in other parts of the unit tests and e.g. to save it to file and run it through e.g. IRAF or SExtractor to check the results.

I think the best would be to add make_gauss_catalog and make_gauss_image functions to simulate catalogs of Gaussian sources with known position, flux, extension and then to save one small example as a FITS file in photutils/datasets/data/gauss_sources.fits.gz and use that for unit tests.

@larrybradley I can add the functions, but I'll need you or someone else to run the image through IRAF as I have no experience with that. Or maybe you could add a wiki page with the commands and parameters you used to generate the reference results ascii files in photutils/detection/tests/data?

Projection of apertures along a dimension: photutils->3D spectrum extraction

This is a feature request for IFU and Spectral Cube purposes. A wrapper to extract photometry from each channel in a cube (and preserve the relevant WCS information) would be very helpful.

For radio cubes, this is generally trivial (i.e., photutils probably isn't even needed), but for IFUs you often want to weight by the PSF to do "optimal" spectral extraction for stars.

Modeling-related changes to PSF/PRF photometry

Comments from @eteq, copied from #26:

• First, a few comments about how astropy.modeling is used here. It looks to me like DiscretePRF and GaussianPSF are the only allowed classes, rather than arbitrary models from astropy.modeling.models. This seems to mostly stem from the use of the fit method of the PSF/PRF classes. Why is this necessary? I can certainly imagine wanting to try other PSF models (most crucially, the Moffat function). So I think it would be better if DiscretePRF and GaussianPSF obeyed standard convnetions for Fittable2DModels, and the work of massaging those models into a form appropriate for psf_photometry should be done elsewhere (either a container class or in the psf_photometry function itself. Otherwise, we effectively have two slightly different model-fitting frameworks
• Why doesn't GaussianPSF use modeling.models.Gaussian2D? It seems to have implemented all the gaussian behavior separately?
• Why is the fitter hard-coded into the PSF/PRF class instead of following the convention used in astro.modeling of having it be a separate object? Don't we want to allow other fitting methods?
• I think it would make sense to try to make psf_photometry and aperture_photometry more consistent. E.g., right now aperture_photometry gets its centers from 2 arguments xc and yx, while psf_photometry accepts a single positions argument. At the very least, these should be the same. It also might be useful to allow using aperture_photometry apertures to define the region in which a star is fit (although maybe that's more of a feature request).

Implement more robust tests for exact aperture photometry

As demonstrated in #12 and #17, some bugs in the exact algorithms only appear when the aperture has a particular alignment on the grid, i.e. when pixel vertices end up on special locations relative to the aperture, for example on the edge of the aperture. We need to implement more tests for these 'special' positions to ensure that these do not cause any issues.