astropy / specutils Goto Github PK

One of the key utilities needed for model fitting, interactive or otherwise, is selection of a subset of the data. This is not extraordinarily difficult, but I think may be the most important single feature for interaction with Models.

A select_data routine should be able to select arbitrary subsets of the data. For example, the highlighted region in this figure:

selects two separate regions that have a set of X-axis data points that are not linear (i.e., there is a step-function jump where there's empty data).

The approach I've taken has been to have separate options for selecting and deselecting data, e.g. select=True / deselect=True, so you could first select the whole region:

and then deselect the region containing the line.

These comments were taken from #51 and I suggested a restructure of the extinction code:

We should have a similar interface (which the functions have), but not the same as they do need different input parameters. Furthermore, I believe that we should give the functions more meaningful names and make them none-private. The different ccm89-like coefficients should also not be stored in the functions itself (IMHO), but maybe as a dictionary on top of the file (or externally). For the models, they could be loadable through a load_dust_model('wd01') function and give back a table. These data(coefficients, tables) might then be given as parameters to the appropriate evaluation functions.
Once this is nice and clean, we can argue if we want our users to quickly think and pick the right input parameters and evaluation function (showing this in the documentation; my choice ;-) ) or a convenience script.

I am trying to get the Spectrum1D.interpolate method working in a simple example but probably missing something:

flux = np.random.random(3000) * u.Unit('W m-2 angstrom-1 sr-1')
linear_wcs = specwcs.Spectrum1DPolynomialWCS(degree=1, unit='angstrom', c0=6000, c1=1.0)
spec1d = Spectrum1D(flux=flux, wcs=linear_wcs)

wave2 = (wave + 0.5 * u.Angstrom)[100:-100]
spec1d.interpolate(wave2)

This is failing because wcs.invert is not implemented when I run from the master branch.

I also noticed that the interpolate method is not listed when I used spec1d? in a notebook, but I can't see anything in the code that would explain this.

Hello,

I was looking for specutils.extinction.reddenning and when I followed the downloading procedure on the README file (that clones the github repo) I didn't find it. I can't see it on the Github either.
I was asked on twitter by Steven Crawford to create an issue and told that I could find what I was looking for in the legacy branch.

Cheers

This issue is here to raise the issue of echelle spectra. They are also 1D spectra in the sense that they only have one axis (wavelength), however, they have multiple orders. They should be kept in one object, as all the echelle order typically have the same metadata (GAIN, OBS-TIME etc.).

We could subclass Spectrum1D for that purpose or just allow dispersion and flux to be 2 D arrays and add a method like flatten(some options here) to merge all echelle orders.

Clearly flatten is a major undertaking that requires many different options and methods (cross-correltate features to improve wavelength solution? How to set the merge points? Rebin to common grid?), but the first step is to decide if echelle spectra need a separate object.

Is this the right place for such a request? The request was, "... spectrum analysis tasks such as standard, sensfunc, identify, and reidentify... I hope in future we will have sophisticated Python library for spectrum analysis."

c/c @SaraOgaz

As it's been done for Legendre and Chebyshev tests, is it possible to prepare a similar file for linear multispec test case from this file?
@wkerzendorf @nhmc

This comes out of some discussion in #129 relating to how the wcs and dispersion interact in the Spectrum1DRef.

Currently, Spectrum1DRef attaches a FITS WCS to the wcs attribute and then uses that in dispersion to figure out what the pixel-to-wavelength mapping is. However, this is potentially problematic because it means there's a subtle difference in how Spectrum1D.wcs and Spectrum1DRef.wcs are interpreted: the former is a "wcs" as defined in specutils, which assumes a WCS interface that's basically "call to go from pix to world". Whereas Spectrum1DRef uses a wcs that's the astropy.wcs.WCS class. Both yield consistent results for dispersion, so it doesn't affect anything that is only accessing the wcs indirectly through dispersion... But it's a bit inconsistent.

A reasonable short-term fix is to essentially move the machinery currently in Spectrum1DRef.dispersion to instead be inside an intermediate WCS adpater class that mocks up the Spectrum1D interface around an astropy.wcs.WCS object. This is not great because it would probably lead to awkward constuctions like self.wcs.wcs.wcs... But is at least slightly more internally consistent. So that's probably a good first step that @nmearl said he could try to work out.

Longer-term, we need a solution to the "WCS dillema" - that is, it may be best to work out some sort of interface for spectral WCS's that we can agree on.

Discussion thread for init function.

Question: I see that the mask attribute on Spectrum1D can be used to store a mask for the wavelength/flux arrays, but those wavelength/flux attributes currently are not returned as masked arrays. Is that a planned feature in the future, or is the mask attribute just there to hold a mask, but not to return wavelengths/fluxes as masked arrays?

Currently I have disabled the Sphinx build in .travis.yml:

   # - python: 2.7
   #   env: SETUP_CMD='build_sphinx -w

because there are warnings in the docs. This is a reminder to fix the warnings in the docs, and re-enable the Travis build. I won't have time to do it, but maybe @wkerzendorf or @shaileshahuja can do this?

thread for discussing inplace vs copy

Look at the headers of the FITS file: http://pastebin.com/v2dMfP52
This file has been taken from pyspeckit/ pyspeckit-tests project.
@keflavich How do I read this file? Can you help me identify which reader in that project can read this file?

The first two tests at https://github.com/astropy/specutils/blob/master/specutils/io/tests/test_read_fits_spec.py should fail as the resulting arrays read from the two different files are shifted by one element.
Example:
iraf wave array:
[4344.5500522758402, 4344.59028429296, 4344.6305137707004, 4344.6707407088797, 4344.7109651072797]
fits dispersion array:
[4344.5098177195632, 4344.5500522758412, 4344.5902842929554, 4344.6305137707031, 4344.6707407088788]

The (n+1)th element in fits dispersion array is the same as the nth element in the iraf wave array.

The test pass because assert_almost_equal is somehow true (maybe the value are too close)

https://github.com/astropy/specutils/blob/master/specutils/io/tests/test_read_fits_spec.py#L20
iraf wave array:
[4344.5500522758402, 4344.59028429296, 4344.6305137707004, 4344.6707407088797, 4344.7109651072797]
fits dispersion array:
[4344.5098177195632, 4344.5500522758412, 4344.5902842929554, 4344.6305137707031, 4344.6707407088788]

The (n+1)th element in fits dispersion array is the same as the nth element in the iraf wave array.
@wkerzendorf @nhmc

Currently the IRAF class do the transformations manually in (specwcs.py)[https://github.com/astropy/specutils/blob/master/specutils/wcs/specwcs.py]. However, these transformations should be done on the functions' coefficients directly so that there is no need for these extra inherited classes.

I get these failures trying to pip install specutils on travis:

    error: [Errno 2] Could not find C file specutils/cextinction.c for Cython file specutils/cextinction.pyx when building extension specutils.cextinction. Cython must be installed to build from a git checkout.: 'specutils/cextinction.c'

https://travis-ci.org/pyspeckit/pyspeckit/jobs/148715120#L679

So it looks like the pip version should have been cython'd before uploading...

Hi I have been using the specutils package without problem on my MAC. Now I want to run some code on a linux server and the same code gives me this problem:

---> 10 from specutils.extinction import extinction_ccm89 as ext_ccm89

/home/gennaro/anaconda3/envs/astroconda/lib/python3.5/site-packages/specutils/extinction.py in ()
18
19
---> 20 from specutils import cextinction
21
22 try:

ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/gennaro/anaconda3/envs/astroconda/lib/python3.5/site-packages/specutils/cextinction.cpython-35m-x86_64-linux-gnu.so)

Any suggestion/help?
Thanks,
Mario

Spectrum1D.from_fits and Spectrum1D.from_ascii should not exist - instead, specutils should be defining readers/writers via the unified I/O interface and registering them for the Spectrum1D class.

Spectrum1D should inherit from the NDIOMixin class to enable this.

@wkerzendorf - was there a reason for not doing this that I am missing?

The following bit of code works using astropy dev but fails when using astropy v1.0.3 (the current pip installed version of astropy). specutils was installed via pip in both test cases and is v0.10.

Here is the code:

 import specutils
 from astropy import units as u
 thar_spec = specutils.io.read_fits.read_fits_spectrum1d('thar.fits', dispersion_unit=u.angstrom)
 thar_spec.wavelength

And here is the error produced when run using astropy v1.0.3:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/crawford/temp/lib/python2.7/site-packages/specutils/spectrum1d.py", line 243, in __getattr__
return self.dispersion.to(self._wcs_attributes[name]['unit'], equivalencies=self.wcs.equivalencies)
File "/Users/crawford/temp/lib/python2.7/site-packages/specutils/spectrum1d.py", line 268, in dispersion
return self.wcs(self.indexer(pixel_indices))
File "/Users/crawford/temp/lib/python2.7/site-packages/specutils/wcs/specwcs.py", line 187, in __call__
self.param_sets))
File "/Users/crawford/temp/lib/python2.7/site-packages/specutils/wcs/specwcs.py", line 197, in evaluate
*coeffs) * unit
TypeError: unbound method evaluate() must be called with Spectrum1DPolynomialWCS instance as first argument (got ndarray instance instead)

I've copy of the file that is having the problem is available at:
http://pysalt.salt.ac.za/data/linelists/thar.fits

While sorting out the test matrix, I wonder about the plans for the python and numpy support?

Could you please register specutils on PyPI to avoid someone else making a Python package with that name?

See astropy/package-template#17 .

Greetings maintainers,

I finally found some spare cycles that I can re-contribute to this effort.
Unfortunately, I see that WCS has crept back in as a fundamental
piece of the Spectrum1d object (via the OneDSpectrumMixin object).

For me, putting WCS at the core of this object is a non-starter
(as discussed at length at the STScI meeting in April). I see
from the comments in spectrum_mixin.py that I'm not the only
dissenting voice on this topic.

To wit, I ask: can we strip out this 'feature' as required?
I'm fine with using a WCS when it is supplied, of course.
But the overwhelming majority of the (current) community
wants nothing to do with a WCS definition for the spectral axis.
At least not for 1D spectra.

Happy to strip the WCS out. In fact, I'll do so
in my fork and see where that goes..

Thoughts?

Hi there,
For some reason I am unable to import the extinction module.

from specutils import extinction
ImportError: cannot import name extinction

When I tried:
import extinction.py
I got:
ImportError: cannot import name cextinction

Currently, only arrays can be passed, otherwise the WCS raises an error. Need to fix this, add test cases and update documentation for this model.
Docs also need to be updated for the composite model

I'm trying to follow the documentatiion here
http://specutils.readthedocs.org/en/latest/specutils/specwcs.html

but Spectrum1DLookupWCS(wave, unit='angstrom') fails with exception

TypeError: __init__() got an unexpected keyword argument 'lookup_table_parameter'

there has been a change in the tables in the latest astropy development verison, which is causing the ascii.read function in test_read_fits_spec.py to fail. @wkerzendorf Do you know who can help me fix this?

import logging
is missing in the file specutils/specutils/io/default_loaders/tabular_fits_writer.py

In read_iraf_spec, the implementation of linear spline dispersion and cubic spline dispersion follows the standard given here.
In multispec format (specifically the WAT keywords), the first three function parameters are npieces (number of pieces), pmin and pmax. Then, the coefficients follows. For linear spline, npieces + 1 coefficients are there, and for cubic spline npieces + 3 coefficients.
To use scipy's spline evaluation function, the tck parameters needs to be derived from the WAT keywords.
tck is a tuple of (knots, coefficients, degree). The degree is easy, 1 for linear and 3 for cubic. knots and coefficients are supposed to be vectors of the same length. We can get the coefficients from the WAT keywords. But how do you get knots corresponding to these coefficients? @wkerzendorf @nhmc

@embray How do I change my models to work with the latest model framework. I am getting an "Cannot instantiate abstract class" error in all my builds

Hi all,
I'm working with millimeter-radio spectra from the IRAM 30m telescope, and my FITS spectra were generated by CLASS (part of the gildas package). When I try to load a CLASS-exported FITS file into specutils, it chokes on a SingularMatrixError. (Further details in this gist, including the FITS header in question.)

The spectrum is attached (and zip'd):
spectrum.zip

@keflavich recommended I file an issue here so here it is. (:

@shaileshahuja @wkerzendorf

There is a useful implementation of spectral coordinate transformations as outlined in the Greisen 2006 FITS document:

https://github.com/radio-astro-tools/spectral-cube/blob/master/spectral_cube/spectral_axis.py

I don't think this has been implemented in specutils yet but it probably belongs here.

It looks like Travis is now failing in master:

https://travis-ci.org/astropy/specutils/builds/117507001

This is due to new versions of packages (scipy?) in conda - since the last build in master passed 3 months ago but failed today (I restarted it to check)

We should fix this and then rebase #129

cc @nmearl

Hi,

I am currently packaging the specutils affiliated package for Debian. When running the tests (with astropy-1.0.5) with python-2.7.10, I get a number of failures which are caused by the following TypeError:

specutils/spectrum1d.py:268: in dispersion
    return self.wcs(self.indexer(pixel_indices))
specutils/wcs/specwcs.py:187: in __call__
    self.param_sets))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'specutils.wcs.specwcs.Spectrum1DPolynomialWCS'>
Name: Spectrum1DPolyno...le parameters: <astropy.utils.decorators.lazyproperty object at 0x7f76b6bad990>
pixel_indices = array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
         4.09300000e+03,   4.09400000e+03,   4.09500000e+03])
unit = Unit("km / s"), coeffs = (array([ 524.88543398]), array([-0.25258831]))

@classmethod
    def evaluate(cls, pixel_indices, unit=None, *coeffs):
        return super(Spectrum1DPolynomialWCS, cls).evaluate(pixel_indices,
>                                                           *coeffs) * unit
E       TypeError: unbound method evaluate() must be called with Spectrum1DPolynomialWCS 
                   instance as first argument (got ndarray instance instead)

This happens on test_multispec_linear, test_multispec_log_linear, test_1dspec_UVES, test_1dspec_vrad, test_linear_write1, test_linear_write2, test_multispec_linear, and test_spectrum_slicing. Also the first test test_multispec_linear has the additional output

----------------------------- Captured stderr call -----------------------------
WARNING: File may have been truncated: actual file length (895664) is smaller than the expected size (895680) [astropy.io.fits.file]
specutils/wcs/specwcs.py:75: UserWarning: Initializing a Spectrum1D WCS with units set to `None` is not recommended
  'Initializing a Spectrum1D WCS with units set to `None` is not recommended')

which may or may not be connected with this issue.

It seems specutils can't read a FITS file with an Nx2 (or 2xN) array, i.e. stacked wavelength and flux arrays. Is this intentional?

Having created a spectrum with Spectrum1D, using a lookup table as the WCS, write_fits then fails with:
'Spectrum1DLookupWCS' object has no attribute 'write_fits_header'

Thanks,
Dave

Do we plan to add features to combine (average, sum, etc.) 1D spectra?
For that, first we will need to implement function to rebin/interpolate spectrum to various dispersion points. (This will be the tricky part to make sure flux/counts etc are conserved.)
After that I believe a spectra combiner will be straight forward.

For example to cross-correlate templates with spectra and determine velocity offsets

Hi,

I am including the specutils package into AstroConda but hit a minor snag in the process. Tag v0.2 cannot be built under Python 3.5.2 due to the distutils.config.ConfigParser() bug in setup.py. However, the problem has been fixed in the master branch, so would it be possible for you to generate a v0.2.1 tag so that people can build/distribute specutils using the latest interpreter?

Thank you for your time.

Joe

Would it be possible to generate the sphinx docs and put the online.
E.g. photutils docs are here:
http://photutils.readthedocs.org/en/latest/

Right now, the class uses dispersion as the field that holds the dispersed axis and that's fine for a general class. However, for practical use it would be great to by able to also acess that as flux or frequency. If I just load a randomn HST/STIS spectrum in astropy
for preview, that wave or wavelength is what I would intuatively expect, not a generic dispersion. Now, with the astropy.units framework in place we can certainly offer acess as wavelength (or frequency) on-the-fly, independent of the original unit of the dispersion.

I was previously using specutils with astropy==1.0.dev10669 just fine.
I have recently pulled to astropy==1.0.dev11368
and am finding a number of simple specutils calls are failing.

For example:

from specutils.spectrum1d import Spectrum1D
wv = np.array( [1., 2., 3., 4])
fx = np.ones(4)
spec = Spectrum1D.from_array(wv, fx)

TypeError: init() got an unexpected keyword argument 'flags'

If I try 'fix' this error, e.g. by removing flags from init()
other problems still arise.

I fear I don't know enough about astropy development to even
suggest what the issues may be.

X

This project would benefit from a clear set of goals leading up to a release. Responses to this should include checkboxes.

• Slice methods
• X-axis (dispersion, velocity, wavelength, frequency, energy, etc.)
• As lookuptable
• As Model (e.g., polynomials)
• X-axis unit conversion methods (wrapping astropy.units.to)
• Read methods from:
• IRAF multispec fits files
• IRAF echelle fits files
• FITS files with wavelength, flux, and error (e.g., 2D arrays)
• ASCII columnar FITS files (wavelength, flux, error)

Additional items not necessary for a first release:

• Read methods from:
• SDSS-generated FITS files (these follow a unique standard)
• CLASS files (.cls, binary files)
• GBTIDL SDFITS (nontrivial: these generally represent large collections of 1D spectra)
• CASA cubes (?)

Add to this list as you see fit. Also, the list currently suffers from bad formatting errors; help would be appreciated to fix them.

intersphinx seems to have trouble working in the documentation

The specutils setup.py doesn't work for me:

$ python setup.py install --user
ERROR: AttributeError: 'module' object has no attribute 'set_build_mode' [__main__]
Traceback (most recent call last):
  File "setup.py", line 36, in <module>
    setup_helpers.set_build_mode()

Once this is fixed, would it make sense to test on travis?
It's very simple to set up, e.g. just copy
https://github.com/astropy/photutils/blob/master/.travis.yml
and activate it in the admin settings.

See example below. It must be related to how the lower limit is applied in np.interp.

>>> import astropy.units as u
>>> import numpy as np
>>> from specutils import Spectrum1D

>>> wave = np.arange(10.) * u.AA
>>> flux = np.ones_like(wa)
>>> spec = Spectrum1D.from_array(wave, flux)
>>> spec.dispersion
<Quantity [ nan,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.] Angstrom>

As requested by @mwcraig, please update specutils's entry in the link given below.

"Please take a look at the information for your affiliated package at https://github.com/mwcraig/astropy-affiliated-numfocus-blog and edit the yml file there to add a bit more information information about your package: a representative image, a link to educational materials (e.g. a notebook or tutorial or ...) and an update of your packages description, if it is out of date or missing." -- Matt Craig

Hello,

I am trying to use specutils on the online jupyter notebook from sciserver (compute.sciserver.org) and I get the following error:

ImportErrorTraceback (most recent call last)
in ()
8 from matplotlib import pyplot as plt
9 from astropy.io import fits
---> 10 from specutils.io import read_fits
11

ImportError: No module named 'specutils'

Any hints? I'm a newbie here...

Thanks

Cecilia

I just tried
x = 1.0*astropy.units.c
expecting the value to in units of the speed-of-light. This attribute was not being used and I think it would be useful to allocate it as such.