astropy / astroquery

Functions and classes to access online data resources. Maintainers: @keflavich and @bsipocz and @ceb8

Documentation | Blog | View on Github | Download Stable ZIP | Download Stable TAR

Accessing Online Astronomical Data

Astroquery is an astropy affiliated package that contains a collection of tools to access online Astronomical data. Each web service has its own sub-package. For example, to interface with the SIMBAD website, use the simbad sub-package:

>>> from astroquery.simbad import Simbad
>>> theta1c = Simbad.query_object('tet01 Ori C')
>>> theta1c.pprint()
   MAIN_ID          RA           DEC      ... COO_QUAL COO_WAVELENGTH     COO_BIBCODE
------------- ------------- ------------- ... -------- -------------- -------------------
* tet01 Ori C 05 35 16.4637 -05 23 22.848 ...        A              O 2007A&A...474..653V

Installation and Requirements

Astroquery works with Python 3.7 or later. As an astropy affiliate, astroquery requires astropy version 4.2.1 or later.

astroquery uses the requests module to communicate with the internet. BeautifulSoup and html5lib' are needed for HTML parsing for some services. The keyring module is also required for accessing services that require a login. These can all be installed using pip or anaconda. Running the tests requires curl to be installed.

The latest version of astroquery can be pip installed (note the --pre for picking up released developer versions, and -U for upgrade):

$ python -m pip install -U --pre astroquery

To install all the mandatory and optional dependencies add the [all] identifier to the pip command above (or use [docs] or [test] for the dependencies required to build the documentation or run the tests):

$ python -m pip install -U --pre astroquery[all]

To install the 'bleeding edge' version:

$ python -m pip install git+https://github.com/astropy/astroquery.git

or cloned and installed from source:

$ # If you have a github account:
$ git clone [email protected]:astropy/astroquery.git
$ # If you do not:
$ git clone https://github.com/astropy/astroquery.git
$ cd astroquery
$ python -m pip install .

Using astroquery

Importing astroquery on its own doesn't get you much: you need to import each sub-module specifically. See the documentation for a list of Available Services. The API shows the standard suite of tools common to most modules, e.g. query_object and query_region.

To report bugs and request features, please use the issue tracker. Code contributions are very welcome, though we encourage you to follow the API and contributing guidelines as much as possible.

Citing Astroquery

If you use astroquery, please cite the paper we published in The Astronomical Journal.

The BibTeX entry is available from the package itself:

import astroquery
astroquery.__citation__

In addition you may also want to refer to specific versions of the package. We create a separate Zenodo DOI for each version, they can be looked up at the following Zenodo page

Additional Links

Download Development ZIP | Download Development TAR

Maintained by Adam Ginsburg and Brigitta Sipocz ([email protected])

astroquery's Issues

Simbad multi-queries

As per https://groups.google.com/d/topic/astropy-dev/PM-AUgntB1Q/discussion.

Current best I can do is:

In [1]: import astroquery.simbad

In [2]: tables = [astroquery.simbad.QueryId(x).execute().table for x in ['m31','m51','omc1']]

Need to make astroquery.simbad.sim_queries.QueryMulti work and add usage examples in the docs.

Switch to use astropy's ``get_readable_fileobj``

We recently put a lot of effort into developing a robust I/O routine in astropy.utils.data that can read local or remote data, supports caching, and shows a progress bar for downloads, e.g.:

In [5]: with get_readable_fileobj('http://www.eso.org/public/archives/images/original/eso0932a.tif') as f:
            results = f.read()
   ...:     
Downloading http://www.eso.org/public/archives/images/original/eso0932a.tif
|==========================================|  26/ 26M (100.00%)        10s

It supports on-the-fly decompression of gzip and bzip2 too. We should probably make use of this instead of chunk_read.

Don't use mechanize in splatalogue

@vilhelmp We are now running astroquery unit tests on travis-ci for Python 3.
In splatalogue you are using mechanize which doesn't support Python 3.

Can you change this to use requests, which does support Python 3 and we decided to accept as a dependency for astroquery (see #71)?

io.votable warnings

All the warnings below from io.votable don't inspire confidence in the result and are likely to generate questions from users. Are these warnings a sign of non-compliant VO-tables from SIMBAD, or something else that can be fixed?

In [4]: tables = [astroquery.simbad.QueryId(x).execute().table for x in ['m31','m51','omc1']]
WARNING: W22: None:3:0: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring [astropy.io.votable.exceptions]
WARNING: W27: None:4:0: W27: COOSYS deprecated in VOTable 1.2 [astropy.io.votable.exceptions]
WARNING: W50: None:12:0: W50: Invalid unit string '"h:m:s"' [astropy.io.votable.exceptions]
WARNING: W50: None:15:0: W50: Invalid unit string '"d:m:s"' [astropy.io.votable.exceptions]
WARNING: W47: None:33:0: W47: Missing arraysize indicates length 1 [astropy.io.votable.exceptions]
WARNING: W47: None:36:0: W47: Missing arraysize indicates length 1 [astropy.io.votable.exceptions]
WARNING: W49: None:44:98: W49: Empty cell illegal for integer fields. [astropy.io.votable.exceptions]
WARNING: W46: None:44:121: W46: char value is too long for specified length of 1 [astropy.io.votable.exceptions]

SIMBAD python2.7 failures?

SIMBAD seems to be unstable to changes in other modules; I suspect there may be uncaught server errors causing test failures:

This fails for python-2.7 on SIMBAD:
https://travis-ci.org/astropy/astroquery/jobs/7566041
(though I restarted it...)

This one passes:
https://travis-ci.org/astropy/astroquery/jobs/7535400

The diff between those two commits shows no changes to SIMBAD:

$ git diff --name-only a9d3ab8 16a7673
astroquery/alfa/__init__.py
astroquery/alfa/core.py
astroquery/alfa/tests/__init__.py
astroquery/alfa/tests/test_alfa.py
astroquery/sdss/__init__.py
astroquery/sdss/core.py
astroquery/sdss/tests/test_sdss.py

Empty response from VizieR queries raises IndexError

When a VizieR query returns no rows, an IndexError occurs. For instance:

>>> import astropy.table
>>> import astroquery.vizier
>>> astroquery.vizier.vizquery({
...     '-source': 'USNO-B1.0',
...     '-c': astropy.table.Table([
...         astropy.table.Column([216.890730367], '_RAJ2000'),
...         astropy.table.Column([16.7969803358], '_DEJ2000')
...     ])[:1],
...     '-c.rs': '0.5'
... })
WARNING: W27: None:31:2: W27: COOSYS deprecated in VOTable 1.2 [astropy.io.votable.exceptions]
WARNING: W27: None:32:2: W27: COOSYS deprecated in VOTable 1.2 [astropy.io.votable.exceptions]
ERROR: IndexError: list index out of range [astroquery.vizier.core]
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "/Users/foo/local/lib/python2.7/site-packages/astroquery/vizier/core.py", line 91, in vizquery
    table = tableList[0]
IndexError: list index out of range

It would be nicer if the function returned an empty table, or at least raised a different exception.

Remove asciitable dependency

The following file (and maybe others) has a dependency on asciitable which should be changed to astropy.io.ascii:

https://github.com/astropy/astroquery/blob/master/astroquery/tests/test_besancon.py

Refactoring To-Do List

This is the priority list for modules that need to be refactored:

1. SIMBAD
2. VizieR
3. NED
4. IRSA
5. UKIDSS
6. Fermi
7. NRAO
8. NVAS
9. MAGPIS
10. NIST
11. SDSS
12. ALFALFA
13. SHA
14. LAMDA
15. OGLE
16. Splatalogue
17. Besancon
18. IRSA_dust

The ordering is up for discussion, but I've put the most general services & the services in greatest need of refactoring (i.e., NED) at the top, while recently-made service tools that nearly fit the API are towards the bottom in the hope that they can be slightly modified rather than majorly refactored. The Besancon tool is at the bottom since it's probably a task I need to do myself and it's at the bottom of my stack right now.

multi-source Galactic extinction querier

This is in part motivated by the discussion in astropy/astropy#733 . There was (and is) a discussion there about whether or not Galactic extinction should be included in the astropy core vs. astroquery.

Regardless of what goes in the core, it is clear that quite a few different online queryable source exist for all-sky or large-field dust maps, either in E(B-V) form or as dust columns or the like. Some possible examples include

• The online queryable for the SFD dustmaps at http://irsa.ipac.caltech.edu/applications/DUST/docs/dustProgramInterface.html
• The http://ogle.astrouw.edu.pl/cgi-ogle/getext.py interface to get E(B-V) near the galactic bulge
• Planck's maps - I don't think there's a dust map yet, but it's bound to show up at some point (at http://irsa.ipac.caltech.edu/Missions/planck.html ?)
• The LAB Survey neutral HI mentioned by @zblz in astropy/astropy#733 -http://vizier.cfa.harvard.edu/viz-bin/Cat?VIII/76

So this issue calls for creating an interface that might allow various sources like this to be queried and return the same/similar type of object. That is, it would be really nice to be able to just do something like:

ebmv1 = query_ebmv(coordinates, 'SFD')
ebmv2 = query_ebmv(coordinates, 'ogle-vvv')

and have ebmv1 and ebmv2 be exactly comparable.

(of course it should match the general API plan for astroquery, but the result object should be something consistent across the disperate tools.)

Use 'requests' instead of urllib/urllib2

As discussed in #70, requests provides a more uniform, nice, and reliable interface for the modern interwebs. One goal when refactoring should be to use requests in place of the built-ins.

As @astrofrog notes, though, we will continue to use get_readable_fileobj because of its caching and progressbar capabilities.

TODO: adapt these packages that currently import urllib and /or urllib2:

• besancon
• fermi
• irsa
• irsadust
• magpis
• ned
• nist
• nrao
• simbad
• splatalogue
• ukidss
• utils

Add NRAO Data Archive query tool

URL: archive.nrao.edu/archive/advquery.jsp

This is a very flexible tool for accessing NRAO data and will require Delayed Queries for the data retrieval component. However, for just getting tables of observations, it should be straightforward to implement.

(note that the current NRAO archive is for Lourant Sjouwerman's VLA Archive Survey project, the NVAS: http://www.aoc.nrao.edu/~vlbacald/, not the NRAO archive)

Nice download progress report in IPython notebook?

I'm writing some astropy tutorials as IPython notebooks:
http://gammapy.github.com/astropy-tutorials/

One issue I found with astroquery (see example here) is that the progress report doesn't show up at all (in Safari at least while where I'm running my notebook), but then when calling nbconvert on it, there's too much output:

Downloaded    0.00098 Mb
Downloaded      0.002 Mb
Downloaded     0.0029 Mb
Downloaded     0.0039 Mb
Downloaded     0.0049 Mb
Downloaded     0.0059 Mb
Downloaded     0.0068 Mb
...

It would be nice if there was a way to make the progress visible while running the notebook and at the end have one line in the output cell.

Spitzer / WISE data query

IRSA has a different interface to the Spitzer and WISE data than other surveys that have VO support (which in principal should make this easier to implement)

http://sha.ipac.caltech.edu/applications/Spitzer/SHA/help/doc/api.html
http://irsa.ipac.caltech.edu/ibe/

Add commonly-used features to utils

Commonly needed features, especially download features, should be implemented in utils.

Examples:

get_fitsfile_from_url
get_votable_from_url
get_file_from_url

With this implemented, we can use astropy.utils.data.get_readable_fileobj with only that single reference to urllib, which will allow us to use requests nearly everywhere but still use the nice caching features built into get_readable_fileobj.

Add resource overview table to docs

Would it be possible to add an overview table of available resources to the docs?

I mean a link to the resource and a one line description what it is.

NED tables are parsed without useful headers

In the docs, the main example shows columns parsed as main_col1, etc., which is rather useless as it doesn't give any information about what that column is supposed to contain.

This seems to be a problem with grabbing FIELD_ID instead of FIELD_NAME from the votable. I think this could be solved by using votable to read the data, then converting to an astropy Table, as is done in Vizier. @jdnc, could you have a look?

Write "overview" documentation

Astroquery now has a lot of functionality, but there's no overview of its capabilities or how it's structured. So it would be nice to have an overview page in the docs.

It would also be nice to have a "developer overview" where you say what the general coding scheme for all the queriers is.

Add ADS query tools

Useful examples would include:

• Getting a list of co-authors (e.g., www.astrobetter.com/ads-how-to-find-author-names-and-affiliations)
• Get all papers referencing an object
• From any paper query, retrieving bibtex-formatted entries for all papers
• Download the PDF given a paper entry [this would probably be useful for something like ADSbibdesk]
• Get all sources listed in a paper as SIMBAD-compatible names

Implement interface to IRSA dust extinction service

It would be great to implement an interface to the IRSA dust extinction service. The API is described here:

http://irsa.ipac.caltech.edu/applications/DUST/

ned.query_ned_basic_posn result table columns not set

The docstring of ned.query_ned_basic_posn says this:

Definition: ned.query_ned_basic_posn(objname='M31', root_url='http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch')
Docstring:
Retrieve best available position data from NED for a particular target
    Equivalent to query_ned_by_objname(objname,tid=2)

keywords:
    objname - astronomical object to search for

Returns NED_PositionDataTable with the following information:

    -----------------------------------------------------------
    |                    Name |       Unit |    Type | Format |
    -----------------------------------------------------------
    |                 pos_ref |       None |    |S19 |    19s |
    |      pos_ra_equ_B1950_d |    degrees | float64 | 25.17e |
    |     pos_dec_equ_B1950_d |    degrees | float64 | 25.17e |
    |      pos_ra_equ_B1950_s |       None |    |S14 |    14s |
    |     pos_dec_equ_B1950_s |       None |    |S14 |    14s |
    |  maj_axis_unc_equ_B1950 | arcseconds | float64 | 25.17e |
...

But actually the column names are not set to meaningful names as advertised in the docstring:

In [37]: ned.query_ned_basic_posn('M32').names[:5]
Out[37]: ('posn_col1', 'posn_col2', 'posn_col3', 'posn_col4', 'posn_col5')

Also, is this the expected error when a query is made for an unknown name (I would have expected a more explicit error)?

In [39]: ned.query_ned_basic_posn('M1234')          
ERROR: Exception: There are no tables present in this file [atpy.votable]
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-39-3cc388c541ef> in <module>()
----> 1 ned.query_ned_basic_posn('M1234')

/Users/deil/Library/Python/2.7/lib/python/site-packages/astroquery/ned/nedpy.pyc in query_ned_basic_posn(objname, root_url)
    356     print >>tf,R
    357     tf.file.flush()
--> 358     t = atpy.Table(tf.name,type='vo',verbose=False)
    359 
    360     # Return atpy table

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in __init__(self, *args, **kwargs)
    166 
    167         if len(args) + len(kwargs) > 0:
--> 168             self.read(*args, **kwargs)
    169 
    170         return

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in read(self, *args, **kwargs)
    212 
    213             if table_type in registry._readers:
--> 214                 registry._readers[table_type](self, *args, **kwargs)
    215             else:
    216                 raise Exception("Unknown table type: " + table_type)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_download_to_file(read, table, filename, *args, **kwargs)
     36 
     37     # Otherwise just proceed as usual
---> 38     return read(table, filename, *args, **kwargs)
     39 
     40 

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_decompress_to_fileobj(read, table, filename, *args, **kwargs)
     55             return read(table, bz2.BZ2File(filename), *args, **kwargs)
     56         else:
---> 57             return read(table, filename, *args, **kwargs)
     58 
     59     return read(table, filename, *args, **kwargs)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_fileobj_to_file(read, table, filename, *args, **kwargs)
     76         filename = output.name
     77 
---> 78     return read(table, filename, *args, **kwargs)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/votable.pyc in read(self, filename, pedantic, tid, verbose)
     72             tid = 0
     73         elif len(tables) == 0:
---> 74             raise Exception("There are no tables present in this file")
     75         else:
     76             raise TableException(tables, 'tid')

Exception: There are no tables present in this file

Bug: get_query_payload should be rejected by synchronous version

If you try Service.query_region(...., get_query_payload=True), the query_region_async will return the payload, but then query_region will try to parse that.

Options to deal with this:

1. Add a kwarg-parsing mechanism in async_to_sync, e.g.:

if kwargs.pop('get_query_payload'):
    return result
else:
    return parse(result)

1. Perform some kind of caller-checking in the _async function
2. Have the _async functions return an object that will be parsed properly into a payload

I think I prefer solution 1.

Error when querying NED for "crab"

This is the error I get:

In [56]: ned.query_ned_by_objname('crab')
ERROR: E21: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpVu4BSs:690:27: E21: Data has fewer columns (88) than are defined in the header (91) [astropy.io.votable.exceptions]
---------------------------------------------------------------------------
E21                                       Traceback (most recent call last)
<ipython-input-56-89e83481cd10> in <module>()
----> 1 ned.query_ned_by_objname('crab')

/Users/deil/Library/Python/2.7/lib/python/site-packages/astroquery/ned/nedpy.pyc in query_ned_by_objname(objname, root_url, tid)
     76     print >>tf,R
     77     tf.file.flush()
---> 78     t = atpy.Table(tf.name,type='vo',tid=tid,verbose=False)
     79 
     80     # Return atpy table

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in __init__(self, *args, **kwargs)
    166 
    167         if len(args) + len(kwargs) > 0:
--> 168             self.read(*args, **kwargs)
    169 
    170         return

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in read(self, *args, **kwargs)
    212 
    213             if table_type in registry._readers:
--> 214                 registry._readers[table_type](self, *args, **kwargs)
    215             else:
    216                 raise Exception("Unknown table type: " + table_type)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_download_to_file(read, table, filename, *args, **kwargs)
     36 
     37     # Otherwise just proceed as usual
---> 38     return read(table, filename, *args, **kwargs)
     39 
     40 

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_decompress_to_fileobj(read, table, filename, *args, **kwargs)
     55             return read(table, bz2.BZ2File(filename), *args, **kwargs)
     56         else:
---> 57             return read(table, filename, *args, **kwargs)
     58 
     59     return read(table, filename, *args, **kwargs)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_fileobj_to_file(read, table, filename, *args, **kwargs)
     76         filename = output.name
     77 
---> 78     return read(table, filename, *args, **kwargs)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/votable.pyc in read(self, filename, pedantic, tid, verbose)
     76             raise TableException(tables, 'tid')
     77 
---> 78     votable = parse(filename, pedantic=pedantic)
     79     for id, table in enumerate(votable.iter_tables()):
     80         if id==tid:

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/table.pyc in parse(source, columns, invalid, pedantic, chunk_size, table_number, table_id, filename, _debug_python_based_parser)
    113         _debug_python_based_parser=_debug_python_based_parser) as iterator:
    114         return tree.VOTableFile(
--> 115             config=config, pos=(1, 1)).parse(iterator, config)
    116 
    117 

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in parse(self, iterator, config)
   3083             if start:
   3084                 tag_mapping.get(tag, self._add_unknown_tag)(
-> 3085                     iterator, tag, data, config, pos)
   3086             elif tag == 'DESCRIPTION':
   3087                 if self.description is not None:

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in _add_resource(self, iterator, tag, data, config, pos)
   3014         resource = Resource(config=config, pos=pos, **data)
   3015         self.resources.append(resource)
-> 3016         resource.parse(self, iterator, config)
   3017 
   3018     def _add_coosys(self, iterator, tag, data, config, pos):

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in parse(self, votable, iterator, config)
   2870             if start:
   2871                 tag_mapping.get(tag, self._add_unknown_tag)(
-> 2872                     iterator, tag, data, config, pos)
   2873             elif tag == 'DESCRIPTION':
   2874                 if self.description is not None:

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in _add_table(self, iterator, tag, data, config, pos)
   2827         table = Table(self._votable, config=config, pos=pos, **data)
   2828         self.tables.append(table)
-> 2829         table.parse(iterator, config)
   2830 
   2831     def _add_info(self, iterator, tag, data, config, pos):

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in parse(self, iterator, config)
   2211                             'TABLEDATA', data.iterkeys(), config, pos)
   2212                         self.array = self._parse_tabledata(
-> 2213                             iterator, colnumbers, fields, config)
   2214                         break
   2215                     elif tag == 'BINARY':

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in _parse_tabledata(self, iterator, colnumbers, fields, config)
   2332 
   2333                 if i < len(fields):
-> 2334                     vo_raise(E21, (i, len(fields)), config, pos)
   2335 
   2336                 array_chunk.append(tuple(row))

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/exceptions.pyc in vo_raise(exception_class, args, config, pos)
     88     Raise an exception, with proper position information if available.
     89     """
---> 90     raise exception_class(args, config, pos)
     91 
     92 

E21: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpVu4BSs:690:27: E21: Data has fewer columns (88) than are defined in the header (91)

simbad unit tests fail

With #73 the simbad unit tests now are executed and with astropy master there's the error:

self = <astroquery.simbad.result.SimbadResult object at 0x105387950>

    @property
    def table(self):
        if self.__file is None:
            self.__file = tempfile.NamedTemporaryFile()
            self.__file.write(self.data.encode('utf-8'))
            self.__file.flush()
            array = votable.parse(self.__file,
                            pedantic=self.__pedantic).get_first_table().array
>           self.__table = array.to_table()
E           AttributeError: 'MaskedArray' object has no attribute 'to_table'

astroquery/simbad/result.py:113: AttributeError

Full log here:
https://gist.github.com/cdeil/5386471

Cannot install default profile

I installed the latest astroquery master with python setup.py install to an astropy-rc1 virtualenv. I then got a warning upon import:

In [2]: import astroquery.simbad
WARNING: ConfigurationDefaultMissingWarning: Requested default configuration file 
/Users/aldcroft/vpy/astropy-rc1/lib/python2.7/site-packages/astroquery-0.0.dev226-
py2.7.egg/astroquery/astroquery.cfg is not a file. Cannot install default profile. If you are 
importing from source, this is expected. [astroquery]

I was not importing from source.

Python 3 compatibility

Already begun in #70, but there are other python3 incompatibilities that need to be fixed; they will probably be fixed if we do #71.

Update server URLs on a regular basis & include API versioning

As discussed in the June 7 2013 telecon, it is necessary to update URLs of the query services if they change. There should therefore be a routine to download the latest server URLs on a regular basis and check that they haven't changed in order to raise exceptions about incompatible APIs before any queries are run.

The proposal is to use a JSON object, probably hosted at astropy.org/astroquery/services.json.

We will also store our own local API versions for the remote service, and try to note whenever an API change occurs that will break our code. Therefore, the astroquery services should do an API version consistency check on import.

Create a frontpage (not just frontpage docs)

Similar to
www.astropy.org/montage-wrapper

Disable remote tests for all non-refactored modules

Remote tests rely on the remote server being operational and therefore render travis unreliable. Even before refactoring the module, it's important to refactor the tests such that a remote server failure will not break the build.

NED docstrings cause travis failures

The NED docstrings don't work, largely because of tables that look like this:

---------------
|col1|col2|col3|
.....
---------------

We need to find a way to get those tables to play nice with numpydoc.

Converting from a `io.votable.tree.Table` to a `astropy.table.Table`

I'm just having a first read-through of astroquery and wanted to note something. There are a number of places where io.votable.parse is used to read a VOTable and then it is converted to astropy.table.Table. There now exists a much better way to do that that will preserve all of the metadata as well as the mask. See astropy.io.votable.tree.Table.to_table.

Unify how ROW_LIMIT works for all services

To set the SIMBAD row limit one has to do

Simbad.ROW_LIMIT = 15

but for VIZIER one has to do

Vizier.ROW_LIMIT.set('999999999')

Using the SIMBAD method results in this error; not immediately, but on the next query.

Is it possible to make this uniform way for all services?
Maybe Vizier.ROW_LIMIT could be a property that sets the ConfigurationItem so that Vizier.ROW_LIMIT = 15 works?

Error reproducing the Vizier Getting Started example

Hi, I get the following error

ERROR: AttributeError: 'module' object has no attribute 'get_data_fileobj'  [astropy.io.vo.ucd]

when trying to reproduce the example here
http://astroquery.readthedocs.org/en/latest/astroquery/vizier.html#functions

Is this a problem just on my machine, my installation? How can I make it work?
Thanks!

astroquery.__version__
Out[111]: '0.0.dev189'
astropy.__version__
Out[112]: '0.2.dev2710'

Here is the example that produces the error and the complete error:

from astroquery import vizier
query = {}
query["-source"] = "VII/258/vv10"
query["-out"] = ["Name", "Sp", "Vmag"]
query["Vmag"] = "5.0..11.0"
table1 = vizier.vizquery(query)

WARNING: W27: None:29:2: W27: COOSYS deprecated in VOTable 1.2 [astropy.io.vo.exceptions]
ERROR: AttributeError: 'module' object has no attribute 'get_data_fileobj' [astropy.io.vo.ucd]
Traceback (most recent call last):
  File "astroquerytest.py", line 6, in <module>
    table1 = vizier.vizquery(query)
  File "/usr/local/lib/python2.7/dist-packages/astroquery/vizier/vizier.py", line 77, in vizquery
    voTable = parse(cStringIO.StringIO(resp.read()), pedantic=False)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/table.py", line 108, in parse
    config=config, pos=(1, 1)).parse(iterator, config)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 2881, in parse
    iterator, tag, data, config, pos)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 2812, in _add_resource
    resource.parse(self, iterator, config)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 2666, in parse
    iterator, tag, data, config, pos)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 2623, in _add_table
    table.parse(iterator, config)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 2010, in parse
    iterator, tag, data, config, pos)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 1939, in _add_field
    field = Field(self._votable, config=config, pos=pos, **data)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 1001, in __init__
    self.ucd        = ucd
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 321, in ucd
    check_ucd(ucd, self._config, self._pos)
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/tree.py", line 211, in check_ucd
    has_colon=config.get('version_1_2_or_later', False))
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/ucd.py", line 103, in parse_ucd
    _ucd_singleton = UCDWords()
  File "/usr/local/lib/python2.7/dist-packages/astropy/io/vo/ucd.py", line 29, in __init__
    with config.get_data_fileobj("data/ucd1p-words.txt") as fd:
AttributeError: 'module' object has no attribute 'get_data_fileobj'

Open Exoplanet Catalogue

I'm playing with the idea of adding support for the Open Exoplanet Catalogue. Do you think this would be a feature worth having?

vizier unit test fails

With #73 thetest_simple vizier unit test fails:

IndexError: list index out of range
astroquery/vizier/core.py:91: IndexError

Full log here:
https://travis-ci.org/astropy/astroquery/jobs/6345626#L956

Fermi LAT data access

I would like to add access to the Fermi LAT data to astroquery.

There's a bunch of different data products, most are FITS files with tables, which I guess is easy:
http://fermi.gsfc.nasa.gov/ssc/data/access/

However the main data server is via this web form where the user enters some parameters:
http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

With acceptable parameters one gets a page that contains info on the status of the data preparation on their data server and after a minute or so the download links. Here's an example:
http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/QueryResults.cgi?id=L1301071121485C26556A03

@keflavich Is it possible to interface this data server from astroquery?
(I'll also ask their helpdesk as well if this is allowed and if there is a documented web API I don't know about)

Update "Contributing" with an outline of how to create a new module

The new module format developed by @jdnc and I does not yet have an overview document. We need to create one of these and add it/link it to Contributing so that new module developers have an easy time making a fresh module.

Module `coords` imported by NRAO subpackage

The current NRAO subpackage tries to import a coords module that isn't shipped here and that isn't present in the required section of setup.py either. Where is it supposed to be ?

IRSA queries do not work

tbl = astroquery.irsa.query_gator_box('pt_src_cat','83.808 -5.391',300)

Apparently IRSA returns invalid VO tables.

Ned get_table doesn't work for table='references'.

The astroquery.Ned.get_table fails for table='references' keyword argument.
For instance:

In [1]: from astroquery.ned import Ned
In [3]: Ned.get_table("m1", table='references')
/tmp/tmpQy0_JB:37:4: ValueError: invalid literal for int() with base 10: '1.' (in row 0, col 'ref_col1')
WARNING: Error in parsing Ned result. Returning raw result instead.    [astroquery.ned.core]

The reason is that the VOTable returned by the service cannot be parsed correctly. For details see astropy/astropy#1266

LAMDA database for astroquery

Hi all,

I'm interested in trying to add functionality in astroquery for the Leiden Atomic and Molecular Database (http://home.strw.leidenuniv.nl/~moldata/) which has a variety of relevant astrophysics data (statistical weights, collisional rate coefficients, etc.).

I'm not very familiar with git but have been able to get by on my own projects with a "single developer workflow" of commit/checkout/push. I've forked the main repository and made a project branch called "lamda". Since only things within the new sub-module should change, I expect the branch can be merged in one-go when it's shippable. @keflavich has shown me files from the despotic project to read LAMDA URL's and I'm so-far using the NIST sub-module as a rough template.

A rough outline of features I will try to implement for the sub-module lamda:

• List available atoms and molecules for query -> lamda.print_mols()
• Parse LAMDA files and return appropriate items (energy levels, radiative transitions, collisional rates)
• Have function names that match the API -> lamda.query
• Return the appropriate astropy.table.Table instance

I know to write doc-strings and handle exceptions, and will try to follow the current style of astroquery modules. Is this issue thread the best place to ask questions or is there an email list that would be better?

-- Brian

File List Downloader - add examples, docs

File list downloader needs documentation and reliable examples (see #69)

Add interface to CDS's xmatch API

Just released, with python example:

http://cdsxmatch.u-strasbg.fr/xmatch/doc/xmatch-API-usage-examples.html

IRSA bug - probably VO bug?

>>> astroquery.irsa.query_gator_cone('irasgal','w51',30)

WARNING: W21: /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-/tmpPZtxCq:3:0: W21: vo.table is designed for VOTable version 1.1 and 1.2, but this file is 1.0 [astropy.io.vo.tree] WARNING: W42: /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-/tmpPZtxCq:3:0: W42: No XML namespace specified [astropy.io.vo.tree] ERROR: TypeError: __init__() keywords must be strings [astropy.io.vo.tree] Traceback (most recent call last): File "<ipython-input-73-6b69229ec10b>", line 1, in <module> astroquery.irsa.query_gator_cone('irasgal','w51',30) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astroquery-0.0.dev90-py2.6.egg/astroquery/irsa/irsa.py", line 167, in query_gator_cone return _query_gator(options) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astroquery-0.0.dev90-py2.6.egg/astroquery/irsa/irsa.py", line 349, in _query_gator array = parse(output.name, pedantic=False).get_first_table().array File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astropy-0.2.dev1466-py2.6-macosx-10.6-i386.egg/astropy/io/vo/table.py", line 108, in parse config=config, pos=(1, 1)).parse(iterator, config) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py", line 34, in __exit__ self.gen.throw(type, value, traceback) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astropy-0.2.dev1466-py2.6-macosx-10.6-i386.egg/astropy/utils/xml/iterparser.py", line 213, in get_xml_iterator yield iter(context) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astropy-0.2.dev1466-py2.6-macosx-10.6-i386.egg/astropy/io/vo/table.py", line 108, in parse config=config, pos=(1, 1)).parse(iterator, config) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astropy-0.2.dev1466-py2.6-macosx-10.6-i386.egg/astropy/io/vo/tree.py", line 2881, in parse iterator, tag, data, config, pos) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/astropy-0.2.dev1466-py2.6-macosx-10.6-i386.egg/astropy/io/vo/tree.py", line 2815, in _add_coosys coosys = CooSys(config=config, pos=pos, **data) TypeError: __init__() keywords must be strings

Issue with UKIDSS catalog query

The following query returns no results:

In [3]: from astroquery import ukidss

In [4]: Q = ukidss.UKIDSSQuery()

In [5]: Q.get_catalog_gal(10.2,-0.44,radius=0.43)
Loading page...
Downloaded    7.7e-05 Mb
Out[5]: []

but using the web interface returns many records:

http://surveys.roe.ac.uk:8080/wsa/WSASQL?database=UKIDSSDR7PLUS&programmeID=102&from=source&formaction=region&ra=10.2&dec=-0.44&sys=G&name=&radius=0.43&xSize=&ySize=&boxAlignment=RADec&emailAddress=&format=HTML&compress=GZIP&rows=30&select=default&where=

This was originally discovered by @eztean.

@keflavich - feel free to look into it if you have time. I'll try looking into it too.

python 3 incompatibility: htmllib vs htmlparser

Based on a failed Travis run: https://travis-ci.org/astropy/astroquery/jobs/3844490. htmllib has been replaced by HTMLParser
http://www.python.org/dev/peps/pep-3108/

Use astropy cache/data infrastructure throughout

This needs some thought about which things should be cached, but in general the astropy data downloading framework should be used

UKDSS query crashes

If I try and run:

from astroquery import ukidss
u = ukidss.UKIDSSQuery()
u.get_images_radius(281.9027, -1.9425, 1.0, filter='all', directory='raw', save=True, n_concurrent=8)

I get:

Loading page...
ERROR: AttributeError: 'module' object has no attribute 'chunk_read' [astroquery.ukidss.ukidss]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/tom/W43/UKIDSS/<ipython-input-4-44960c0f46b3> in <module>()
----> 1 u.get_images_radius(281.9027, -1.9425, 1.0, filter='all', directory='raw', save=True, n_concurrent=8)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astroquery-0.0.dev87-py2.7.egg/astroquery/ukidss/ukidss.pyc in get_images_radius(self, ra, dec, radius, filter, frametype, directory, n_concurrent, save, verbose, overwrite)
    307         if verbose:
    308             print "Loading page..."
--> 309             results = progressbar.chunk_read(page, report_hook=progressbar.chunk_report)
    310         else:
    311             results = page.read()

AttributeError: 'module' object has no attribute 'chunk_read'

Test errors with astropy master

I get plenty of astroquery test errors with the latest astropy master (0.3.dev3759):
https://gist.github.com/cdeil/5377244

Refactor / Monkeypatch most commonly failing Travis tests

This is a to-do list that needs to be built up and added to any time a Travis test fails because of a server communication failure.

• SIMBAD
• Vizier

Add docs for ALFA service

@mirochaj I see you've added the ALFA service in #86.

Can you please add an docs/astroquery/alfa.rst file and an entry for lambda in docs/astroquery/index.rst so that it will show up in the astroquery docs?

Question: should it be called astroquery/alfa (as it is now) or rather astroquery/alfalfa?
(I'm just asking ... I've never heard of it before and have no opinion one way or the other. Just don't mix ALFA and ALFALFA to avoid confusion.)

Thanks!