Error when querying NED for "crab" about astroquery

[cdeil]

Here's another error I ran into... should I keep reporting these errors? ... make a separate issue for each?

In [63]: t = ned.query_ned_basic_posn("PG 1553+11")
ERROR: ValueError: 21:2: mismatched tag [astropy.io.votable.tree]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-63-f549bd83b32e> in <module>()
----> 1 t = ned.query_ned_basic_posn("PG 1553+11")

/Users/deil/Library/Python/2.7/lib/python/site-packages/astroquery/ned/nedpy.pyc in query_ned_basic_posn(objname, root_url)
    356     print >>tf,R
    357     tf.file.flush()
--> 358     t = atpy.Table(tf.name,type='vo',verbose=False)
    359 
    360     # Return atpy table

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in __init__(self, *args, **kwargs)
    166 
    167         if len(args) + len(kwargs) > 0:
--> 168             self.read(*args, **kwargs)
    169 
    170         return

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/basetable.pyc in read(self, *args, **kwargs)
    212 
    213             if table_type in registry._readers:
--> 214                 registry._readers[table_type](self, *args, **kwargs)
    215             else:
    216                 raise Exception("Unknown table type: " + table_type)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_download_to_file(read, table, filename, *args, **kwargs)
     36 
     37     # Otherwise just proceed as usual
---> 38     return read(table, filename, *args, **kwargs)
     39 
     40 

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_decompress_to_fileobj(read, table, filename, *args, **kwargs)
     55             return read(table, bz2.BZ2File(filename), *args, **kwargs)
     56         else:
---> 57             return read(table, filename, *args, **kwargs)
     58 
     59     return read(table, filename, *args, **kwargs)

<string> in read(self, filename, pedantic, tid, verbose)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/decorators.pyc in _auto_fileobj_to_file(read, table, filename, *args, **kwargs)
     76         filename = output.name
     77 
---> 78     return read(table, filename, *args, **kwargs)

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/votable.pyc in read(self, filename, pedantic, tid, verbose)
     68     # If no table is requested, check that there is only one table
     69     if tid==-1:
---> 70         tables = _list_tables(filename, pedantic=pedantic)
     71         if len(tables) == 1:
     72             tid = 0

/Users/deil/Library/Python/2.7/lib/python/site-packages/atpy/votable.pyc in _list_tables(filename, pedantic)
     32 
     33 def _list_tables(filename, pedantic=False):
---> 34     votable = parse(filename, pedantic=pedantic)
     35     tables = {}
     36     for i, table in enumerate(votable.iter_tables()):

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/table.pyc in parse(source, columns, invalid, pedantic, chunk_size, table_number, table_id, filename, _debug_python_based_parser)
    113         _debug_python_based_parser=_debug_python_based_parser) as iterator:
    114         return tree.VOTableFile(
--> 115             config=config, pos=(1, 1)).parse(iterator, config)
    116 
    117 

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in parse(self, iterator, config)
   3083             if start:
   3084                 tag_mapping.get(tag, self._add_unknown_tag)(
-> 3085                     iterator, tag, data, config, pos)
   3086             elif tag == 'DESCRIPTION':
   3087                 if self.description is not None:

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in _add_resource(self, iterator, tag, data, config, pos)
   3014         resource = Resource(config=config, pos=pos, **data)
   3015         self.resources.append(resource)
-> 3016         resource.parse(self, iterator, config)
   3017 
   3018     def _add_coosys(self, iterator, tag, data, config, pos):

/Users/deil/Library/Python/2.7/lib/python/site-packages/astropy/io/votable/tree.pyc in parse(self, votable, iterator, config)
   2867             }
   2868 
-> 2869         for start, tag, data, pos in iterator:
   2870             if start:
   2871                 tag_mapping.get(tag, self._add_unknown_tag)(

ValueError: 21:2: mismatched tag

In [64]: t
Out[64]: <Table name='NED_MainTable' rows=0 fields=0>

[keflavich]

Yeah, if you continue to find failed queries on NED, maybe keep them going in this issue unless we find that they're truly unrelated.

[willettk]

Yes - please keep reporting these errors.

This is odd - it's not related to the NED name identifier, since it does return a VOtable associated with the given object. It also seems to vary by query: for example, this works fine for me:

In [308]: nedpy.query_ned_basic_posn("Crab")
Out[308]: <Table name='NED_PositionDataTable' rows=1 fields=35>

but this doesn't:

In [307]: nedpy.query_ned_by_objname("Crab")
ERROR: E21 [vo.voexceptions]
---------------------------------------------------------------------------
E21                                       Traceback (most recent call last)
/Users/willettk/python/astroquery/astroquery/ned/<ipython-input-307-94460cf88c00> in <module>()
----> 1 nedpy.query_ned_by_objname("Crab")

etc.

This looks like a problem with the formatting on the NED end - the string it retrieves looks like a readable XML file. The mismatch is between the number of columns and the header - anyone else know where that's defined?

[astrofrog]

Does the XML file validate with the volint command installed by Astropy?

[willettk]

Hi Thomas,

That's a good suggestion. I can't find volint in my current version of
astropy (0.1), however - which module is it called by?

• thanks,
  Kyle

On 22 January 2013 15:35, Thomas Robitaille [email protected]:

Does the XML file validate with the volint command installed by Astropy?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/46#issuecomment-12567904.

[astrofrog]

Hmm, it might have been added later. It's a command-line script that gets installed at the same time as Astropy. However, it may not be in your path by default. If you try installing the latest developer version of Astropy (or 0.1) you can check the last few lines which should say something like:

Installing volint script to /Users/tom/Library/Python/3.2/bin

That will tell you where the script is installed, and you can make sure this is in your $PATH.

[willettk]

I found the problem - the NED files are missing cells when the object doesn't have a redshift (so queries for 'crab' didn't work, but queries for 'M33' did). It's been fixed with my latest pull request.

[willettk]

volint didn't end up being the answer for the moment (see below). I did
find the answer to Christoph's question: for objects in NED that are not
extragalactic (ie, no redshift), there's an error in the tables NED
provide. There's a table containing derived cosmological parameters, which
should be blank for these objects. The problem is that NED provides the
wrong number of blank entries (88 instead of 91). I added a check to
ned.query_ned_by_objname that fixes this. I sent an email to NED, so
hopefully it will be fixed at the source soon.

• cheers,
  Kyle

When I try volint on the xml files that are downloaded from NED, I get
complaints that the encoding isn't correct.

dhcp-239Desktop$ volint nph-objsearch.xml
Validation report for nph-objsearch.xml

ERROR: LookupError: unknown encoding: [astropy.utils.xml.iterparser]
Traceback (most recent call last):
File "/Library/Frameworks/EPD64.framework/Versions/Current/bin/volint",
line 5, in
pkg_resources.run_script('astropy==0.1', 'volint')
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py",
line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py",
line 1239, in run_script
execfile(script_filename, namespace, namespace)
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/astropy-0.1-py2.7-macosx-10.5-x86_64.egg/EGG-INFO/scripts/volint",
line 5, in
astropy.io.vo.volint.main()
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/astropy-0.1-py2.7-macosx-10.5-x86_64.egg/astropy/io/vo/volint.py",
line 19, in main
table.validate(args.filename[0])
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/astropy-0.1-py2.7-macosx-10.5-x86_64.egg/astropy/io/vo/table.py",
line 185, in validate
xml_lines = iterparser.xml_readlines(filename)
File
"/Library/Frameworks/EPD64.framework/Versions/7.2/lib/python2.7/site-packages/astropy-0.1-py2.7-macosx-10.5-x86_64.egg/astropy/utils/xml/iterparser.py",
line 251, in xml_readlines
with io.open(source, 'rt', encoding=encoding) as input:
LookupError: unknown encoding:

On 25 January 2013 17:36, Thomas Robitaille [email protected]:

Hmm, it might have been added later. It's a command-line script that gets
installed at the same time as Astropy. However, it may not be in your path
by default. If you try installing the latest developer version of Astropy
(or 0.1) you can check the last few lines which should say something like:

Installing volint script to /Users/tom/Library/Python/3.2/bin

That will tell you where the script is installed, and you can make sure
this is in your $PATH.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/46#issuecomment-12726190.

[cdeil]

This is what I get now. Looks OK?

In [5]: ned.query_ned_by_objname('crab')
WARNING: W42: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:2:0: W42: No XML namespace specified [astropy.io.votable.exceptions]
WARNING: W22: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:3:0: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring [astropy.io.votable.exceptions]
WARNING: W04: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:18:0: W04: content-type 'char' must be a valid MIME content type [astropy.io.votable.exceptions]
WARNING: W04: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:30:0: W04: content-type 'char' must be a valid MIME content type [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:32:0: W50: Invalid unit string 'degrees' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:35:0: W50: Invalid unit string 'degrees' [astropy.io.votable.exceptions]
WARNING: W04: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:40:0: W04: content-type 'char' must be a valid MIME content type [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:42:0: W50: Invalid unit string 'km/sec' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:262:0: W50: Invalid unit string 'degrees' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:265:0: W50: Invalid unit string 'degrees' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:274:0: W50: Invalid unit string 'arcseconds' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:277:0: W50: Invalid unit string 'arcseconds' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:280:0: W50: Invalid unit string 'arcseconds' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:283:0: W50: Invalid unit string 'degrees' [astropy.io.votable.exceptions]
WARNING: W50: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:286:0: W50: Invalid unit string 'degrees' (suppressing further warnings of this type...) [astropy.io.votable.exceptions]
WARNING: W06: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:652:0: W06: Invalid UCD 'vol.moving;': Invalid word '' [astropy.io.votable.exceptions]
WARNING: W06: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:745:0: W06: Invalid UCD 'src.gal.extinc.E(B-V)': UCD has invalid character '(' in 'src.gal.extinc.E(B-V)' [astropy.io.votable.exceptions]
WARNING: W33: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:736:0: W33: Column name 'refcode' renamed to 'refcode 2' to ensure uniqueness [astropy.io.votable.exceptions]
WARNING: W33: /var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/tmpcvfG_X:742:0: W33: Column name 'refcode' renamed to 'refcode 3' to ensure uniqueness [astropy.io.votable.exceptions]
Out[5]: 
<Table rows=1 names=('main_col1','main_col2','main_col3','main_col4','main_col5','main_col6','main_col7','main_col8','main_col9','main_col10','main_col11','main_col12','main_col13','main_col14','main_col15','main_col16','main_col17')>
masked_array(data = [ (1, 'MESSIER 001', 83.63321, 22.01446, 'SNR', --, --, '', '', --, 73, 0, 44, 4, 0, 2, 0)],
             mask = [ (False, False, False, False, False, True, True, False, False, True, False, False, False, False, False, False, False)],
       fill_value = (999999, 'N/A', 1e+20, 1e+20, '?', 1e+20, 1e+20, '?', '?', 1e+20, 999999, 999999, 999999, 999999, 999999, 999999, 999999),
            dtype = [('main_col1', '<i4'), ('main_col2', 'S30'), ('main_col3', '<f8'), ('main_col4', '<f8'), ('main_col5', 'O'), ('main_col6', '<f8'), ('main_col7', '<f8'), ('main_col8', 'O'), ('main_col9', 'O'), ('main_col10', '<f8'), ('main_col11', '<i4'), ('main_col12', '<i4'), ('main_col13', '<i4'), ('main_col14', '<i4'), ('main_col15', '<i4'), ('main_col16', '<i4'), ('main_col17', '<i4')])

[cdeil]

This query still gives an error, so I'm leaving this issue open for now:

In [6]: ned.query_ned_basic_posn("PG 1553+11")
ERROR: ExpatError: mismatched tag: line 21, column 2 [xml.dom.expatbuilder]
---------------------------------------------------------------------------
ExpatError                                Traceback (most recent call last)
<ipython-input-6-32386f3cbd31> in <module>()
----> 1 ned.query_ned_basic_posn("PG 1553+11")

/Users/deil/Library/Python/2.7/lib/python/site-packages/astroquery-0.0.dev270-py2.7.egg/astroquery/ned/nedpy.pyc in query_ned_basic_posn(objname, root_url)
    497     # Check to see if NED returns a valid query
    498 
--> 499     validtable = check_ned_valid(R)
    500 
    501     if validtable:

/Users/deil/Library/Python/2.7/lib/python/site-packages/astroquery-0.0.dev270-py2.7.egg/astroquery/ned/nedpy.pyc in check_ned_valid(str)
     49     retval = True
     50 
---> 51     strdom = parseString(str)
     52     p = strdom.getElementsByTagName('PARAM')
     53 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/minidom.pyc in parseString(string, parser)
   1928     if parser is None:
   1929         from xml.dom import expatbuilder
-> 1930         return expatbuilder.parseString(string)
   1931     else:
   1932         from xml.dom import pulldom

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/expatbuilder.pyc in parseString(string, namespaces)
    938     else:
    939         builder = ExpatBuilder()
--> 940     return builder.parseString(string)
    941 
    942 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/expatbuilder.pyc in parseString(self, string)
    221         parser = self.getParser()
    222         try:
--> 223             parser.Parse(string, True)
    224             self._setup_subset(string)
    225         except ParseEscape:

ExpatError: mismatched tag: line 21, column 2

[keflavich]

works now:

In [3]: Ned.query_object('crab').pprint()
No. Object Name  RA(deg)    DEC(deg)  Type  Velocity   Redshift  ... References Notes Photometry Points Positions Redshift Points Diameter Points Associations
--- ----------- ---------- ---------- ---- ---------- ---------- ... ---------- ----- ----------------- --------- --------------- --------------- ------------
  1 MESSIER 001   83.63321   22.01446  SNR         --         -- ...         75     0                44         4               0               2            0