Fermi LAT data access about astroquery

[keflavich]

Yes, it's possible. I will probably try to unify the interface before
implementing new query services, but adding this one shouldn't be hard.
On Jan 7, 2013 8:33 AM, "Christoph Deil" [email protected] wrote:

I would like to add access to the Fermi LAT data to astroquery.

There's a bunch of different data products, most are FITS files with
tables, which I guess is easy:
http://fermi.gsfc.nasa.gov/ssc/data/access/

However the main data server is via this web form where the user enters
some parameters:
http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

With acceptable parameters one gets a page that contains info on the
status of the data preparation on their data server and after a minute or
so the download links. Here's an example:

http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/QueryResults.cgi?id=L1301071121485C26556A03

@keflavich https://github.com/keflavich Is it possible to interface
this data server from astroquery?
(I'll also ask their helpdesk as well if this is allowed and if there is a
documented web API I don't know about)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/37.

[cdeil]

Great!
I'm curious, are you thinking screen scraping, or is there a proper solution?

[keflavich]

Essentially scraping... that's what we've done for most projects, I think.

[keflavich]

Upon closer look, the POST queries are not so simple, and the data is returned in a slightly different way than most others. The besancon model will serve as a decent model for the data retrieval process, but I'm not so sure how to make the POST query just yet.

[kialio]

Interesting idea. I have one request: let us know how we (at the FSSC at Goddard) can locate these queries in our logs (probably easy to find since the user agent would be pretty unique but would need to know what it actually is). This way, if there are issues on our end responding to these we can find the problem.

[keflavich]

@kialo - is there a description of the POST keywords available somewhere? Perhaps we could add one identifying the query source by astroquery version?

[kialio]

Let me talk to our software developers tomorrow at our weekly meeting and see if I can get this information for you.

[kialio]

I talked to the folks around here and the word is that we won't be able to provide you with the POST keywords (our IT security folks would frown on that). Other than that, we encourage you to do this and will help you if we can (just email me, chat here or send a query via the helpdesk).

[cdeil]

@keflavich Can you give this a try and sketch a solution (preferably in Python :-)? I would very much like this but don't know how to get started.

[keflavich]

@cdeil - I let this one slide because of the last respons from kialio. However, let's see if we can piece together what basic queries look like. I think we need to encode the query in json (which can be done without too much difficulty, though I can't tell you how off the top of my head) and then send that json bundle to the server.

This site supposedly says which http methods (POST, GET, etc.) allow you to do that:
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-14#page-17

I'm investigating further, but - while I have some idea of where to start - I'm not sure I'll get anywhere.

[keflavich]

This gives a useful explanation:
http://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python

[keflavich]

Woo! got it! Requires a new dependency, requests, which looks pretty mature.

payload = {'shapefield':1,'coordsystem':'J2000', 'coordfield':'m31', 'destination':'query', 'timefield':'', 'timetype': 'Gregorian', 'energyfield':'', 'photonOrExtendedOrNone': 'Photon', 'spacecraft':'on'}
result = requests.post(url, data=payload)

# if you want to see the result:
with open('/var/tmp/test.html','w') as f: f.write(result.content)

OK, now have to scrape the link on that page.

[astrofrog]

I'm not sure I understand - you can just do the query with urllib, right? (it handles POST requests). Or am I missing some subtlety?

[keflavich]

You're missing a subtlety: http://bugs.python.org/issue3244 (urllib/urllib2 currently doesn't support the type of requests FERMI/LAT requires)

[keflavich]

(just fyi, I wasn't sneaky enough to find that issue directly; got linked from here http://stacyprowell.com/blog/2009/05/29/handling-multipartform-data-in-python/)

[astrofrog]

Ok, I see now :)

[cdeil]

I think using the requests package should be fine.
It looks much easier than the Python build-in web modules.

[keflavich]

Further discussion on #68