Comments (12)
Thanks everybody for your links and suggestions! This is really helpful.
from astroquery.
The https://masttest.stsci.edu/api/v0/_c_a_o_mfields.html link is already in the docstring, so adding a really short sentence along the line of e.g. "Please note that obsid
MAST specific and is not the same as the mission-specific obs-id
".
from astroquery.
Evidently this problem has come up before, e.g. here #1535
from astroquery.
cc @jaymedina
from astroquery.
Having the same problem popping up again shows that we need much better testing for the mast module.
(unrelated, but I also see 11 test failures, some from a rather long time now (#2801), would be nice to get that number down to zero)
from astroquery.
Hi @aggle! So for this case, the issue is that the obs_id
s metadata is not supposed to be fed into get_product_list
; it's supposed to be the obsid
metadata, or the Product Group ID. See this page for the difference between obs_id
and obsid
. The gist is that: obs_id
is mission-specific and obsid
is MAST-specific, and is just an arbitrary identifier used in the backend for querying purposes. If you feed a set of obsid
s instead, you should get this output:
In
# Looking for observations of GJ 758
observations = mast.Observations.query_object("GJ 758")
# select 2 obs_ids
obs_ids = observations[observations["project"] == 'JWST']['obsid'][:2]
# get the associated products from MAST
mast.Observations.get_product_list(obs_ids)
Out
<Table masked=True length=73>
obsID obs_collection dataproduct_type obs_id ... size parent_obsid dataRights calib_level
str9 str4 str5 str34 ... int64 str9 str6 int64
--------- -------------- ---------------- ---------------------------------- ... ------- ------------ ---------- -----------
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 34560 152009184 PUBLIC 1
... ... ... ... ... ... ... ... ...
Let me know if this helps and is what you expected!
This would not have been caught in the unit tests because get_product_list
isn't supposed to be utilized with obs_id
. But this brings to my attention that the obsid
and obs_id
criteria are unnecessarily confusing, and either the documentation needs to be updated to emphasize their differences, or we should get rid of obsid
as a criteria to search with altogether. I'll bring this up during our standup tomorrow, as one of our members is heavily involved with the backend database that uses obsid
, and we'll work on getting this sorted out @bsipocz . The other broken unit tests are on my radar as well.
Thanks @aggle for bringing this to our attention and for using astroquery.mast
!
from astroquery.
@jaymedina that answers everything. It's a little confusing, as you say, but I can work with it for now. Thanks very much for your help!
from astroquery.
Oops, I pressed "close" by accident :-/
from astroquery.
Apologies, my comment was incomplete. This does solve my immediate problem of "how to use get_product_list
", but it doesn't help with the larger problem I'm trying to solve, which is: given a set of JWST observations, for which I have the obs_id
but not the obsid
, how can I use astroquery to find the related products?
In my particular case, I am looking for background observations that are linked to particular science observations. I think that is outside the scope of this ticket, though.
from astroquery.
If things all work as expected, I would still suggest to document this behaviour with an example in the docs, or at least leave a note about the differences of obs_id
and obsid
. Maybe even try to catch this particular exception and reraise a more descriptive message in the error?
from astroquery.
given a set of JWST observations, for which I have the obs_id but not the obsid, how can I use astroquery to find the related products?
You can use mast.Observations.query_criteria()
to search with obs_id
. To demonstrate, I'll use one of the obs_id
s you produced from your example. Since obs_id
is an Observations criteria, you can feed it as a keyword argument in a query_criteria
call, like so:
from astroquery import mast
# Given a known `obs_id`, retrieve my observation
obs = mast.Observations.query_criteria(obs_id="jw01413001001_02101_00001_mirimage")
# Get products from observation
prods = mast.Observations.get_product_list(obs)
The output Table
assigned to prods
will look something like this:
<Table masked=True length=45>
obsID obs_collection dataproduct_type obs_id ... size parent_obsid dataRights calib_level
str9 str4 str5 str34 ... int64 str9 str6 int64
--------- -------------- ---------------- ---------------------------------- ... ------- ------------ ---------- -----------
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 406080 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 420480 152009184 PUBLIC 1
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 1284480 152009184 PUBLIC 2
152009184 JWST image jw01413001001_02101_00001_mirimage ... 34560 152009184 PUBLIC 1
... ... ... ... ... ... ... ... ...
query_criteria
is very useful and versatile, because it also allows for cone searches. If you'd like to see the list of criteria you can work with, in addition to obs_id
, you can run the following command, and it'll give you back a table:
mast.Observations.get_metadata("Observations")
Where the values under Column Name
are the criteria you can work with. For more examples on the utility of astroquery.mast
, feel free to visit the astroquery section of the mast_notebooks
repo: https://github.com/spacetelescope/mast_notebooks/tree/main/notebooks/astroquery
from astroquery.
If things all work as expected, I would still suggest to document this behaviour with an example in the docs, or at least leave a note about the differences of
obs_id
andobsid
. Maybe even try to cat this particular exception and reraise a more descriptive message in the error?
Sounds good - if it turns out we need to keep the obsid
criteria for backend purposes, I'll start a new PR with a few of these changes whipped up.
from astroquery.
Related Issues (20)
- Add support for upload tables for queries in ALMA
- BUG: CI -- no space left on device HOT 2
- ESA Herschel download_data connection error HOT 2
- MAINT: retire/rename exceptions.TimeoutError in favour of requests
- TST: esasky tests take too long time and donwload too much data HOT 8
- BUG: alma test downloads 850+Mb of data HOT 8
- How to download Level 1 ancillary (*cbcd.fits and *cbunc.fits) from IRSA?
- Vizier.query_region() returns incorrect index for some catalog HOT 3
- CI: rerun failing tests at the end
- Bug in rows limitation in module vizier HOT 2
- Unexpected behavior when passing `obj_names` to `SDSS.query_region()`
- Handling No astronomical object found response HOT 5
- IRSA: adding SSA method HOT 1
- Astroquery.gaia API Error: RA cannot be converted to hour or hourangle. HOT 2
- MAST cloud dataset doesn't show level 3 data products
- SimbadClass got an unexpected keyword argument HOT 5
- API cleanup to make it more self consistent
- API discussion for `list_collections` HOT 4
- Astrometry.net: can query for job id once, rather than each time HOT 2
- BUG: vo_conesearch server SSL cert needs fixing HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from astroquery.