Irsa query_region polygon fails in 0.4.7 about astroquery

[keflavich]

Could you please give an example of the failing query?

Also, point us to the documentation that led you to trying clear_cache. If that method doesn't exist, we shouldn't be suggesting it in the docs.

[nugent-lab]

Here's an example [edited by bsipocz for copy pasteability]:

from astroquery.ipac.irsa import Irsa
frame_corner = [(342.08839797, 15.98217039), (342.0897537, 14.84387902), (340.91116881, 14.83933207), (340.90373378, 15.97857899)]
results = Irsa.query_region(catalog="allwise_p3as_psd", spatial="Polygon", polygon=frame_corner)

The frame corners were (in degrees):
(342.08839797, 15.98217039) (342.0897537, 14.84387902) (340.91116881, 14.83933207) (340.90373378, 15.97857899)

Here's the documentation that suggested to me that the clear_cache method was possible:
https://astroquery.readthedocs.io/en/latest/
Under the subheading "Service specific settings" there's an example with Simbad, and then it reads: "Each service’s cache is cleared with the clear_cache function within that service."

[bsipocz]

We switched to using pyvo as the backend for the irsa module in 0.4.7 and thus don't do caching at all. I don't see any caching or clear_cache() mentioned in the IRSA modude's docs either, but we certainly have to rephrase the cache parts on the landing page to make it clear that not all modules use the machinery shown there. cc @ceb8

I tried to run your example and got back results as expected. If the only problems you run into with the polygon search was the caching, then I would suggest to close the issue, otherwise we will need more info about the version you use etc to be able to reproduce the failures on our end:

In [7]: results
Out[7]: 
<Table length=19055>
    designation          ra         dec      sigra   ...          z           spt_ind      htm20     
                        deg         deg      arcsec  ...                                             
       object         float64     float64   float64  ...       float64         int64       int64     
------------------- ----------- ----------- -------- ... ------------------- --------- --------------
J224818.28+151558.3 342.0761952  15.2662042   0.6377 ...  0.2633040618468260 200333320 13468532754820
J224800.82+151308.9 342.0034317  15.2191599   0.4735 ...  0.2625118687134020 200333320 13468546284286
J224814.40+152545.4 342.0600122  15.4292897   0.0701 ...  0.2660489299782870 200333320 13468530822000
J224811.78+153150.3 342.0491034  15.5306560   0.2793 ...  0.2677539264789150 200333320 13468516074529
J224802.54+151653.1 342.0105923  15.2814207   0.1935 ...  0.2635602591042980 200333320 13468545608125
J224817.61+150959.0 342.0733885  15.1664061   0.0868 ...  0.2616233211171400 200333320 13468534045016
J224801.39+150608.2 342.0058173  15.1022810   0.0897 ...  0.2605429448287190 200333320 13468540417777
J224813.30+150957.5 342.0554191  15.1659851   0.4482 ...  0.2616162291986450 200333320 13468533616314
J224807.32+152000.4 342.0305111  15.3334669   0.2801 ...  0.2644364102333660 200333320 13468526907823
J224814.20+153255.6 342.0591928  15.5487879   0.2811 ...  0.2680588195719410 200333320 13468515901455
J224811.88+152642.0 342.0495021  15.4450160   0.3869 ...  0.2663135034324990 200333320 13468529461637
J224809.90+153159.9 342.0412773  15.5333190   0.3753 ...  0.2677987072674870 200333320 13468516036986
J224815.30+153211.3 342.0637659  15.5364991   0.3969 ...  0.2678521828091330 200333320 13468515975695
J224820.99+152729.8 342.0874791  15.4582832   0.4618 ...  0.2665366902960820 200333320 13468530078702
                ...         ...         ...      ... ...                 ...       ...            ...
J224521.29+155800.5 341.3387397  15.9668324   0.5485 ...  0.2750808508167910 200333012 13465152376849
J224544.52+155632.5 341.4355243  15.9423616   0.4666 ...  0.2746702066061790 200333012 13465157835733
J224532.10+155846.6 341.3837601  15.9796318   0.0408 ...  0.2752956174178050 200333012 13465151818812
J224557.46+155515.1 341.4894548  15.9208751   0.0784 ...  0.2743096005865770 200333012 13465157467254
J224522.65+155753.0 341.3443890  15.9647426   0.5775 ...  0.2750457838658110 200333012 13465152336713
J224443.05+155615.4 341.1794112  15.9376130   0.6382 ...  0.2745905145977830 200332221 13463218704594
J224516.88+155543.9 341.3203632  15.9288679   0.1106 ...  0.2744437475472670 200333012 13465151377278
J224510.24+155636.4 341.2926828  15.9434597   0.1911 ...  0.2746886348837950 200332221 13463219774088
J224449.02+155603.3 341.2042881  15.9342773   0.2171 ...  0.2745345330478260 200332221 13463218278931
J224432.43+155544.9 341.1351450  15.9291581   0.2379 ...  0.2744486180113140 200332221 13463217805203
J224355.40+155756.0 340.9808722  15.9655635   0.2099 ...  0.2750595586534730 200332221 13463243950030
J224451.72+155645.8 341.2155322  15.9460607   0.1234 ...  0.2747322843765970 200332221 13463220019698
J224653.45+155607.7 341.7227128  15.9354725   0.5617 ...  0.2745545916578250 200333013 13465252135214
J224602.63+155828.6 341.5109596  15.9746181   0.1487 ...  0.2752114920396020 200333012 13465156893193

[nugent-lab]

Thanks for the clarification on caching.

The major issue is, as stated in the first comment, when I run Irsa.query_region on a polygon, it returns an empty table in astroquery 0.4.7. I assume you ran the query on astroquery 0.4.7 and it worked?

Please let me know which packages you would like to know the version of. I've got astropy 5.3.1. The empty table is not unique to those coordinates, I just gave those as an example.

I would love to upgrade to astroquery 0.4.7 and not have to worry about the caching, so I appreciate your help here.

[bsipocz]

I tried it both with the most recent dev version, but also with 0.4.7 and haven't run into any issues. No point in checking it with 0.4.6 as the module has been totally rewritten and accesses a different service on the server side, so we compare apples to pears with that.

Do you still have empty results? There is an off chance that you hit an IRSA server downtime, but we usually have those on Tuesdays.

What version of pyvo do you have? I'm really just guessing blindly as cannot reproduce the empty table with the latest releases, nor with astropy 5.3.1

[bsipocz]

Also, you can try to extract the ADQL string from the query with setting get_query_payload=True:

Irsa.query_region(catalog="allwise_p3as_psd", spatial="Polygon", polygon=frame_corner, get_query_payload=True)

And the pasting it as the query string directly in the ADQL query box on the IRSA website. If you still see an empty table after all the ideas from the previous comment, my best suspicion is a typo somewhere. So you can play around with the query until it returns something. The URL:
https://irsa.ipac.caltech.edu/irsaviewer/?api=tap&service=https://irsa.ipac.caltech.edu/TAP&schema=fp_2mass&table=fp_psc&adql=select

[nugent-lab]

Hi,

I'm confident it's not IRSA server downtime; both from checking the website and because it resolves as soon as I switch between versions. I think I have another/better example for you. The following works great on 0.4.6 but does the following on 0.4.7:

from astroquery.ipac.irsa import Irsa
from astropy.coordinates import SkyCoord
Irsa.ROW_LIMIT = 100000 

frame_corner_1=SkyCoord(0,    -24,   frame='icrs', unit='deg')
frame_corner_2=SkyCoord(0.5,  -24,   frame='icrs', unit='deg')
frame_corner_3=SkyCoord(0,    -23.5, frame='icrs', unit='deg')
frame_corner_4=SkyCoord(0.5,  -23.5, frame='icrs', unit='deg')
results = Irsa.query_region(catalog="allwise_p3as_psd", spatial="Polygon", polygon=[frame_corner_1,frame_corner_2,frame_corner_3,frame_corner_4])

>>> results
<Table length=0>
designation    ra     dec    sigra   sigdec ...    y       z    spt_ind htm20
              deg     deg    arcsec  arcsec ...                              
   object   float64 float64 float64 float64 ... float64 float64  int64  int64
----------- ------- ------- ------- ------- ... ------- ------- ------- -----
>>> 

I'm using pyvo version 1.4.1.

[bsipocz]

your corner points are not a closed convex polygon. Try switching corner3 and corner4.

>>> from astroquery.ipac.irsa import Irsa
>>> from astropy.coordinates import SkyCoord
>>> frame_corner_1=SkyCoord(0,    -24,   frame='icrs', unit='deg')
>>> frame_corner_2=SkyCoord(0.5,  -24,   frame='icrs', unit='deg')
>>> frame_corner_3=SkyCoord(0,    -23.5, frame='icrs', unit='deg')
>>> frame_corner_4=SkyCoord(0.5,  -23.5, frame='icrs', unit='deg')
>>> results = Irsa.query_region(catalog="allwise_p3as_psd", spatial="Polygon", polygon=[frame_corner_1,frame_corner_2,frame_corner_4,frame_corner_3])
>>>
>>> results
<Table length=3321>
    designation          ra         dec      sigra   ...          z           spt_ind      htm20    
                        deg         deg      arcsec  ...                                            
       object         float64     float64   float64  ...       float64         int64       int64    
------------------- ----------- ----------- -------- ... ------------------- --------- -------------
J000006.89-235740.0   0.0287318 -23.9611319   0.2070 ... -0.4061168219295780 100120012 8899575241866
J000027.94-234554.4   0.1164216 -23.7651244   0.2312 ... -0.4029882911237760 100120001 8899240909725
J000025.97-235246.1   0.1082459 -23.8794948   0.1917 ... -0.4048143646612140 100120032 8900115576758
J000005.89-235527.5   0.0245807 -23.9243099   0.2528 ... -0.4055294570968990 100120012 8899575094676
J000131.69-235834.3   0.3820590 -23.9761945   0.0468 ... -0.4063570441344990 100120012 8899583527302
J000035.91-234608.0   0.1496402 -23.7688954   0.2852 ... -0.4030485257270480 100120001 8899241357892
J000017.45-234950.8   0.0727113 -23.8308031   0.3612 ... -0.4040371345231300 100120001 8899239710668
J000010.31-234840.8   0.0429929 -23.8113539   0.3464 ... -0.4037265995237640 100120001 8899242844058
J000020.84-235605.8   0.0868686 -23.9349504   0.2983 ... -0.4056992058745450 100120012 8899578324250
J000006.23-235911.6   0.0259895 -23.9865734   0.2009 ... -0.4065225531497170 100120012 8899578777660
J000032.66-234636.3   0.1360851 -23.7767696   0.4410 ... -0.4031742955718550 100120001 8899241404526
J000004.00-235419.6   0.0166935 -23.9054462   0.2193 ... -0.4052284885561920 100120032 8900111788326
J000027.35-235000.2   0.1139887 -23.8333988   0.3725 ... -0.4040785751595420 100120032 8900114128650
J000033.07-234900.3   0.1378093 -23.8167651   0.5006 ... -0.4038130019395840 100120032 8900114343463
                ...         ...         ...      ... ...                 ...       ...           ...
J000038.34-234507.4   0.1597887 -23.7520708   0.5464 ... -0.4027797710377560 100120001 8899254158682
J000032.30-233032.0   0.1346025 -23.5089136   0.2342 ... -0.3988917326649170 100120001 8899298030635
J000007.50-233255.2   0.0312749 -23.5486674   0.2815 ... -0.3995278816519930 100120001 8899294111956
J000028.74-233726.9   0.1197749 -23.6241501   0.3068 ... -0.4007352427361460 100120001 8899250559459
J000024.83-234429.8   0.1034929 -23.7416339   0.2746 ... -0.4026130354535110 100120001 8899255077317
J000033.31-233115.6   0.1388213 -23.5210229   0.4601 ... -0.3990855286794230 100120001 8899298038551
J000036.81-233431.0   0.1534150 -23.5753055   0.5762 ... -0.3999540427436510 100120001 8899295510985
J000002.82-233912.4   0.0117674 -23.6534488   0.0411 ... -0.4012036941192010 100120001 8899249360537
J000034.05-233736.0   0.1418945 -23.6266813   0.4308 ... -0.4007757177522270 100120001 8899250432491
J000043.32-233419.4   0.1805222 -23.5720748   0.6629 ... -0.3999023620316090 100120001 8899295145505
J000144.53-233155.3   0.4355607 -23.5320398   0.2255 ... -0.3992618265483160 100120001 8899265249975
J000027.04-233611.2   0.1126886 -23.6031179   0.6233 ... -0.4003988982435830 100120001 8899250220505
J000032.39-233559.9   0.1349619 -23.5999828   0.4580 ... -0.4003487574678720 100120001 8899250842228
J000008.22-234446.9   0.0342795 -23.7463723   0.5228 ... -0.4026887358221220 100120001 8899241630107

[nugent-lab]

Was there a check in 0.4.6 that corrected for this that is not present in 0.4.7? I don't mind updating my code, but I could see this causing issues for other users. For general usability, it would be good if 0.4.7 worked for code that worked with 0.4.6, or at least returned a helpful error message instead of a blank array. Thank you for your help.

[bsipocz]

As I said above we have fully refactored which IRSA backend we use in the module. The older one (GATOR) was more lenient and made some assumptions about the points you entered even if they don't properly define a convex polygon. The TAP server OTOH requires the users to be more precise. Please see more docs on this under geometric constraint in the docs here: https://irsa.ipac.caltech.edu/docs/program_interface/TAP.html

As for astroquery, we do return the exact same results that you would get interacting with the server's website directly.
We don't plan to keep running against both backends and raise warnings when the results differ and we definitely don't track ambiguous queries like this in our test suite. Please note that you are searching along a Z shape, so that it worked previously is a bug.

The astroquery documentation is also quite explicit that you need a convex polygon for the polygon query. This was the very same sentence for v0.4.6, too "This is a list of coordinate pairs that define a convex polygon", which your example fails to do:
https://astroquery.readthedocs.io/en/latest/ipac/irsa/irsa.html#queries-over-a-polygon

I would strongly advise updating your code to define closed, convex polygons.