Comments (9)
Can you confirm with NCAR's HPC support that this is even expected to work? Many HPC environments disable access to external networks.
from siphon.
Can you test and see if this works?
import urllib.request
resp = urllib.request.urlopen('http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=2022&MONTH=08&FROM=1100&TO=1100&STNM=72632')
print(resp.read())
from siphon.
Thank you, dopplershift -- your suggestion works! I am very grateful.
If possible, I would like to see NCAR JupyterHub be able to handle such data transfers by default.
Would this entail changes on the Unidata side or on the NCAR/UCAR side?
from siphon.
@rvalent Wait, are you saying that sample code worked on Casper? Because I was expecting it to fail given your previous problem.
from siphon.
@dopplershift You are correct! I just checked it now from the JH window. That sample code fails in the same way:
Traceback (most recent call last):
File "", line 3, in
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/glade/work/valent/conda-envs/casper-rozoff-siphon/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
from siphon.
@rvalent Ok, well that sample code will work on any machine with a non-impeded network configuration--and involves Siphon in no way.
In other words, the only way this works is if NCAR tweaks something, but I want to be clear: it is very common, if not expected, that you do not have arbitrary network access in an HPC environment.
Either way, there's no problem with Siphon here and there's nothing we can do to solve this on our end.
from siphon.
@dopplershift Thank you. This is very helpful.
from siphon.
The network on this cluster is not bound by any outgoing rules. It's more closer architected to a cloud like element. I believe it might be a data server issue. running this several times on various machines, I would potentially get a 403 on Casper systems or sometimes a too many request or system is too busy. This suggests a rate limiter impoementation on the server side that may have blocked our system at some point (forever because of lots of requests). Hard to say without getting in touch with the data server administrator.
from siphon.
@jbaksta That's interesting. I regularly get a 503 from the Wyoming server when they're overloaded, but I have never experienced a 403.
from siphon.
Related Issues (20)
- filter_time_nearest : TypeError: can't subtract offset-naive and offset-aware datetimes HOT 2
- Client side NCSS query validation HOT 1
- Suggested improvement for api/catalog
- issues getting CFS point data
- Update dataset in Wyoming upper air example
- ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) HOT 2
- Endian-ness issues when selecting on a coordinate HOT 3
- Datasets in nested catalogs have empty access_urls
- TDSCatalog does not include base path in access_url HOT 2
- No access urls created for Dataset HOT 4
- NDBC Mid Great Lakes Station Do Not Show Up HOT 1
- WyomingUpperAir Missing Station Information
- Requesting a new release HOT 1
- Exception handling needed in _get_data method of WyomingUpperAir for missing or erroneous metadata HOT 2
- Protobuf module started making it impossible to access the UCAR THREDDS server using siphon TDSCatalog HOT 4
- TDSCatalog misses the last months entry HOT 2
- Add a walk function for navigating THREDDS catalogue HOT 2
- how to add the Specify additional variables you want to include HOT 1
- Descriptors cannot be created directly
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from siphon.