Comments (5)
The read timeout is most likely because @ludwigschwardt seems to like setting short timeouts for some reason. Try searching for timeout=10
in your katdal installation (should be in chunkstore_s3.py) and changing it to something much bigger e.g. timeout=300
.
I can't imagine what would cause scans to be skipped non-deterministically, but maybe see if increasing the timeout fixes that problem too.
from katdal.
@sirothia @spassmoor did the suggested workaround work?
from katdal.
@bmerry the problem re-occurered for me so I rewrote my data reading function to read a scan (~50 reads) at a time for all antennas instead of originally reading for one Antenna at a time , one scan at a time ( about ~3000 reads). This seems to work , also 20 times faster read times.
from katdal.
@sirothia, just be aware that MS conversion only considers scans where the reference antenna tracks an radec target, so any 'slew' or 'scan' antenna activity, or azel or satellite targets, are ignored.
from katdal.
So i tried using mvftoms a quite heavily over the last few days. I found that the timeouts occur whether you use the haproxy or any of the gateways into ceph. However, once I used the suggestion by @bmerry I found I could quite happily convert observations without a problem.
from katdal.
Related Issues (20)
- issue error message when dataset is not on Ceph HOT 1
- cityhash issue with windows HOT 3
- improve target_[xy] documentation HOT 1
- Log level for data read is too low HOT 1
- Error in van_vleck.py HOT 2
- Populate SIGMA_SPECTRUM HOT 15
- BadChunk: Array '1543661287/correlator_data HOT 1
- Dump of hdf5 files on disk HOT 1
- Simple download does not work HOT 1
- Question about applying calibration solutions HOT 7
- mvftoms produced malformed MS (TiledStMan::headerFileGet: mismatch in #row) HOT 23
- mvtoms.py: AttributeError: module 'numpy' has no attribute 'bool'. HOT 3
- OutOfRangeError when changing the projection in datasets HOT 2
- Replace underscores with dashes when loading buckets from RDBs HOT 4
- casacore missing as a dependency HOT 2
- mvftoms creates incorrect ms file HOT 4
- ms does not exist and cannot be created HOT 2
- mvftoms.py reading time-out on Jupyter core HOT 5
- Problem when downloading raw data from SARAO using mvf_download.py HOT 7
- katdal does not work with python 3.12 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from katdal.