Not converting all scans in archive data to ms and Read timed out about katdal

[bmerry]

The read timeout is most likely because @ludwigschwardt seems to like setting short timeouts for some reason. Try searching for timeout=10 in your katdal installation (should be in chunkstore_s3.py) and changing it to something much bigger e.g. timeout=300.

I can't imagine what would cause scans to be skipped non-deterministically, but maybe see if increasing the timeout fixes that problem too.

[bmerry]

@sirothia @spassmoor did the suggested workaround work?

[spassmoor]

@bmerry the problem re-occurered for me so I rewrote my data reading function to read a scan (~50 reads) at a time for all antennas instead of originally reading for one Antenna at a time , one scan at a time ( about ~3000 reads). This seems to work , also 20 times faster read times.

[ludwigschwardt]

@sirothia, just be aware that MS conversion only considers scans where the reference antenna tracks an radec target, so any 'slew' or 'scan' antenna activity, or azel or satellite targets, are ignored.

[ctgschollar]

So i tried using mvftoms a quite heavily over the last few days. I found that the timeouts occur whether you use the haproxy or any of the gateways into ceph. However, once I used the suggestion by @bmerry I found I could quite happily convert observations without a problem.