Comments (7)
Hi Sriram,
Does this mean that the solutions are applied directly to the data column?
Yes. This is mostly to reduce the size / download time.
Is there a way to keep the data column intact and have the solutions applied to only the corrected column?
I like your suggestion. The -m
flag predates the applycal machinery by several years, but they could certainly work better together. I would probably keep the model data column equal to zero.
Are the solutions only applied to the target field?
I'm somewhat surprised by this. As far as I remember the L1 solutions like G
are applied everywhere, while the L2 selfcal solutions like GPHASE
are only applied to the target. Maybe there was a mixup of some sort there?
I'm also don't expect the weights and sigma columns to be empty. If there were network issues downloading the dataset, the weights could end up zero but then the data_lost
flag should also be set.
Could you please supply the capture block ID and flags passed to mvftoms, if possible?
from katdal.
Agreed Jordan. I still have to see whether the dataset contains sufficient information to reconstruct the model data column...
from katdal.
And also are the solutions only applied to the target field as pointed out in caracal issue #1454
from katdal.
The WEIGHTS_SPECTRUM and SIGMA_SPECTRUM columns of data obtained with the 'default calibrated' setting also appear to be '0' and 'inf' (See discussion in caracal issue #1451)
from katdal.
Hi Ludwig,
Thanks for your response.
I'm also don't expect the weights and sigma columns to be empty. If there were network issues downloading the dataset, the weights could end up zero but then the
data_lost
flag should also be set.
The data was pushed to Ilifu from the backend, I suspect there could have been network issues. The behaviours also seem to be consistent across multiple datasets. The weights seem to be modified at some stage.
Could you please supply the capture block ID and flags passed to mvftoms, if possible?
Due to a bug in the UI of the archive, I am unable to push the data by myself. (jira service ticket) . I can request this information from ilifu staff who, initiate the push from the backend. Otherwise, I can also provide the observation ID.
from katdal.
Thanks, the service ticket has all the info I need. I'll check whether the raw data suffers from the same weight=0 issues.
from katdal.
Hi @ludwigschwardt, as we discussed previously, if possible, I think it would great if using the mvftoms.py --applycal
parameter in conjunction with the -m, --model-data
parameter caused the data column to remain as the level 0 data with no solutions applied (as is the case when not using the --applycal
parameter), and the solutions to be applied to the CORRECTED_DATA
column. Similarly, it would be great if any model data general by the SDP pipelines (e.g. from cross-cal and/or selfcal, according to the --applycal
selection) be written to the MODEL_DATA
column.
For now, since I think it doesn't make sense to use the -m, --model-data
parameter during raw data transfers, I've added its removal from the SARAO archive here, as a sub-task for an upcoming archive UI upgrade.
from katdal.
Related Issues (20)
- POINTING table is not filled HOT 3
- issue error message when dataset is not on Ceph HOT 1
- cityhash issue with windows HOT 3
- improve target_[xy] documentation HOT 1
- Log level for data read is too low HOT 1
- Error in van_vleck.py HOT 2
- Populate SIGMA_SPECTRUM HOT 15
- BadChunk: Array '1543661287/correlator_data HOT 1
- Dump of hdf5 files on disk HOT 1
- Simple download does not work HOT 1
- mvftoms produced malformed MS (TiledStMan::headerFileGet: mismatch in #row) HOT 23
- mvtoms.py: AttributeError: module 'numpy' has no attribute 'bool'. HOT 3
- OutOfRangeError when changing the projection in datasets HOT 2
- Replace underscores with dashes when loading buckets from RDBs HOT 4
- casacore missing as a dependency HOT 2
- mvftoms creates incorrect ms file HOT 4
- ms does not exist and cannot be created HOT 2
- mvftoms.py reading time-out on Jupyter core HOT 5
- Problem when downloading raw data from SARAO using mvf_download.py HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from katdal.