mwatelescope / birli Goto Github PK
View Code? Open in Web Editor NEWA Rust library for preprocessing tasks in the Murchison Widefield Array (MWA) data pipeline.
License: Mozilla Public License 2.0
birli's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
birli's Issues
Apply "receiver type" phase correction
this thingy
PhaseCorrection.txt
macOS CI is broken
build succeeds in https://github.com/MWATelescope/Birli/runs/5314505207 but not https://github.com/MWATelescope/Birli/runs/5370492134 even though the CI has not changed
originally failed at linking to boost
dyld: lazy symbol binding failed: Symbol not found: __ZN5boost10filesystem4pathdVERKS1_
Referenced from: /usr/local/opt/aoflagger/lib/libaoflagger.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_filesystem-mt.dylib
dyld: Symbol not found: __ZN5boost10filesystem4pathdVERKS1_
Referenced from: /usr/local/opt/aoflagger/lib/libaoflagger.0.dylib
Expected in: /usr/local/opt/boost/lib/libboost_filesystem-mt.dylib
but after this run https://github.com/MWATelescope/Birli/runs/5548128912 , it fails with
Last 15 lines from /Users/runner/Library/Logs/Homebrew/aoflagger/02.make:
^
/tmp/aoflagger-20220414-56808-1m5guqg/lua/optionsfunction.cpp:10:6: error: template specialization requires 'template<>'
std::map<std::string, Options> OptionsFunction::GetOptions(lua_State *state, const Options& cmdLineOptions)
^ ~~~~~~~~~~~~~~~~~~~~~~
template<>
/tmp/aoflagger-20220414-56808-1m5guqg/lua/optionsfunction.cpp:10:6: error: no variable template matches specialization; did you mean to use 'max' as function template instead?
/tmp/aoflagger-20220414-56808-1m5guqg/lua/optionsfunction.cpp:10:31: error: expected ';' after top level declarator
std::map<std::string, Options> OptionsFunction::GetOptions(lua_State *state, const Options& cmdLineOptions)
^
;
11 errors generated.
make[2]: *** [CMakeFiles/aoflagger-lib.dir/lua/optionsfunction.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/aoflagger-lib.dir/all] Error 2
make: *** [all] Error 2
If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/mwatelescope/homebrew-tap/issues
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.
Error: Process completed with exit code 1.
This happens in both macOS 10.15.7 (19H1824) and 11.6.4 (20G417)
mwalib 0.14.* changes cable_delays_applied from bool to enum
cable_delays_applied will have 3 options:
- No = 0
- CableAndRecClock = 1,
- CableAndRecClockAndBeamformerDipoleDelays = 2
For Birli's purposes, the existing logic of cable_delays_applied == True is now the equivalent of cable_delays_applied == 1 || 2
more helpful error messages for legacy correlator issues.
instead of NAXIS2 in first gpubox image HDU 32 does not match expected value 128 (metafits fine chans per coarse [128])
it should point to the correct wiki article on the issue. https://mwatelescope.atlassian.net/wiki/spaces/MP/pages/24969295/Legacy+Correlator+Issues#LegacyCorrelatorIssues-NAXIS2Mismatch
steps to reproduce
birli *_gpubox??_*.fits -m 1221838184.metafits -u 1221838184_birli.uvfits
The error message produced when providing an invalid averaging value could be clearer
This is a placeholder so that I don't forget.
An invalid averaging value was provided through asvo, and the error message just said something like "error parsing cli", but not which specific argument was at fault.
mwaf file header keys
MWA legacy observation 1090008640 has Cotter flag files (a.k.a. mwaf files) with the key "GPSTIME = 1090008640". Birli currently produces mwaf files that also have this GPSTIME. However, this time does not correspond to any of the gpubox HDU GPS times. Greg believes that this is a rare example of gpubox HDUs being written at times at an offset to the scheduled start time. Obviously if we can't tell what the first timestep of the flags was derived from, there's an ambiguity on what the flags should be applied to.
I don't believe there has been much formalisation on the mwaf file format, so this is an opportunity to tighten things.
I think there are two paths to take with GPSTIME - leave it as the obsid, or make it the first GPS timestamp of the contained flags (we cannot assume that all the flags in an mwaf file are derived from all the available data for that observation). I am in favour of the second option, as having an OBSID key is clearer. On the other hand, GPSSTART (or similar) would also be clear for when the flags start. Maybe we can have both and leave GPSTIME has a historic quirk.
I've looked at the RTS source and it does not use GPSTIME (reads it, but does not use it). I think the RTS assumes that there are enough flags for all data timesteps, and it assumes the first flag timestep corresponds to the first data timestep. So I don't think we need to consider the RTS here; there should be a lot more rigour around what it's doing.
Finally, RTS can take advantage of keys structured "REFLG_{:02}" (e.g. REFLG_00) to indicate flags that have a very high occupancy (e.g. in this case, channel 0 is flagged say >80%). Obviously the threshold is a variable, but the default for RTS utilities is 80%, and I think that's sensible. It would be nice to have these automatically included in Birli mwaf files.
Birli creates multiple files when middle coarse channels are missing
Before picket fence, Birli would create a single file, and flag any missing coarse channels. Now, multiple files are created instead.
No distinction is made between:
- an observation where a non-contiguous band of coarse channels were selected in scheduling
- an observation where a contiguous band of coarse channels were selected in scheduling, but one of the middle channels was not provided to Birli.
Birli should detect when the scheduled coarse channels are contiguous, and only create multiple files when an actual picket fence observation was scheduled.
Bad error message on data without "common good" coarse channels
i.e.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NoCommonTimesteps { hdu_info: "{1466188255000: {1: (0, 1), 2: (0, 1), 3: (0, 1), 4: (0, 1), 5: (0, 1), 7: (0, 1), 8: (0, 1), 9: (0, 1), 10: (0, 1), 11: (0, 1), 12: (0, 1), 13: (0, 1), 14: (0, 1), 15: (0, 1), 16: (0, 1), 17: (0, 1), 18: (0, 1), 19: (0, 1), 20: (0, 1), 21: (0, 1), 22: (0, 1), 23: (0, 1), 24: (0, 1)}, 1466188256000: {6: (0, 1)}, 1466188257000:
(I assume Birli attempted to preprocess the obsid into a ms/uvfits.) This happens on obsid 1150224192 (I have the data on /astro).
Ideally, Birli shouldn't have any unwraps; all instances of unrecoverable errors should emit a human-readable error message instead. I'm happy to offer experience here as I've tried to be thorough with hyperdrive. The presence of unwrap here is ultimately responsible for the bad error message.
Bad measurement sets being written out
Natasha has reported that she's seeing problems with some measurement sets created by Birli. I believe that she was trying to apply a calibration solution using Casacore and ran into
what(): FilebufIO::readBlock - incorrect number of bytes read for file /astro/mwasci/nhurleywalker/Epoch0032/1338206432/1338206432.ms/table.f0i
This was seen with the following obsids
1338205248
1338205544
1338206432
1338212768
1338217088
1338217976
I have recreated the measurement set and left it in /astro/mwaops/asvo/577447 for somebody to investigate the issue further.
Make use of new mwalib >=v1.* exposed metafits attributes
Build with mwalib >=1.0.0 and use the following newly exposed metafits values:
- Rfinput::rec_type which allows Birli to differentiate between the different receiver types (may or may not be useful right now)
- MetafitsContext::oversampled will allow Birli to alter the passband gains applied to each coarse channel as OS data will have very different values.
- MetafitsContext::deripple_applied and MetafitsContext::deripple_param will allow Birli to once again alter the passband gains applied to each coarse channel as the deripple will have very different values.
uvfits should have information about when it was created.
it is not possible to determine the date that a uvfits file was created without metadata from the filesystem. This metadata can be lost when transferring between filesystems like when uploading a file to acacia. It would be good to store the date a uvfits file was created in the header somewhere.
AIPS117 has a DATE-MAP field for this purpose. The marlu::context::History object could be extended to take an Option<Epoch> (defaults to SystemTime::now()) to facilitate this.
Cable lengths and geometry corrections are applied to visibilities in single precision
As with the digital gains correction, these should be done in double precision. It could be a bit of work if it breaks tests.
Also this code could be better by not allocating:
Lines 133 to 140 in 78fcebb
Add static features that hook into e.g. mwa_rust_core
I can do this myself, will only take 2 min, but just making an issue to remind myself.
Check output filename extensions before writing
I did something a little silly, but it exposed a poor error message:
# 1090008640.ms
birli *gpubox* -m *.metafits -u 1090008640.ms
# prints out
thread 'main' panicked at 'couldn't initialise uvfits writer: IO(Os { code: 21, kind: IsADirectory, message: "Is a directory" })', src/main.rs:1036:14(I'm using the v0.5.0 branch and it failed on the line containing .expect("couldn't initialise uvfits writer"))
This error happens because the "file" already exists, and Rust notes that this is a directory when trying to initialise the output as a file. It would be good to have existing files checked and handled in this case, but also user input filenames checked; .ms files write as MS, .uvfits write as uvfits, etc.
Polarization convention and IAU compliance
- Birli uses the same polarization convention as Cotter for backwards compatibility.
- The Aips117 memo is ambiguous about which polarization convention to use.
- This convention appears to currently be X as EW and Y as NS [Citation needed, it would be good to show some evidence here]
- The IAU convention appears to be the opposite, X as NS and Y as EW [Note: it would be good to point to where speicifically it says this]
This is contentious topic and there are arguments for both staying consistent, and switching to the "correct" convention. Birli should be able to accommodate both options.
The plan is to:
- introduce an optional command line flag that will flip from the polarisation convention
- give people some time to play with this flag and provide feedback on the implications of switching convention
- if the benefits outweigh the costs, then switch the default convention to IAU
Missing mandatory keywords in FITS headers.
Thanks for the hard work getting MWAX up and running. This package is really nicely done.
I work on pyuvdata and I noticed some mandatory keywords from AIPS 117 are missing from birli UVFITS files. Our software is pretty flexible reading in different interferometric datasets but the "FRAME" keyword in particular in the "AIPS AN" table is important to know which frame the antenna positions are recorded in. I've made some downstream issues to address this on read (assuming it is "ITRF" like previous MWA files), but though I would make you all aware as well.
Option to phase to the tile beam center rather than raphase/decphase if no phasing done in MWAX
Many if not all of the EoR analyses that use uvfits as a starting place have historically wanted uvfits to be phased to the tile beam center NOT the raphase/decphase. It would be good to have this as an easily set option, rather than only happening if those keywords do not exist.
can't supply a negative phase centre coordinate.
a negative value can't be supplied to the --phase-centre cli argument:
--phase-centre <RA> <DEC> Override Phase centre from metafits (degrees)
e.g.
Args { inner: ["/opt/cargo/bin/birli", "--max-memory", "300", "--avg-freq-res", "40", "--avg-time-res", "4", "--flag-edge-width", "80", "--phase-centre", "220", "-45"
error: Found argument '-4' which wasn't expected, or isn't valid in this context
If you tried to supply `-4` as a value rather than a flag, use `-- -4`
adding the -- doesn't help.
birli --metafits 1352206016.metafits --dry-run --phase-centre -- -1 -1 1352206016_20221111124638_ch151_000.fits
and adding 360 to the negative degree values gives different results in the uvfits file, compared to just using the pointing centre.
birli --metafits 1352206016.metafits --phase-centre 358.9233 333.1709 1352206016_20221111124638_ch151_000.fits -u pctest.phase.uvfits
# Phase centre: (358.9233°, 333.1709°) => (23h55m41.5919s, 255d10m15.2400s)
# Pointing centre: (358.9233°, -26.8291°) => (23h55m41.5838s, -26d49m44.7774s)
python -c 'from astropy.io import fits; fits.open('pctest.phase.uvfits')[0].header
# OBSRA = 358.9233
# OBSDEC = 333.1709
birli --metafits 1352206016.metafits --pointing-centre 1352206016_20221111124638_ch151_000.fits -u pctest.point.uvfits
python -c 'from astropy.io import fits; fits.open('pctest.point.uvfits')[0].header
# OBSRA = 358.923265883196
# OBSDEC = -26.8291048533211
Conveniently list fine-channel flags
For measurement sets, can we make a new MS table or table row detailing flagged fine channels? Not sure if uvfits has something for this purpose too.
Preprocessing Errors could be more granular
Hyperdrive needs to show different information to the user depending on what stage of preprocessing failed.
flaky assertion in scrunching can sometimes fail
+ which hyperdrive
/pawsey/mwa/software/python3/hyperdrive/v0.2.0-alpha.11/bin/hyperdrive
+ DATADIR=/astro/mwavcs/asvo/253386
+ CAL_METAFITS=/astro/mwavcs/asvo/253386/1331134096.metafits
+ SRCLIST=/pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt
+ [[ ! -r srclist_1000.yaml ]]
+ hyperdrive srclist-by-beam -n 1000 -m /astro/mwavcs/asvo/253386/1331134096.metafits /pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt srclist_1000.yaml
2022-03-16 12:00:12 INFO hyperdrive srclist-by-beam 0.2.0-alpha.11
2022-03-16 12:00:12 INFO Compiled on git commit hash: 0abc675898de8b8302f646e82edeb6e12647db02
2022-03-16 12:00:12 INFO git head ref: refs/heads/main
2022-03-16 12:00:12 INFO Wed, 09 Mar 2022 05:55:10 +0000
2022-03-16 12:00:12 INFO with compiler rustc 1.59.0 (9d1b2106e 2022-02-23)
2022-03-16 12:00:12 INFO
2022-03-16 12:00:15 INFO Successfully read /pawsey/mwa/software/python3/srclists/master/srclist_pumav3_EoR0aegean_fixedEoR1pietro+ForA_phase1+2.txt as a rts-style source list
2022-03-16 12:00:15 INFO 306894 points, 1182 gaussians, 62 shapelets
2022-03-16 12:00:22 INFO Wrote hyperdrive-style source list to srclist_1000.yaml
+ hyperdrive di-calibrate -s srclist_1000.yaml -d /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch109_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch110_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch111_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch112_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch113_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch114_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch115_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch116_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch117_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch118_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch119_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch120_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch121_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch122_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch123_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch124_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch125_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch126_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch127_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch128_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch129_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch130_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch131_000.fits /astro/mwavcs/asvo/253386/1331134096_20220312152758_ch132_000.fits /astro/mwavcs/asvo/253386/1331134096.metafits
2022-03-16 12:00:22 INFO hyperdrive di-calibrate 0.2.0-alpha.11
2022-03-16 12:00:22 INFO Compiled on git commit hash: 0abc675898de8b8302f646e82edeb6e12647db02
2022-03-16 12:00:22 INFO git head ref: refs/heads/main
2022-03-16 12:00:22 INFO Wed, 09 Mar 2022 05:55:10 +0000
2022-03-16 12:00:22 INFO with compiler rustc 1.59.0 (9d1b2106e 2022-02-23)
2022-03-16 12:00:22 INFO
2022-03-16 12:00:41 INFO Geometric delays will be applied
2022-03-16 12:00:41 INFO Calibrating obsid 1331134096
2022-03-16 12:00:41 INFO Using metafits: /astro/mwavcs/asvo/253386/1331134096.metafits
2022-03-16 12:00:41 INFO Using 24 gpubox files
2022-03-16 12:00:41 INFO Using 'Jake Jones' PFB gains
2022-03-16 12:00:41 WARN No mwaf flag files supplied
2022-03-16 12:00:43 INFO Using observation centroid frequency 154.235 MHz to convert lambdas to metres
2022-03-16 12:00:43 INFO Minimum UVW cutoff: 50λ (97.187m)
2022-03-16 12:00:43 INFO Array longitude, latitude: (116.6708°, -26.7033°)
2022-03-16 12:00:43 INFO Array latitude (J2000): -26.5901°
2022-03-16 12:00:43 INFO Phase centre (J2000): (139.5240°, -12.0956°)
2022-03-16 12:00:43 INFO Pointing centre: (137.3917°, -11.1220°)
2022-03-16 12:00:43 INFO LMST of first timestep: 158.9366°
2022-03-16 12:00:43 INFO LMST of first timestep (J2000): 158.6785°
2022-03-16 12:00:43 INFO Total number of tiles: 128
2022-03-16 12:00:43 INFO Number of unflagged tiles: 110
2022-03-16 12:00:43 INFO Tile flags: [12, 13, 39, 41, 48, 76, 79, 86, 98, 100, 104, 106, 110, 111, 112, 115, 122, 125]
2022-03-16 12:00:43 INFO Available timesteps: [0..240)
2022-03-16 12:00:43 INFO Unflagged timesteps: [6..240)
2022-03-16 12:00:43 INFO Using timesteps: [6..240)
2022-03-16 12:00:43 INFO First timestep (GPS): 1331134099.25
2022-03-16 12:00:43 INFO Last timestep (GPS): 1331134215.75
2022-03-16 12:00:43 INFO Input data time resolution: 0.50 seconds
2022-03-16 12:00:43 INFO Input data freq. resolution: 10.00 kHz
2022-03-16 12:00:43 INFO Total number of fine channels: 3072
2022-03-16 12:00:43 INFO Number of unflagged fine channels: 2688
2022-03-16 12:00:43 INFO Input data's fine-channel flags per coarse channel: [0, 1, 2, 3, 4, 5, 6, 7, 120, 121, 122, 123, 124, 125, 126, 127]
2022-03-16 12:00:43 INFO First unflagged fine-channel frequency: 138.96 MHz
2022-03-16 12:00:43 INFO Last unflagged fine-channel frequency: 169.51 MHz
2022-03-16 12:00:43 INFO Averaging 234 timesteps into each timeblock
2022-03-16 12:00:43 INFO Averaging 1 fine-freq. channels into each chanblock
2022-03-16 12:00:43 INFO Number of calibration timeblocks: 1
2022-03-16 12:00:43 INFO Number of calibration chanblocks: 2688
2022-03-16 12:00:43 INFO Using 1000 sources with a total of 1001 components
2022-03-16 12:00:43 INFO 1001 point components
2022-03-16 12:00:43 INFO 0 Gaussian components
2022-03-16 12:00:43 INFO 0 shapelet components
2022-03-16 12:00:43 INFO Writing calibration solutions to: hyperdrive_solutions.fits
2022-03-16 12:00:43 INFO Generating sky model visibilities on the GPU (double precision)
2022-03-16 12:02:23 INFO Reading input data and sky modelling
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
left: `(0, 1.0000000000000004)`,
right: `(0, 1.0)`', /pawsey/mwa/software/python3/rust/cargo/registry/src/github.com-1ecc6299db9ec823/birli-0.5.1/src/corrections.rs:617:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
/var/spool/slurm/job2149668/slurm_script: line 43: 11890 Aborted (core dumped) hyperdrive di-calibrate -s srclist_1000.yaml -d ${DATADIR}/*.fits ${CAL_METAFITS}
Non-contiguous coarse bands could be better supported.
Currently the only way to process non-contiguous coarse band observations in Birli is to separately process each contiguous group of coarse bands. The uvfits spec does allow for a FQ table which stores information about spectral windows, and pyuvdata does support reading this, so it would be testable.
Looking for an owner for this ticket, so if there is anyone out there who particularly wants to have all the channels of their picket-fence MWAX observations in one file, please let me know. Cotter never did this, so it seems like people doing these obs must have workarounds.
Additional discussion in #9
UVFits centre frequency (of centre fine chan) is not worked out correctly
Currently, UvfitsWriter::from_mwalib uses context.metafits_context.centre_freq_hz for the centre_freq_hz (CRVAL4) value. This is not going to be correct in many instances, since metafits_context.centre_freq_hz in mwalib is just the FREQCENT key from the metafits file. THAT value is the centre frequency (not nececsarily the centre of the centre fine channel) assuming all 24 (or N in the future) coarse channels of the observation.
When running Birli against an observation that only had data files for 12 of the 24 coarse channels, this resulted in the UVFITS CRVAL4 value being incorrect.
The fix should be to use the mwalib_coarse_chan_range and/or the mwalib fine_chan_freq_hz vector to determine the centre fine channel frequency.
it is unclear from the AIPS117 Spec whether DATE-OBS, RDATE keywords should include time
obs: http://ws.mwatelescope.org/observation/obs/?obs_id=1321012576
uvfits and logs: /astro/mwaeor/asvo/252163
raw files: /astro/mwaeor/asvo/252220
DATE-OBS in cotter is '2021-11-15T00:00:00.0'
DATE-OBS in metafits is '2017-12-01T14:54:38' / [UT] Date and time of observation
DATE-OBS in AIPS117 is Obs start date YYYY-MM-DD
fitsheader 1321012576.uvfits
# HDU 0 in 1321012576.uvfits:
SIMPLE = T / file does conform to FITS standard
BITPIX = -32 / number of bits per data pixel
NAXIS = 6 / number of data axes
NAXIS1 = 0 / length of data axis 1
NAXIS2 = 3 / length of data axis 2
NAXIS3 = 4 / length of data axis 3
NAXIS4 = 3072 / length of data axis 4
NAXIS5 = 1 / length of data axis 5
NAXIS6 = 1 / length of data axis 6
EXTEND = T / FITS dataset may contain extensions
GROUPS = T / random group records are present
PCOUNT = 5 / number of random group parameters
GCOUNT = 495360 / number of random groups
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
BSCALE = 1.000000000E+00
PTYPE1 = 'UU '
PSCAL1 = 1.000000000E+00
PZERO1 = 0.000000000E+00
PTYPE2 = 'VV '
PSCAL2 = 1.000000000E+00
PZERO2 = 0.000000000E+00
PTYPE3 = 'WW '
PSCAL3 = 1.000000000E+00
PZERO3 = 0.000000000E+00
PTYPE4 = 'BASELINE'
PSCAL4 = 1.000000000E+00
PZERO4 = 0.000000000E+00
PTYPE5 = 'DATE '
PSCAL5 = 1.000000000E+00
PZERO5 = 2.459533500E+06
DATE-OBS= '2021-11-15T00:00:00.0'
CTYPE2 = 'COMPLEX '
CRVAL2 = 1.000000000E+00
CRPIX2 = 1.000000000E+00
CDELT2 = 1.000000000E+00
CTYPE3 = 'STOKES '
CRVAL3 = -5
CDELT3 = -1
CRPIX3 = 1.000000000E+00
CTYPE4 = 'FREQ '
CRVAL4 = 9.792000000E+07
CDELT4 = 1.000000000E+04
CRPIX4 = 1537
CTYPE5 = 'RA '
CRVAL5 = 3.504156604E+02
CDELT5 = 1
CRPIX5 = 1
CTYPE6 = 'DEC '
CRVAL6 = -2.682145090E+01
CDELT6 = 1
CRPIX6 = 1
OBSRA = 3.504156604E+02
OBSDEC = -2.682145090E+01
EPOCH = 2.000000000E+03
OBJECT = 'SpaceFest_AzEl0.0,90.0_Ch77_28_of_60'
TELESCOP= 'MWA '
INSTRUME= 'MWA '
HISTORY AIPS WTSCAL = 1.0
COMMENT Created by birli v0.1.9
SOFTWARE= 'birli '
GITLABEL= 'v0.1.9 '
# HDU 1 in 1321012576.uvfits:
XTENSION= 'BINTABLE' / binary table extension
BITPIX = 8 / 8-bit bytes
NAXIS = 2 / 2-dimensional binary table
NAXIS1 = 78 / width of table in bytes
NAXIS2 = 128 / number of rows in table
PCOUNT = 0 / size of special data area
GCOUNT = 1 / one data group (required keyword)
TFIELDS = 11 / number of fields in each row
TTYPE1 = 'ANNAME ' / label for field 1
TFORM1 = '8A ' / data format of field: ASCII Character
TTYPE2 = 'STABXYZ ' / label for field 2
TFORM2 = '3D ' / data format of field: 8-byte DOUBLE
TUNIT2 = 'METERS ' / physical unit of field
TTYPE3 = 'NOSTA ' / label for field 3
TFORM3 = '1J ' / data format of field: 4-byte INTEGER
TTYPE4 = 'MNTSTA ' / label for field 4
TFORM4 = '1J ' / data format of field: 4-byte INTEGER
TTYPE5 = 'STAXOF ' / label for field 5
TFORM5 = '1E ' / data format of field: 4-byte REAL
TUNIT5 = 'METERS ' / physical unit of field
TTYPE6 = 'POLTYA ' / label for field 6
TFORM6 = '1A ' / data format of field: ASCII Character
TTYPE7 = 'POLAA ' / label for field 7
TFORM7 = '1E ' / data format of field: 4-byte REAL
TUNIT7 = 'DEGREES ' / physical unit of field
TTYPE8 = 'POLCALA ' / label for field 8
TFORM8 = '3E ' / data format of field: 4-byte REAL
TTYPE9 = 'POLTYB ' / label for field 9
TFORM9 = '1A ' / data format of field: ASCII Character
TTYPE10 = 'POLAB ' / label for field 10
TFORM10 = '1E ' / data format of field: 4-byte REAL
TUNIT10 = 'DEGREES ' / physical unit of field
TTYPE11 = 'POLCALB ' / label for field 11
TFORM11 = '3E ' / data format of field: 4-byte REAL
EXTNAME = 'AIPS AN ' / name of this binary table extension
ARRAYX = -2.559454079E+06
ARRAYY = 5.095372144E+06
ARRAYZ = -2.849057185E+06
FREQ = 9.792000000E+07
GSTIA0 = 5.430044686E+01
DEGPDY = 3.609850000E+02
RDATE = '2021-11-15T00:00:00.0'
POLARX = 0.000000000E+00
POLARY = 0.000000000E+00
UT1UTC = 0.000000000E+00
DATUTC = 0.000000000E+00
TIMSYS = 'UTC '
ARRNAM = 'MWA '
NUMORB = 0
NOPCAL = 3
FREQID = -1
IATUTC = 3.300000000E+01
XYZHAND = 'RIGHT ' fitsheader 1321012576.uvfits
# HDU 0 in 1321012576.uvfits:
SIMPLE = T / file does conform to FITS standard
BITPIX = -32 / number of bits per data pixel
NAXIS = 6 / number of data axes
NAXIS1 = 0 / length of data axis 1
NAXIS2 = 3 / length of data axis 2
NAXIS3 = 4 / length of data axis 3
NAXIS4 = 3072 / length of data axis 4
NAXIS5 = 1 / length of data axis 5
NAXIS6 = 1 / length of data axis 6
EXTEND = T / FITS dataset may contain extensions
GROUPS = T / random group records are present
PCOUNT = 5 / number of random group parameters
GCOUNT = 495360 / number of random groups
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
BSCALE = 1.000000000E+00
PTYPE1 = 'UU '
PSCAL1 = 1.000000000E+00
PZERO1 = 0.000000000E+00
PTYPE2 = 'VV '
PSCAL2 = 1.000000000E+00
PZERO2 = 0.000000000E+00
PTYPE3 = 'WW '
PSCAL3 = 1.000000000E+00
PZERO3 = 0.000000000E+00
PTYPE4 = 'BASELINE'
PSCAL4 = 1.000000000E+00
PZERO4 = 0.000000000E+00
PTYPE5 = 'DATE '
PSCAL5 = 1.000000000E+00
PZERO5 = 2.459533500E+06
DATE-OBS= '2021-11-15T00:00:00.0'
CTYPE2 = 'COMPLEX '
CRVAL2 = 1.000000000E+00
CRPIX2 = 1.000000000E+00
CDELT2 = 1.000000000E+00
CTYPE3 = 'STOKES '
CRVAL3 = -5
CDELT3 = -1
CRPIX3 = 1.000000000E+00
CTYPE4 = 'FREQ '
CRVAL4 = 9.792000000E+07
CDELT4 = 1.000000000E+04
CRPIX4 = 1537
CTYPE5 = 'RA '
CRVAL5 = 3.504156604E+02
CDELT5 = 1
CRPIX5 = 1
CTYPE6 = 'DEC '
CRVAL6 = -2.682145090E+01
CDELT6 = 1
CRPIX6 = 1
OBSRA = 3.504156604E+02
OBSDEC = -2.682145090E+01
EPOCH = 2.000000000E+03
OBJECT = 'SpaceFest_AzEl0.0,90.0_Ch77_28_of_60'
TELESCOP= 'MWA '
INSTRUME= 'MWA '
HISTORY AIPS WTSCAL = 1.0
COMMENT Created by birli v0.1.9
SOFTWARE= 'birli '
GITLABEL= 'v0.1.9 '
# HDU 1 in 1321012576.uvfits:
XTENSION= 'BINTABLE' / binary table extension
BITPIX = 8 / 8-bit bytes
NAXIS = 2 / 2-dimensional binary table
NAXIS1 = 78 / width of table in bytes
NAXIS2 = 128 / number of rows in table
PCOUNT = 0 / size of special data area
GCOUNT = 1 / one data group (required keyword)
TFIELDS = 11 / number of fields in each row
TTYPE1 = 'ANNAME ' / label for field 1
TFORM1 = '8A ' / data format of field: ASCII Character
TTYPE2 = 'STABXYZ ' / label for field 2
TFORM2 = '3D ' / data format of field: 8-byte DOUBLE
TUNIT2 = 'METERS ' / physical unit of field
TTYPE3 = 'NOSTA ' / label for field 3
TFORM3 = '1J ' / data format of field: 4-byte INTEGER
TTYPE4 = 'MNTSTA ' / label for field 4
TFORM4 = '1J ' / data format of field: 4-byte INTEGER
TTYPE5 = 'STAXOF ' / label for field 5
TFORM5 = '1E ' / data format of field: 4-byte REAL
TUNIT5 = 'METERS ' / physical unit of field
TTYPE6 = 'POLTYA ' / label for field 6
TFORM6 = '1A ' / data format of field: ASCII Character
TTYPE7 = 'POLAA ' / label for field 7
TFORM7 = '1E ' / data format of field: 4-byte REAL
TUNIT7 = 'DEGREES ' / physical unit of field
TTYPE8 = 'POLCALA ' / label for field 8
TFORM8 = '3E ' / data format of field: 4-byte REAL
TTYPE9 = 'POLTYB ' / label for field 9
TFORM9 = '1A ' / data format of field: ASCII Character
TTYPE10 = 'POLAB ' / label for field 10
TFORM10 = '1E ' / data format of field: 4-byte REAL
TUNIT10 = 'DEGREES ' / physical unit of field
TTYPE11 = 'POLCALB ' / label for field 11
TFORM11 = '3E ' / data format of field: 4-byte REAL
EXTNAME = 'AIPS AN ' / name of this binary table extension
ARRAYX = -2.559454079E+06
ARRAYY = 5.095372144E+06
ARRAYZ = -2.849057185E+06
FREQ = 9.792000000E+07
GSTIA0 = 5.430044686E+01
DEGPDY = 3.609850000E+02
RDATE = '2021-11-15T00:00:00.0'
POLARX = 0.000000000E+00
POLARY = 0.000000000E+00
UT1UTC = 0.000000000E+00
DATUTC = 0.000000000E+00
TIMSYS = 'UTC '
ARRNAM = 'MWA '
NUMORB = 0
NOPCAL = 3
FREQID = -1
IATUTC = 3.300000000E+01
XYZHAND = 'RIGHT ' Option to suppress stdout progress bars when using Birli as a library
** Low Priority! **
When using Birli as a library, some methods, like context_to_jones_array output progress information to stdout. It would be nice to be able to suppress this output.
e.g. this is output from my binary which uses the logging crate (the log lines include dates and times, etc) but is then interspersed with std out of progress bars which I would prefer I did not see.
...
[2021-11-11T07:00:28Z DEBUG mwax_stats::fringes] Generating jones array...
coarse_chan 012 : [=========================================================================================================================================================================] 1 /1
loading hdus : [00:00:00] [===========================================================================================================================================================] 100% (0s )
[2021-11-11T07:00:28Z DEBUG mwax_stats::fringes] Jones array shape (timesteps, fine_chans, baselines)[1, 128, 8256]
...
Measurement set files written with old timestamp
This causes users who have Birli generated measurement sets located on filesystems with purge policies to "lose" files in the following MS directories:
- FLAG_CMD
- POINTING
- PROCESSOR
- STATE
Which causes CASA to throw errors.
The files in those MS dirs are all timestamped Sep 22 2021.
Possible solution: touch all the files as Birli writes them?
can't apply calibration solutions to partial observation
[2022-02-11T07:53:27Z DEBUG birli] args:
Args { inner: ["/opt/cargo/bin/birli", "--max-memory", "300", "--apply-di-cal", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/calibration.bin", "-M", "/nvmetmp/1276488456.ms", "-m", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456.metafits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox01_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox02_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox03_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox04_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox05_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox06_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox07_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox09_00.fits", "/astro/mwaasvo/mwaservice/asvo/prod/jobs/556842/raw/1276488456_20200618040752_gpubox10_00.fits"] }
channels
[2022-02-11T07:53:28Z INFO birli] Coarse channel details (metafits=24, provided=9, common=9, good=9, select=10, flag=15):
gpu corr rec cen [MHz] p c g s f
cc0: 24 23 157 200.9600 f
cc1: 23 22 158 202.2400 f
cc2: 22 21 159 203.5200 f
cc3: 21 20 160 204.8000 f
cc4: 20 19 161 206.0800 f
cc5: 19 18 162 207.3600 f
cc6: 18 17 163 208.6400 f
cc7: 17 16 164 209.9200 f
cc8: 16 15 165 211.2000 f
cc9: 15 14 166 212.4800 f
cc10: 14 13 167 213.7600 f
cc11: 13 12 168 215.0400 f
cc12: 12 11 169 216.3200 f
cc13: 11 10 170 217.6000 f
cc14: 10 9 171 218.8800 p c g s
cc15: 9 8 172 220.1600 p c g s
cc16: 8 7 173 221.4400 s f
cc17: 7 6 174 222.7200 p c g s
cc18: 6 5 175 224.0000 p c g s
cc19: 5 4 176 225.2800 p c g s
cc20: 4 3 177 226.5600 p c g s
cc21: 3 2 178 227.8400 p c g s
cc22: 2 1 179 229.1200 p c g s
cc23: 1 0 180 230.4000 p c g s
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ChannelSizeMismatch { calsol_chans: 3072, data_chans: 1280 }', src/main.rs:1218:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
srun: error: mwa073: task 0: Exited with exit code 101
Optimisations in correction code
There's some "low-hanging fruit" in the correction code. It's hard to know if changing these will make a difference, because the compiler might be smart enough already, but we should fix these anyway, and do a benchmark.
Line 130 in 64c35b5
Line 139 in 64c35b5
Line 296 in 64c35b5
Note that .clone() on the Range structs remains necessary, but those are very cheap. Otherwise .clone() should be avoided as much as possible.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.