ymaan4 / rficlean

Mitigation of periodic as well as narrow-band and spiky/bursty RFI from time-domain filterbank data.

RFIClean

Mitigation of periodic and spiky RFI from filterbank data.

RFIClean excises periodic RFI (broadband as well as narrow-band) in the Fourier domain, and then mitigates narrow-band spectral line RFI as well as broadband bursty time-domain RFI using robust statistics. Currently, RFIClean anticipates the input data either in SIGPROC filterbank format or the GMRT's pulsar data format.

RFIClean was primarily designed to efficiently search and mitigate periodic RFI from GMRT time-domain data. Over the time, RFIClean has evolved to mitigate any spiky (in time or frequency) RFI as well, and from any SIGPROC filterbank format data file. It is written primarily in C. For handling the filterbank format I/O, RFIClean uses several modules from Duncan Lorimer's SIGPROC (https://sourceforge.net/p/sigproc/wiki/Home/; thank you Dunc) which are included here in the src/ext/ folder (some of these codes are also modified suitably).

Installation:

• For compiling RFIClean, a Makefile is included in the package.
• If you want to install the executable at a location other than RFIClean/bin/, then change MYBIN in the Makefile accordingly.
• RFIClean has the following dependencies: FFTW3 and PGPLOT. If these are not included in the regular library paths then amend LIBS in the Makefile accordingly.
• To compile, run make. For installing the executable in your favourite location, run make install.
• Once installed, use rficlean -h or rficlean --help or just rficlean to see the usage information.

Bash-script based parallel processing

For faster processing, use the bash script crp_rficlean_fil.sh in RFIClean/bin/ that uses RFIClean on different parts of a single data file simultaneously and then combines the output products at the end --- parallel processing in a rather crude but very efficient way.

To see the usage information, just execute the script without any command line arguments. The flags to be used with rficlean while using this script are picked up from a simple text file containing two rows: the first one indicating the block size and the second containing any other flags. For an example of this text file, see RFIClean/bin/rficlean.flags.

Diagnostic plot

RFIClean produces a diagnostic plot showing which Fourier frequencies are mitigated from the data and how frequently. The plot data are also output in a file.

RFIClean paper

Details of the methods used in RFIClean as well as some of the early scientific contributions of RFIClean can be found in this paper.

If you find RFIClean useful, it would be great if you could cite the following paper that describes the underlying method in detail: https://arxiv.org/abs/2012.11630 . Thanks!

Yogesh Maan <ymaan4[@]gmail.com>

rficlean's Issues

Hardcoded paths in crp_rficlean_gm.sh

The absolute path of the rficlean executable (/home/ymaan/bin/rficlean) is hardcoded in crp_rficlean_gm.sh.
crp_rficlean_gm.sh fails in a new installation due to this.

Fails with filterbanks by written by Your

RFIClean exits with error ERROR: ERROR: Input data to rficlean is not in filterbank format
when run on filterbanks written by Your.
I'm not sure where the problem is, the same files with headers made in alternative ways run fine.
My guess is there is some strange interaction with how Your headers are written and RFIClean's header parser, since
Sigproc's header and Presto's readfile work are able to read Your headers.

I've attached
small_fil.tar.gz
a small filterbank that reproduces the issue.

Ignoring channels in statistics

Hello!
We are trying to use this for CHIME/Pulsar filterbank style data and was wondering if there was a feature similar to -ignorechan from rfifind. We have many channels that have been 0'd out prior to rficlean.
I noticed that using the -pcl flag makes rficlean take the first channel and use the statistics on that, which is not desirable as that could be one of the zeroed out channels.

Incorrect size allocation

In code rficlean_data.c, the following
cblock=(unsigned char *) malloc(nsblk*sizeof(unsigned short));
should actually be
cblock=(unsigned char *) malloc(nsblk*sizeof(unsigned char));

Installation instructions

Can you please add a few lines about how to install and configure the package in the README file?