uvot-data-analysis / uvot-mosaic

Create a calibrated mosaic of UVOT images

This code creates a calibrated mosaic from the individual UVOT snapshots downloaded from the archive.

The code and documentation are in active development. Please consult @lea-hagen before using this.

How to use

Required packages: astropy

Python 3 is required.

This is built around the UVOT processing tools that are part of HEASOFT/FTOOLS. Instructions to download and install HEASOFT are here: https://heasarc.gsfc.nasa.gov/lheasoft/install.html

You will also need the latest CALDB files. Download/installation information is here: https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/install.html

Installation of uvot-mosaic: Either download or clone the repository. You can keep the code wherever you like as long as it's in your python path.

Running uvot_deep.py to combine and stack images

Download the desired images from HEASARC (https://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/swift.pl) and ensure that you've chosen to download both the UVOT and Swift Auxiliary data. The downloads will be organized in folders named with the Observation ID (e.g., 00037723002), which is a combination of the target ID (00037723) and segment (002). All of the fits files will need to be unzipped (gunzip */*.gz, gunzip */*/*.gz, etc.). For all of the observations you wish to stack, put their folders in the same directory, and run uvot_deep.py from that directory.

Example: Download two observations of the edge of the M31 disk, with Obs IDs 00037723001 and 00037723002. You will have a directory structure something like

~/example/00037723001
~/example/00037723001/auxil
~/example/00037723001/uvot
~/example/00037723001/uvot/hk
~/example/00037723001/uvot/image
~/example/00037723001/uvot/products
~/example/00037723002
~/example/00037723002/auxil
~/example/00037723002/uvot
~/example/00037723002/uvot/hk
~/example/00037723002/uvot/image
~/example/00037723002/uvot/products

From ~/example, run

> import uvot_deep
> uvot_deep.uvot_deep(['00037723001','00037723002'], 'test_', ['w2','m2','w1'])

After a lot of verbosity from the UVOT tools, this will create several files in ~/example for each filter (replace ff with the filter name).

• test_ff_sk_all.fits: each extension is a counts ("sky") image, in units of counts per pixel
• test_ff_sk.fits: all extensions from test_ff_sk.fits added together
• test_ff_ex_all.fits: each extension is an exposure map, in units of seconds
• test_ff_ex.fits: all extensions from test_ff_ex.fits added together
• test_ff_cr.fits: count rate image (test_ff_sk.fits divided by test_ff_ex.fits), in counts per second per pixel

It will also create files in ~/example/[obsid]/uvot/image for each exposure, which you most likely won't need to look at. But if you're curious, this is the list, where [obsid] is the Observation ID and [ff] is the filter.

• sw[obsid]u[ff].badpix: bad pixel map
• sw[obsid]u[ff]_ex_mask.img: masked exposure map
• sw[obsid]u[ff].lss: large scale sensitivity (LSS) map
• sw[obsid]u[ff]_mask.img: mask image
• sw[obsid]u[ff]_sk_corr.img: sky (counts) image, corrected for LSS and masked
• sw[obsid]u[ff].sl: scattered light image (assuming this option is enabled)

Running offset_mosaic.py to adjust background for individual snapshots

The background values in UVOT images are known to change, likely due to scattered light from the Earth/sun/moon, but sometimes also from UV-bright sources in or near the field of view. offset_mosaic.py is being written to do offsets between snapshots to better account for this.

uvot-mosaic's Issues

code breaks when attitude file can't be updated

It appears that running uvotattcorr fails when the input attitude file doesn't have any observations with aspect corrections.

Example: for the housekeeping files 00084284007/auxil/sw00084284007sat.hk and 00084284007/uvot/hk/sw00084284007uac.hk (for M81-dwA, segment 7), here's the output from uvotattcorr:

uvotattcorr: verbose: loaded 2 records from 00084284007/uvot/hk/sw00084284007uac.hk
uvotattcorr: error: no corrections to apply
ERROR: No such process
Task uvotattcorr 1.1 terminating with status 3

I think the solution will be to check if the updated file (called att_uat) exists, and if not, skip the observation altogether.

Frame time Correction

Hi Lea,

Thanks for transferring this to python! I noticed two bugs, but since they are individual problems I will have them as two separate issues.

If the frametime is not around the 11ms standard value and the frame is included, uvotimsum will crash. I updated the script so that only 11ms frames are included, and that I copy that solution here:

def append_ext(hdu_all, new_fits_file, image_info):
    """
   append extenstions from new_fits_file to hdu_all (checking that the new
    extensions have an aspect correction and are 2x2 binned)
    Parameters
    ----------
    hdu_all : HDU object
        the HDU that we're appending to
    new_fits_file : string
        name of the fits file that has extensions to append
    
    image_info : dict
        dictionary that has the extacted info about binning and aspect correction
    Returns
    -------
    hdu_all : HDU object
        the same as the input HDU, with the new extensions appended
  """
    
    with fits.open(new_fits_file) as hdu_new:

        # if this is the first image, copy over the primary header
        if len(hdu_all) == 0:
            hdu_all.append(fits.PrimaryHDU(header=hdu_new[0].header))

        # if the binning is 2x2 and the aspect corrections are ok, append the array
        for i in range(1,len(hdu_new)):
            dict_ind = 1+i-len(hdu_new)
            if (image_info['binning'][dict_ind] == 2) & \
                ((image_info['aspect_corr'][dict_ind] == 'DIRECT') | (image_info['aspect_corr'][dict_ind] == 'UNICORR')):
                if ((image_info['frame_time'][dict_ind] > 0.01081135) & (image_info['frame_time'][dict_ind] < 0.01125265)):
                    hdu_all.append(fits.ImageHDU(data=hdu_new[i].data, header=hdu_new[i].header))


    return hdu_all

option to include small scale sensitivity (SSS)

There are maps of the tiny regions with reduced sensitivity:
https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/swift/docs/uvot/uvotcaldb_sss_01b.pdf

Should add an option to mask these.

incorporate sensitivity loss

The most recent version is here:
https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/swift/docs/uvot/uvotcaldb_throughput_04.pdf

offsetting snapshots when biweight is nan

offset_mosaic.py calculates offsets using the biweight. If too many of the pixels are close to 0 (likely because of a short exposure time), the biweight is NaN, because it's calculated from the MAD, which is 0.

The exposure time threshold was intended to avoid this issue, but I might need to forcibly delete any extensions with NaN biweight.

create documentation

Make nicer documentation with readthedocs

LSS Correction

Hi,

Here is my second correction to your code for the LSS images. Sometimes the lss image is not perfectly aligned with the sky/exp images. To work around this, I just reprojected the LSS image into the coordinates of the sky image. The snippet fixing this code is here:

Note that this correction requires the package reproject, and the following imports:

from reproject import reproject_exact
from astropy.utils.data import get_pkg_data_filename

def corr_sk(obs_folder, obs_filter):

    """
    Correct counts images for LSS and mask them

    counts_new = counts_old / lss * mask


    Parameters
    ----------
    obs_folder : string
        the 11-digit name of the folder downloaded from HEASARC

    obs_filter : string
        one of the UVOT filters ['w2','m2','w1','uu','bb','vv']
    

    Returns
    -------
    nothing

    """

    print('')
    print('  ** correcting sk images')
    print('')

    # counts image (labeled as sk)
    sk_image = obs_folder+'/uvot/image/sw'+obs_folder+'u'+obs_filter+'_sk.img'
    # LSS image
    lss_image = obs_folder+'/uvot/image/sw'+obs_folder+'u'+obs_filter+'.lss'
    # mask image
    mask_image = obs_folder+'/uvot/image/sw'+obs_folder+'u'+obs_filter+'_mask.img'

    
    with fits.open(sk_image) as hdu_sk, fits.open(lss_image) as hdu_lss, fits.open(mask_image) as hdu_mask:

        # create HDU for the new counts image
        hdu_sk_new = fits.HDUList()
        # copy over the primary header
        hdu_sk_new.append(fits.PrimaryHDU(header=hdu_sk[0].header))

        # for each image extension, make the new image
        for i in range(1,len(hdu_sk)):
            if len(hdu_lss[i].data) != len(hdu_sk[i].data) or len(hdu_lss[i].data[0]) != len(hdu_sk[i].data[0]):
                print('LSS Misaligned...')
                hd_test=fits.open(get_pkg_data_filename(sk_image))[i]
                hdu1=fits.open(get_pkg_data_filename(lss_image))[i]
                print('Aligning LSS to Sky Image')
                lss_test, footprint=reproject_exact(hdu1,hd_test.header)
                print('Aligned Images')
                new_sk_array = hdu_sk[i].data / lss_test * hdu_mask[i].data
            else:
                # divide by lss and multiply by mask
                new_sk_array = hdu_sk[i].data / hdu_lss[i].data * hdu_mask[i].data



            # remove NaNs from dividing by 0
            new_sk_array[np.isnan(new_sk_array)] = 0
 
            # append to the big fits file
            hdu_sk_new.append(fits.ImageHDU(data=new_sk_array, header=hdu_sk[i].header))

    # write out the new fits file
    sk_image_corr = obs_folder+'/uvot/image/sw'+obs_folder+'u'+obs_filter+'_sk_corr.img'
    hdu_sk_new.writeto(sk_image_corr, overwrite=True)
            

too many files open error and fix

Processing galaxies with lots of data can lead to an error when too many files are open at once. This number is set at the OS level and may need to be increased.