keck-datareductionpipelines / kcwidrp

KCWI data reduction pipeline. Developers: Don Neill, Matt Matuszewski, Chris Martin (Caltech)

kderp

KCWI Data Extraction and Reduction Pipeline

See AAAREADME for details

kcwidrp's Issues

Clarification on Sky Subtraction

Hello, I am a little confused about how exactly or whether to run the Stage5sky step. I have observed three difference science targets whose extended emission warranted separate sky frames for each. According to the AAAREADME file, I would need to input said sky frames as the sky master in the kcwi.proc file. (I presume these would be the *_intf.fits versions, or the *_sky.fits files? Would the latter be generated if stage5 wasn't run?) But more importantly, the readme file also said that this step should be skipped if the sky is not well sampled in the frame or is mixed with a large fraction of object light. So should stage5 be run with the separate sky frames specified in the kcwi.proc, or not? Or would stage6cube just generate separate sky cubes if they're correctly identified in the kcwi.proc file (after stage 5 is skipped)? Thank you for any pointer in advance.

undefined function: 'List' in kcwi_stage5sky with display=2

To Whom It May Concern,

I am running into over subtraction issues, thus I am trying to run the kcwi_stage5sky with the display=2.
However, an error is popping up kctv.pro at line 344. There appears to be a missing function called
"list".

Any help is appreciated, thanks!

wavemap error when using archival filename convention

Hi again - On line 89 of MakeMasterSky.py the code appears to be reading in the original file name (but it's possible that I'm missing something):

groot = tab['OFNAME'][0].split('.fits')[0]

Is there a workaround for dealing with data downloaded from the archive (that have a different filename convention)? Here is the full traceback:

2021-06-11 11:23:35:KCWI:INFO: Looking for ARCLAMP frames
2021-06-11 11:23:35:KCWI:INFO: Reading image: kb191219_00036_wavemap.fits
[Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits'
2021-06-11 11:23:35:DRPF:ERROR: Exception [Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits' while invoking MakeMasterSky
Traceback (most recent call last):
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/core/framework.py", line 229, in execute
    action_output = pipeline.get_action(action_name)(action, context)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/pipelines/base_pipeline.py", line 84, in f
    return obj.apply()
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/primitives/base_primitive.py", line 57, in apply
    self.output = self._perform()
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/MakeMasterSky.py", line 96, in _perform
    wmf))[0]
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/kcwi_file_primitives.py", line 559, in kcwi_fits_reader
    raise e
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/kcwi_file_primitives.py", line 556, in kcwi_fits_reader
    hdul = fits.open(file)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 164, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1051, in _readfrom
    fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py", line 535, in wrapper
    return function(*args, **kwargs)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 175, in __init__
    self._open_filename(fileobj, mode, overwrite)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 564, in _open_filename
    self._file = fileobj_open(self.name, IO_FITS_MODES[mode])
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/util.py", line 397, in fileobj_open
    return open(filename, mode, buffering=0)
FileNotFoundError: [Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits'
2021-06-11 11:23:35:DRPF:INFO: Event failed: name object_make_sky, action MakeMasterSky, arg name KB.20191219.40218.fits.gz, recurr False

Calibration files not found

Kcwi_prep works once the file root is provided explicitly but everything beyond stage 1 of the pipeline runs into problems. Specifically, the DRP does not appear to create / find the necessary calibration files:

41/ 41 88 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458573.138 10.0 object 88 308.59116667 25.06380556 90.00 1.370 wolf1346
KCWI_STAGE2DARK [INFO 1 ] input reduced image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_int.fits
KCWI_DO_DARK [INFO 1 ] DS Test (drk?, msig, tsig): NO 0.1864 0.2841
KCWI_STAGE2DARK [WARNING] cannot associate with any master dark: kb190330_00088_int.fits
KCWI_STAGE2DARK [INFO 1 ] scattered light subtracted
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_mskd.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_vard.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_intd.fits

KCWI_STAGE2DARK [INFO 1 ] run time in seconds: 117.51911
KCWI_STAGE2DARK [INFO 1 ] finished on Fri Apr 12 11:57:07 2019
IDL> kcwi_stage3geom
% Compiled module: KCWI_STAGE3GEOM.
fatal: Not a git repository (or any of the parent directories): .git
KCWI_READ_PROC [INFO 1 ] Master proc file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kcwi.proc
KCWI_READ_PROC [INFO 1 ] Fri Apr 12 11:57:15 2019
KCWI_STAGE3GEOM [INFO 1 ] Number of input images: 41

1/ 41 27 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458572.590 36.7 dflat 27 58.00000000 45.00000000 0.00 1.410
KCWI_STAGE3GEOM [INFO 1 ] input 2-D image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits
KCWI_STAGE3GEOM [WARNING] missing calibration file(s) for: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits

The AAAREADME file provides no information that would help me solve these problems - or did I miss it? All my original FITS files are in the same directory, and I specified "." as the "calibration directory" during the kcwi_prep phase.

Thanks in advance!

Harald

issue with importing lacosmicx

Hi there - I was running the standard pipeline command reduce_kcwi -f KB*.fits.gz, and I received an error that _lacosmicx is not defined. Looking at the RemoveCosmicRays.py file, it seems that _lacosmicx is imported in the init method, rather than at the top of the file. I do have lacosmicx installed, and it seems to be imported fine in the init method. I was able to fix the bug by moving the import to the top of the file (where the other imports are located).

Here is the full traceback:

2021-06-11 11:12:29:DRPF:ERROR: Exception name '_lacosmicx' is not defined while invoking RemoveCosmicRays
Traceback (most recent call last):
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/core/framework.py", line 229, in execute
    action_output = pipeline.get_action(action_name)(action, context)
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/pipelines/base_pipeline.py", line 84, in f
    return obj.apply()
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/primitives/base_primitive.py", line 57, in apply
    self.output = self._perform()
  File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/RemoveCosmicRays.py", line 54, in _perform
    mask, clean = _lacosmicx.lacosmicx(
NameError: name '_lacosmicx' is not defined

Some question about the the standard star database

I recently ran the kcwi_drp for the archive data. I found standard stars that are not in the database (data/stds) of the kcwi_drp.
These standard star are observed at 2019/12/19 named BB0837p5427, BB1245p4238, and BB1417p4941.
Since these standard stars are not within the kcwi_drp database, I cannot run the final step which is kcwi_stage8std.

Is there any way to solve this problem?

Proper Motions

We were observing He3 in the standard star list and it appears to have a proper motion moving it off the slicer from the listed position. Could the coordinates be updated? It may also be useful to check other standards for proper motions.

Understanding sky-subtraction

Hi, we are wanting to skip sky subtraction because of a complicated background field, but I don't quite understand the output files. From the documentation (DATA_PRODUCTS), there should be no difference between the "_ocubes" files with or without sky-subtraction; the difference should be in the "_icubes" files; is this correct?

Also, when I do sky-subtraction, the algorithm doesn't seem as simple as "icube=ocube-scube"; I get residuals of about 1 part in 10^7 (but definitely not zero). Can you please explain?

BTW, we are NOT using N&S.

kcwi_stage6drr

Note: kcwi_stage6drr is not listed in the code flow in AAAReadme. should it be there?