spacetelescope / calcos Goto Github PK

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Issue COSPIP-55 was created by Rachel Plesha:

Cenwave dependent TDS

costools/x1dcorr allows one to extract a spectrum from a corrtag file specifying the location and extrsize at run time. Rather than change the x1dcorr task to make it compatible with recent changes to calcos library functions, it was decided to add the functionality to calcos itself

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Follow up of #30 and should be done with buy-in by COS team.

• Coverage test shows how much code is actually covered in your tests.
• PEP 8 check makes your code easier to read, and thus debug, reducing cost of future maintenance.

Issue COSPIP-40 was created by Nick Indriolo:

Fix behavior where CalCOS uses "bad" NUV shifts when no shift is found

Currently there are many ways a shift of 0.0 can be returned, but very little explanation as to what particular test caused the set to 0. Need to add output each time shift is set to 0.0 so that users can tell what caused the failure.

There is a need for nightly regression tests, for instance, to identify upcoming code deprecations.

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Can CALCOS run with Astropy 2.0? There's no way to find out right now due to lack of regression tests (see #12).

Here's some calcos warnings, they come from the dq blurring code and the two-zone extraction code sections.

/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/cosutil.py:1244: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
dq_array[lower_y_s:upper_y_s,:], info["x_offset"])
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/timetag.py:2423: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
shifted_dq[wcastart:wcastop+1, column] = dq_array[wcastart:wcastop+1, column]
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/timetag.py:2434: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
shifted_dq[outstart:outstop+1, column] = dq_array[instart:instop+1, column]
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/timetag.py:2445: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
shifted_dq[outstart:outstop+1, column] = dq_array[instart:instop+1, column]

and

/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1488: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1490: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1491: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
lowerdq = e_dq_data[lowerstart:lowerstop, column]
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1495: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1497: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1498: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
upperdq = e_dq_data[upperstart:upperstop, column]
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1502: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1504: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
column].sum(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1505: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
innerdq = e_dq_data[innerstart:innerstop, column]
/user/rij/anaconda3/envs/calcos_wave_err/lib/python3.5/site-packages/calcos-3.1.9-py3.5-linux-x86_64.egg/calcos/extract.py:1508: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
outerdq = e_dq_data[outerstart:outerstop, column]

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

From PR# 89521: CalCOS Profile Background Subtraction Bug

Calcos should be able to handle the situation where one of the background
regions is outside the profile image. In fact, every time the background is
calculated there are conditional statements to cover the situation. However,
there is a minor coding oversight in the extraction routine that causes an
inevitable crash when either of the background regions extends outside the
profile image.

1. Make sure any variables using BKG_1 or BKG2 are contained in the case where
   one doesn't exist in "extract.py".

2. Fix typo in "trace.py":
   cosutil.printMsg("Unable to extract background region #2 from",
   " reference profile")

Follow up of #30. This would future-proof CALCOS software.

While working on #30, I noticed this coming from the pipeline. FYI.

.../astropy-3.1.dev22443-py3.6-linux-x86_64.egg/astropy/io/fits/hdu/table.py:345:
AstropyDeprecationWarning: The following keywords are now recognized as special column-related
attributes and should be set via the Column objects: TCRPXn, TCRVLn, TCTYPn, TCUNIn. In future,
these values will be dropped from manually specified headers automatically and replaced with values
generated based on the Column objects.
    "Column objects.".format(keys), AstropyDeprecationWarning)

During the centroid calculation, if CalCOS does not find counts in the extraction region it crashes with a TypeError:
`~/anaconda/envs/astroconda_3/lib/python3.6/site-packages/calcos/trace.py in calculateCentroidError(data_ij, goodcolumns, regions, centroid, background)
489 y = rows[:,np.newaxis]
490 sumi = (i - background).sum(axis=0, dtype=np.float64)[goodcolumns].sum(dtype=np.float64)
--> 491 sumsq = (i*(y - centroid) *
492 (y - centroid)).sum(axis=0,
493 dtype=np.float64)[goodcolumns].sum(dtype=np.float64)

TypeError: unsupported operand type(s) for -: 'int' and 'NoneType'`

Instead, CalCOS should check to make sure there are counts in the exposure to extract in the first place, and if there are none to use the default locations in the XTRACTAB/TWOZXTAB reference files and not calculate a centroid. It should also print a warning that it is doing this.

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

Python 2 will not be maintained past Jan 1, 2020 (see https://pythonclock.org/). Please remove all Python 2 compatibility and move this package to Python 3 only.

For conda recipe (including astroconda-contrib), please include the following to prevent packaging it for Python 2 (https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html?preprocessing-selectors#skipping-builds):

build: 
  skip: true  # [py2k] 

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Calcos segfaults on 0s exposures in operations. segfault probability seems to be lower on regular science machines. Seems to be a problem in astropy.io.fits

As brought up by @stscicrawford and discussed with @stscirij , given that calcos is already in Python, there is no reason to keep costools as a separate package. Therefore, it might be feasible to move part (if not all) of costools to calcos. This way, when a change happens in calcos that would affect costools, you don't need to worry about keeping two different package in sync.

https://python-packaging.readthedocs.io/en/latest/non-code-files.html

Sooner or later, you will need it.

Example log: https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=125033830

Type: NameError Message: POLICY WARNING: Exception message is too long: 36779 > 255 name 'a' is not defined -------------------- >> begin captured stdout << --------------------- CALCOS version 3.2.1 (2017-04-28) numpy version 1.12.1 astropy version 3.0.dev20729 Begin 28-Nov-2017 10:41:28 EST Association file = lckg01070_asn.fits TIME-TAG calibration -- 28-Nov-2017 10:41:31 EST Input lckg01czq_rawtag_a.fits OutTag lckg01czq_corrtag_a.fits OutFlt lckg01czq_flt_a.fits OutCounts lckg01czq_counts_a.fits OutFlash lckg01czq_lampflash_a.fits DETECTOR FUV, segment A EXPTYPE EXTERNAL/SCI OPT_ELEM G140L, CENWAVE 1280, FPOFFSET -2 APERTURE PSA BADTCORR OMIT RANDCORR PERFORM RANDSEED = 100 TEMPCORR PERFORM BRFTAB = lref$x1u1459il_brf.fits GEOCORR PERFORM GEOFILE = lref$x1u1459gl_geo.fits IGEOCORR PERFORM DGEOCORR OMIT XWLKCORR PERFORM XWLKFILE= lref$14o2013ql_xwalk.fits YWLKCORR PERFORM YWLKFILE= lref$14o2013rl_ywalk.fits DEADCORR PERFORM DEADTAB = lref$s7g1700gl_dead.fits Last time int...

https://boyle.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/ is broken by #40 as it currently does not support multiple builds like the push event one. Matt will investigate ways to make this possible for JenkinsfileRT as it is with Jenkinsfile.

I see there is a script/calcos/calcos incantation in package setup. It is better off as an entry point.

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

I see that there is already a doc with some documentation (albeit not very informational for regular user). Why not build it on RTD? That way, the code documentation lives with the code and does not need to wait for Data Handbook release to reflect latest code changes.

In conjunction with this, it is also advisable to add a Sphinx build job to Jenkinsfile to catch any warnings or errors before PR is merged.

Issue COSPIP-61 was created by Elaine Mae Frazer:

Update the NUV SDQFLAGS value

Regression Testing (RT) Summary:

Configuration: dev

Report: https://boyle.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

In line 447 of x1d.py, the call to the function timetag.doDqicorr is missing two required positional arguments: traceprofile and gti.

The delta geometric correction step is identical to the geometric correction step, it applies a delta correction to the coordinates after the regular geometric correction is done. The reference file has the same format as the geo reference file.

Chose: DGEOCORR step name
DGEOFILE reference file

Need to apply correction before dq boundaries are calculated in cosutil.fuvFlagOutOfBounds

Use the same IGEOCORR behaviour as regular geo correction step.

This is an issue for the new cenwave 800 reference files.

Traceback:
Input Reference centroid = 431.459575, row_0 = 254
Background region #2 truncated
Reguested region: 359 to 389
Using 359 to 359, with weight 0.032258
/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/trace.py:542: RuntimeWarning: Mean of empty slice.
dtype=np.float64)[goodcolumns].mean(dtype=np.float64)
/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/numpy/core/_methods.py:78: RuntimeWarning: invalid value encountered in true_divide
ret, rcount, out=ret, casting='unsafe', subok=False)
Adopted background regions for reference profile:
Rows 309 to 339, and rows 359 to 359
&&&
Error is here, profile ref file only has 359 rows so row with index 359 doesn't exist
&&&
Background = nan
Measured reference centroid = nan
Input Reference centroid = nan, row_0 = 254
Traceback (most recent call last):
File "/user/rij/anaconda3/envs/calcos_dev/bin/calcos", line 7, in
exec(compile(f.read(), file, 'exec'))
File "/user/rij/hst/cos/ssbgit/calcos/lib/calcos/calcos", line 7, in
calcos.main(sys.argv[1:])
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/calcos.py", line 211, in main
burstfile=burstfile)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/calcos.py", line 434, in calcos
sci_status = cal.allScience()
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/calcos.py", line 2850, in allScience
obs.info, obs.switches, obs.reffiles)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/calcos.py", line 2729, in basicCal
self.wavecal_info)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/timetag.py", line 317, in timetagBasicCalibration
tracemask, traceprofile, gti)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/timetag.py", line 3814, in doProfileAlignmentCorr
tracemask)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/trace.py", line 173, in doProfileAlignment
regions)
File "/user/rij/anaconda3/envs/calcos_dev/lib/python3.6/site-packages/calcos/trace.py", line 587, in getReferenceCentroid
offset = int(round(profcenter - regions['center']))
ValueError: cannot convert float NaN to integer
(calcos_dev)

Currently the value of N_SIGMA, the chi-squared threshold used in the wavelength calibration, is hard coded at 15.0. This will probably not work at LP4 where some of the wavecal spectra will be gain-sagged, so will be missing lines. For lp4 and later, wcptab ref files will have a new column, N_SIGMA, which has the threshold to be used instead of the default. Calcos should check for this column and use the contents if it is there, reverting to the default if not. PR 86840 describes this:
https://www.ess.stsci.edu/prsystem/servlet/prbrowse/pr.86840

Issue COSPIP-39 was created by Nick Indriolo:

In WCPTAB, increase XC_Range for G230L setting

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Many warnings that look like this:

/opt/conda/envs/tmp_env1/lib/python3.6/site-packages/calcos-3.3.4-py3.6-linux-x86_64.egg/calcos/wavecal.py:827: DeprecationWarning: NPY_ARRAY_UPDATEIFCOPY, NPY_ARRAY_INOUT_ARRAY, and NPY_ARRAY_INOUT_FARRAY are deprecated, use NPY_WRITEBACKIFCOPY, NPY_ARRAY_INOUT_ARRAY2, or NPY_ARRAY_INOUT_FARRAY2 respectively instead, and call PyArray_ResolveWritebackIfCopy before the array is deallocated, i.e. before the last call to Py_DECREF.

Calcos currently has a bug wherein it will fail to correctly determine the name of the associated _spt file if the directory containing the _rawtag file contains the string "_raw"

The new_table function has been removed from astropy.io.fits in version 1.3. It needs to be replace by
BinTableHDU.from_columns. The function signature is the same.

Follow up of #30 and related to #12. When more tests are added, the one added in #30 should be marked as slow by uncommenting @pytest.mark.slow and delete the TODO comment (see below). Currently, it is the only test, so it needs to run regardless.

# TODO: Mark this as slow when there are faster tests added for CI tests
#       so that this only runs in nightly tests.
#@pytest.mark.slow
class TestFUVTimetag(BaseCOS):
    ...

Here I create the error:

Python 3.5.4 |Anaconda, Inc.| (default, Nov 20 2017, 18:44:38) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from costools import x1dcorr
The following tasks in the costools package can be run with TEAL:
         splittag                 timefilter                 x1dcorr
>>> x1dcorr.x1dcorr("temp_corrtag.fits", outdir="sliced/", update_input=True, find=True, cutoff=5., verbosity=2)
Input     /home/paw/science/betapic/COS/data/2014/temp_corrtag.fits
OutFlt    sliced/temp_flt.fits
OutCounts sliced/temp_counts.fits
x1d       sliced/temp_x1d.fits

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paw/code/anaconda3/envs/astroconda/lib/python3.5/site-packages/costools/x1dcorr.py", line 185, in x1dcorr
    verbosity=verbosity)
  File "/home/paw/code/anaconda3/envs/astroconda/lib/python3.5/site-packages/calcos/x1d.py", line 207, in extractSpec
    flt_list[i], counts_list[i])
  File "/home/paw/code/anaconda3/envs/astroconda/lib/python3.5/site-packages/calcos/x1d.py", line 448, in makeFltCounts
    phdr, headers[1], minmax_shift_dict)
TypeError: doDqicorr() missing 2 required positional arguments: 'traceprofile' and 'gti'

The same error occurs when not using astroconda too.

Is this where I submit error reports?
If nobody has time to fix this, could somebody point me in the right direction?
What code do I fork and work on to fix the issue myself?

Thanks.

Follow up of #30. This would prevent issues like #36 from dragging on too long until it is too late.

https://astropy.readthedocs.io/en/latest/api/astropy.tests.helper.enable_deprecations_as_exceptions.html#astropy.tests.helper.enable_deprecations_as_exceptions

When you are ready, uncomment this

As discussed with @stscicrawford , @stscirij , et al.

This is because when ACCUM products are created, a PrimaryHDU is first created, then ImageHDUs are appended in astropy's append mode:

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/

Regression Testing (RT) Summary:

Configuration: release

Configuration: dev

Report: https://plwishmaster.stsci.edu:8081/job/RT/job/calcos/test_results_analyzer/