geminiplanetimager / gpi_pipeline Goto Github PK

The build script for the pipeline runtime-compiled versions needs updating for Github instead of svn.

• Update list of excluded files from zips
• Remove obsolete code for ignoring no-longer-present directories for releases, tags, etc
• Replace svnonly with gitonly option to ignore files which are not committed to repository
• Update version string to use github SHA hash (or a short version of that) instead of svn id.

Previously with svn we had the functionality that the pipeline writes HISTORY keywords that contain the commit ID, time, and responsible person:

primitive_version= '$Id: gpi_assemble_spectral_datacube.pro 4212 2015-12-10 04:50:46Z kfollette $'

We should decide whether this functionality is still worthwhile now, and if so figure out how to implement something like it. Looks like there is a 'ident' feature in gitattributes. See https://git-scm.com/docs/gitattributes

Of course git is rather different than svn in several fundamental aspects of how it handles files so this is not as simple as on svn. For some of the complexities see discussion here: http://stackoverflow.com/questions/16524225/how-can-i-populate-the-git-commit-id-into-a-file-when-i-commit

See discussion in the comment thread for #375.

Initial implementation of a primitive attribute CalFileType as distinct from Type committed in 2303. Right now this is just used by the Recipe Editor to set the filename wildcard to e.g. dark.fits when browsing for files in the calibration directory. CalFileType attribute has only been defined for a subset of primitives in the headers.

Next steps

1. define calfiletype for the rest of the primitives that have CalibrationFile arguments.
2. update __start_primitive so it checks the CalFileType argument attribute instead of relying on hard-coded calfiletype variables at the start of all the primitives.

Caution - right now it's reading the CalFileType attributes directly from the gpi_pipeline_primitives.xml index of all the primitives. So if you try to change the CalFileType for a primitive in one given XML file I don't think that will work. But I can't think of any circumstances in which one would want to do that, so this is probably OK.

New feature for Queue Viewer: It should have an option (enabled on a menu?) to just display today's recipes, versus all recipes. This should be enabled by default. This will let the queue viewer tool become useful again, without having to enforce periodic deletion of completed recipes.

It will be important to implement this in such a way that it looks at just filesystem date stamps, not dates or directories in the recipes themselves. This means it can avoid having to parse the XML slowly for thousands of old files you don't care about.

It should properly label what is in each slice of the datacube.

Right now we have several different indicators of what type of file a given pipeline (particularly an output reduced calibration file) is:

1. the file suffix ( "_wavecal", "_distor")
2. the FILETYPE keyword ("Wavelength Solution Cal File", "Distortion Calibration")
3. the calibration file type label used by the CalDB ("wavecal", "distor")
4. the CalFileType label, which used to be the 'Type' argument to each primitive prior to my work on #376. ("wavecal", "distorcal")

These are right now defined in various places all over the place and it's not very organized. The same text has to be entered in multiple places which can be error prone. There are cases like distortion where the CalFileType label doesn't match the suffix.

Here's my suggestion to clean things up:

• we should use one consistent set of file type labels/suffixes throughout the pipeline.
• the FILETYPE keyword should match this suffix, e.g. for wavecals we should have FILETYPE='wavecal' instead of FILETYPE='Wavelength Solution Cal File'
• create a new file in the config directory output_file_types.txt which gives the mapping between short filetype (suffix) and longer descriptions like 'Wavelength Solution Cal File' . Save_currdata can use this information to automatically populate the FILETYPE keyword's comment field. For back compatibility, the CalDB can use this table to translate long descriptive FILETYPEs into the shorter version.
• as suggested in #376, we/I can update __start_primitive to use the CalFileType argument from the XML files rather than having to hard code in each primitive the calfiletype variable as well as the CalFileType parameter.

For polarization slice pairs, the datacube display menu should have an option to difference the two slices.
Also the polarization label text should be updated or debugged to properly display what each slice is. (including perhaps taking the WPangle into consideration?)

Noticed while working with Li-Wei. The gpi_subtract_mean_stellar_polarization_podc.pro primitive has all the FPM radii hard coded in the primitive source. These values are actually all available in pipeline_constants in the config directory (e.g. "gpi_get_constant('fpm_diam_h')") There's no chance these are changing any time soon, but the general principle of "avoid writing down the same information in multiple places" says the primitive should use the config file rather than having its own lookup table.

Lowest possible priority on this one. This is sort of just an FYI that those values do exist in the constants file.

Due to the fact that K band observations will not be performed in a traditional nodding sequence (most likely), flexure compensation will be important when subtracting the background therefore requiring the calibration be done in cube space and not in detector space.

This can probably be completed using a modified version of " Interpolate spectal datacube" with a different suffix and the writing of the file to calibration directory. Implementing this will also require a new Subtract thermal/sky background that works in cube space.

Use test data taken by Jeff on 12 March 20-22

The current way the filenames are auto-generated for the DRFGUI based on the current time is not very user friendly. Specifically, if you try to resave a file a second time even without making any edits, because the filename is derived from the time the second save will go to a new filename. This is not how users expect File | Save to work in pretty much any other application. You should have to select File | Save As... in order to change the filename. The regular File | Save option should always reuse the last used filename.

Channel bias subtraction for the 32 readout channels

  * Basic linear subtraction already done based on reference pixels by detector server
  * start with Patrick's existing code in destriping routine (also does a linear fit to each channel)

Low priority but should be relatively straightforward. Obvious correlated noise to the eye, therefore should be fixed.

Currently, there is no channel bias offset removal in destripe for darks.

I think this is something that should be implemented no? Unless the bias is changing around some mean, in which case removing the bias would remove the mean. Ideally, I think we want to perform a channel bias correction in every science and dark frame...

... someone please correct me if my logic is flawed...

We need to:

• Update installation documentation for source code install.
• Update the pipeline_dev build script to point to github instead of svn.
• Update developer docs: developers/intro.rst, developers\releases_and_versioning.rst, developers\new_primitives.rst
• Update "New GPI Team Members Start Here" google doc
• Update pipeline version number display to show git commit hash instead of svn id.

If you read in a subarray exposure into GPITV, then try to display the wavecal, the lower corner of the subarray is taken to be (0,0) instead of whatever coordinates it truly is. So the wavecal is displayed in entirely the wrong spot.

Very low priority to fix this.

A step beyond using a rescaled dark file would be using several of them. Specifically it was suggested that we could take the available darks that are just below and above the science frame in exposure time, scale them, then interpolate between them using e.g. linear weights based on the relative differences in exposure time. You could carry this one step more and take the available darks taken closest in date before and after the science observations, and similarly linearly interpolate between them. The end result would be a bilinear interpolation between 4 reference dark frames.

Implement and see if this improves subtraction or tracking of hot pixels in any measurable way?

Sorry, I wanted to delete an update but it deleted the entire thread!

repasting here...

Laurent's PSF subtraction experiments suggest that linear interpolation is not good enough and we should be using something more precise. But that was for the rotation step, not th wavelength rescaling. Still, we should at some point check how much error is introduced in this step.

See

http://gpiwiki.azurewebsites.net/doku.php?id=data_analysis:campaign_data_system#fits_header_standardization

Need to

• Update pipeline to write CREATOR keyword (done)
• Update pipeline primitives for ADI/SDI etc. to write PSFSUB and PSFPARAM keywords
• Update pipeline to write FILE_n keywords.

Linear Polarization Intensity is biased.

It would be great to have a primitive to create an unbiased linear polarization intensity image.

Here's a good reference:
http://adsabs.harvard.edu/abs/2006PASP..118.1340V

Marshall has IDL code that generates a synthetic disk given some simple disk geometry and dust grain properties/assumptions (this is an improved version of the code used in the DST), so it's just a matter of integrating it into the pipeline.

Useful for testing polarimetry mode code and reductions of ADI sequences, etc.

The 'Make Data Log File' tool is essentially never used and is probably not needed. It has several problems as described below, however it is low priority to fix these, so instead we're disabling this function for now.

It does work to output a text log file, however:

• the column widths are not wide enough for the new Gemini-format keyword values like "DISP_PRISM_G1234". Columns should be wider or (better solution) only the short & friendly value names should be printed
• after the script runs, the window does not close. In fact a tiny little blank gray window is create for no apparent reason and that sticks around forever too.
• it ought to pop up some dialog that says "File successfully written to so-and-so.txt"

If you click on the GPI logo in the parser gui, or the About menu in the Launcher, it pops up a window listing all of us that programmed it. This should be kept up to date as needed.

In the combine darks primitive, when it makes an estimate of the read noise in the output images, it doesn't take into account the number of coadds applied, which will effectively reduce the noise by 1/sqrt(n) or thereabouts.

Figure out how to prevent multiple copies of the pipeline from running at once under the same user account. Something with semaphores or file locking?

I get this error when I choose the Contrast button from the ImageInfo menu when looking at a Stokes Data Cube:

% Unable to dereference NULL pointer: <POINTER ()>.

% Execution halted at: GPITV::CONTRPROF_REFRESH 18406 /Users/Max/Library/GPI/gpitv/gpitv__define.pro

% GPITV::CONTRAST 18715 /Users/Max/Library/GPI/gpitv/gpitv__define.pro

% GPITV::TOPMENU_EVENT 1512 /Users/Max/Library/GPI/gpitv/gpitv__define.pro

% GPITVO_MENU_EVENT 530 /Users/Max/Library/GPI/gpitv/gpitv__define.pro

% $MAIN$

IDL's language defaults are woefully behind the times. This can be improved with compile_opt settings, see http://www.scicoder.org/2011/11/idl-compile-options-you-should-always-use/

I propose we add to all GPI DRP routines the options:
"compile_opt defint32, strictarr, logical_predicate"

I talked this over with Patrick and we both agree that not having these settings makes it more likely that you'll waste hours debugging stupid stuff like the fact that nonzero even numbers are by default considered false, different from every other language we use...

Zack showed a long time ago that a Moffat profile fits the lenslet PSFs much better than a gaussian profile. We should implement this.

Now that things are set up so that the low throughput lenslets are in the same lenslets in pol and spectral modes (nominally), we could add a function that verifies this when creating a wave cal or polcal. This would raise an alert if the flexure was ever enough to shift things by 1 entire lenslet.

If you click on the GPI logo in the parser gui, or the About menu in the Launcher, it pops up a window listing all of us that programmed it. This should be kept up to date as needed.

Create template that includes: Accumulate Images, Align & Derotate (new primitive), Median Combine (ADI) datacubes - perhaps rename this last primitive so it is more general. This replaces feature request 132. Original request:

For very fast QL of long sequences, GPItv could open a long sequence of images, do an image registration using the spots, rotate to North up and median combine for each wavelength of the cube. This would allow us to rapidly estimate the quality of long sequences, see if we can see faint planets, etc.

This will be implemented as a pipeline template which will automatically open the results in GPItv.

A similar request has been sent and was complete for the GPItv : it would be nice to have the options to save the contrast as a txt file.
Thank you!

I never want to have to adjust the positions live in front of an audience again.

As noticed by Max, sometimes the polarization spot mapping primitive gets wrong which set of spots go together as pairs. There should be a check for this, either as part of the pol spot calibration primitive or a separate quality check primitive.

Currently, the queue just accumulates old processed XML recipe files, which currently number several thousand. There should probably be some housekeeping task that cleans these out after some time period?

Users can manually delete all files in the queue from the Queue Viewer, but that is both a coarse and cumbersome way to do that; it should be automated somehow.

There are a handful of atypical lenslets on the array, which have either low throughput or kinda messed up lenslet PSFs. We should manually make a list of these and their coordinates, and then have a calibration primitive similar to "interpolate bad pixels in 2d frame" that can either just flag them as bad, or interpolate them to have estimated repaired values.

Scan Dir opening in GPItv: /mnt/ifs/Detector//120330/extu0001.fits
% Attempt to subscript with <LONG ( 276)> is
out of range.
% Execution halted at: GPITV::GETTRACK 4406
/home/LAB/IDL/gpitv/gpitv__define.pro
% GPITV::REFRESH 2887
/home/LAB/IDL/gpitv/gpitv__define.pro
% GPITV::DISPLAYALL 2846
/home/LAB/IDL/gpitv/gpitv__define.pro
% GPITV::LOAD_NEW_IMAGE 5408
/home/LAB/IDL/gpitv/gpitv__define.pro
% GPITV::READFITS 5288
/home/LAB/IDL/gpitv/gpitv__define.pro
% GPITV::OPEN 5183
/home/LAB/IDL/gpitv/gpitv__define.pro
% DIRVIEWER::VIEW_IN_GPITV 1250
/home/LAB/IDL/gpitv/dirviewer__define.pro
% DIRVIEWER::UPDATE 837
/home/LAB/IDL/gpitv/dirviewer__define.pro
% DIRVIEWER::HIGHLIGHT_SELECTED 667
/home/LAB/IDL/gpitv/dirviewer__define.pro
% DIRVIEWER::REFRESH 618
/home/LAB/IDL/gpitv/dirviewer__define.pro
% DIRVIEWER::EVENT 166
/home/LAB/IDL/gpitv/dirviewer__define.pro
% DIRVIEWER_EVENT 88
/home/LAB/IDL/gpitv/dirviewer__define.pro
% WIDGET_PROCESS_EVENTS
% $MAIN$

I was able to get an error like the following by attempting to create a primitive that had an '&' character in its description, which confused the XML parser. The make_drsconfig script should (1) check that only valid characters are present in names, and (2) immediately after finishing, attempt to parse the XML file to determine its validity and alert the user if it is unparsable.

@
% Compiled module: GPIDRSCONFIGPARSER__DEFINE.
% Loaded DLM: XML.
% IDLFFXMLSAX::PARSEFILE: Parser SAX fatal error: File:
/Users/mperrin/GPI/sw/pipeline/local_config/gpi_pipeline_primi
tives.xml, line: 92, column: 51 :: Unterminated entity
reference
@

If you change the .fits extension to DQ and then back to science, GPITV will no longer flag the DQ pixels upon reloading, even when explicitly turned off and on again. Possibly line 8648 in gpitv.pro

The pipeline has in various places a bunch of now-unnecessary code for dealing with older FITS files that don't comply to the standard format: really old ones that don't put the data in an image extension, ones that don't have the right header keywords, etc. See the code in gpi_load_fits, or the entire function gpi_load_and_preprocess_fits_file.

Now that we've got essentially the final file format, and there is little cause to go back and reprocess pre-2012 data with the current pipeline, I think most of this code could simply be deleted. Anyone have any objections to that?

Right now for any given primitive argument, one has to explicitly read in that argument's value from the modules array and assign it to some local variable, generally along with given a default:

if tag_exist( Modules[thisModuleIndex], "display") then display=uint(Modules[thisModuleIndex].display) else display=0

There are a lot of instances of that boilerplate code spread across the primitives.

I just realized it should be possible to eliminate all of them using the scope_varfetch low-level command to automatically set all those variables. It would work something like this:

• __start_primitive already looks up the name of the calling function
• Match that against the primitives config file's list of IDL functions. This will give you a list of all the arguments defined for that primitive, and their defaults, types, etc.
• For each argument name in that use, obtain its value in the current recipe as above. Cast it to the appropriate type based on the Type attribute. Assign it to a local variable of the same name as the attribute name, using scope_varfetch. If that attribute is absent or blank, assign the default value.

Wish I'd thought of that a long time ago! At this point, would redoing it to work that way be worthwhile or just confusing? Note that even if we were to implement this automatic argument-to-variable setting, it wouldn't break any existing code; all the affected variables would just get set twice to the same values.

Should add some functionality using the quality extensions now: Mask out any invalid pixels to NaNs would be easiest as an option, or perhaps mark them with overplotted symbols or something like that. Image medianing and so forth operations could respect the bad pixel mask if it is present.

There should be better filenames for the output files.

e.g. instead of darks0003-darks.fits we should get darks_120323_0003_1.45s-darks.fits, or something like that. What that default filename is could depend on what type of file is being written (e.g. FILETYPE keyword). Alternatively we use the gemini data label keyword? And there will probably need to be some configuration file options to define this behavior.

8. Data Cube Extraction
   Inversion algorithm now implemented in pipeline form. Using microlens PSF.
   right now using simulated microlens PSFs from DST
   works on real data already.
   Next to do is to measure microlens PSF from real data. Jerome working on now.

• Action: Jerome, Marshall, Christian to discuss microlens PSF extraction early next week

Jeff to coordinate, lots of other people to contribute.

Goal of by the second year of the campaign, having all the pipeline propagate DQ frames so that we get DQ data cubes out in the end that warn of any potential artifacts or problems at a pixel level.

Worth thinking about UNCERT extensions as well but that's a much harder problem to do well.

We should teach the pipeline about sky frames. These can be recognized from the P and Q offset keywords in the headers.

1. GPITV should check these keywords and append "sky" to the displayed object name if you're offset more than an arcsecond or two
2. The data parser should check these keywords, mark the sky frames as a separate sequence from the science target data, and use the sky reduction recipe for them.

This would be easy to implement, just grab the appropriate bad pixel mask from the caldb and ignore those pixels when computing statistics.

Feature suggested by Dmitry.

It should be easy to difference two images, sum two images, etc.