ryanbalfanz / pypds

PyPDS is a Python interface to Planetary Data System (PDS) data products.

Hi there 👋

pypds's Issues

gfiles() needs a home

gfiles() is defined in each of the pds-* tools, needs to be moved. I just don't know where it belongs yet.

At first thought, pds.code.common would work, but it's not really a pds function so I don't think that's best.

PEP8 All python files

Ryan, how do you feel about me submitting a pull request with nothing but the code having been autopep8ed?

Update Sphinx documentation to use local template

The Sphinx templates depend on having Sphinx installed.

Support MER 16 bit decompanded 8 bit images.

A special case of 16 bit images is a decompanded 8 bit image as in this case:

http://pds-imaging.jpl.nasa.gov/data/mer/spirit/mer2po_0xxx/data/sol0770/edr/2p194722334esfapbcp2592r1m1.img

Which was provided as an example for Issue #11 .

The specific image provided in this sample is indeed a 16 bit storage format but the data itself is still 8 bit. This is indicated by the SAMPLE_BIT_MASK '2#0000000011111111#' and if you hexdump the image you'll confirm this when you see the data is entirely '0099 009A 0097 ...'. To properly display this image you would need to expand or "DECOMPAND" this image back to its original 12 bits (again stored in 16 bits) using LUT3 as indicated by the label: SAMPLE_BIT_MODE_ID = "LUT3" and then linearly scale it back to 8 bits.

The LUTs are publicly available at ftp://pdsimage2.wr.usgs.gov/cdroms/Mars_Exploration_Rover/Spirit/mer2mw_0xxx/document/pancam_users_guide.pdf

Rename Parser.parse() to better reflect that it now returns the labels

In response to closed issue #5 (http://github.com/RyanBalfanz/PyPDS/issues/closed#issue/5), Parser.parse() might be better named to indicate that it returns the labels.

• .parse_labels

Or, possibly the label property should be resurrected to return parse() (or whatever the new name may be), thus supporting both methods.

ImageNotSupportedError exception

Hello

Thanks for this project. Im trying to open

http://www.uahirise.org/PDS/DTM/PSP/ORB_008500_008599/PSP_008520_2085_PSP_009232_2085/DTEEC_008520_2085_009232_2085_A01.IMG

from

http://www.uahirise.org/dtm/dtm.php?ID=PSP_008520_2085

but I get an exception.

I saved the file to /tmp/x.img and this is the result:

>>> labels = p.parse(open_pds('/tmp/x.img'))
>>> pprint.pprint(labels['IMAGE'])
{'BANDS': '1',
 'LINES': '18055',
 'LINE_SAMPLES': '8086',
 'MISSING_CONSTANT': '16#FF7FFFFB#',
 'NOTE': '"Pixel values in this file represent elevations in meters above the martian equipotential surface (Mars 2000 Datum) defined by Smith, et al. (2001). Conversion from pixel units to geophysical units is given by the keyvalues for SCALING_FACTOR and OFFSET. This DTM was produced using ISIS and SOCET Set (copyright BAE Systems) software as described in Kirk et al. (2008)."',
 'OFFSET': '0.0',
 'SAMPLE_BITS': '32',
 'SAMPLE_BIT_MASK': '2#11111111111111111111111111111111#',
 'SAMPLE_TYPE': 'PC_REAL',
 'SCALING_FACTOR': '1.0',
 'VALID_MAXIMUM': '-2293.22',
 'VALID_MINIMUM': '-2821.92'}
>>> from pds.imageextractor import ImageExtractor
>>> ig = ImageExtractor()
>>> img,lbls = ig.extract(open_pds('/tmp/x.img'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pds/imageextractor.py", line 121, in extract
    if self._check_image_is_supported():
  File "/usr/local/lib/python2.7/dist-packages/pds/imageextractor.py", line 186, in _check_image_is_supported
    raise ImageNotSupportedError(errorMessage)
TypeError: __init__() takes exactly 1 argument (2 given)

MDS Checksum image verification

Add support to verify images with a known MD5 checksum from the labels:

OBJECT = IMAGE
  LINES = 10
  ...
  MD5_CHECKSUM = "6a35e4271c62e31957ebd12888250e0a"
END_OBJECT

Pick a license

Pick an open source license, I will add it to the project with your (Ryans) copyright information. I think I will fork this and use parts of it as the basis of something else and not have to bother you.

I would prefer to license the derived work as BSD or MIT.

RECORD_TYPE = "FIXED_FORMAT"

Found a literal string version of the FIXED_FORMAT RECORD_TYPE in the wild -- i.e. with surrounding quotes.

cannot run tests.py or basic_mer_tests.py

in the Examples part of the README.rst, it stated to run test but it seems that it always failed because there's no test_data directory/files given in the distribution.

Parser labels inconsistency

The Parser provides both attribute access and a return value on parse().
This may be confusing. Probably best to to one or the other.

Actually, parse() doesn't return anything right now, because I haven't committed the change yet. Needing to do that while writing pds-labels brought the issue to light. The IE relies on attribute access.

Casting data types

Add functionality to define casting types. For example, we may store a dictionary mapping a pattern to match with a callable. Then the casting may be as simple as map(pattern, callable).

{
RECORD_TYPE: str,
SAMPLE_BITS: int
}

If in the event that the pattern is not matched we may attempt to auto-detect type or a default callable (probably str).

I believe that either the PDS Standards Reference or Planetary Science Data Dictionary (See Wiki) provide information defining the type of many common PDS labels.

Test failed error: ModuleNotFoundError: No module named 'core'

Test failed.

ModuleNotFoundError: No module named 'core'
Although the core folder is there inside the directory, so I don't know what the issue really is.
I am unable to use pypds in jupyter nlab and it gives the same reason no module named core.

Python 3

Hi Ryan,
I'm working on Python3 support and I think I'm almost there.
I'm trying to read this label file though:
https://gist.github.com/michaelaye/02cdc9e3446129b5047d
and cannot read the inner objects of 'IMAGE_INDEX_TABLE'. I only get this:

{'COLUMN': {'BYTES': '7',
  'DATA_TYPE': 'CHARACTER',
  'DESCRIPTION': '"Used to link a data product (file) to a standard data product (collection of similar files) described within software interface specification document for a particular data set. "',
  'FORMAT': 'A7',
  'NAME': 'STANDARD_DATA_PRODUCT_ID',
  'START_BYTE': '3048'},
 'COLUMNS': '118',
 'INDEX_TYPE': 'SINGLE',
 'INTERCHANGE_FORMAT': 'ASCII',
 'ROWS': '3745',
 'ROW_BYTES': '3057'}

Can you imagine what might be going wrong?

In case you have time to try it out, my branch for Py34 support is here:
https://github.com/michaelaye/PyPDS/tree/py34 (install with python setup.py develop)

Verify input is valid PDS file

Not sure if this is necessary.

Just don't try to read() a non-PDS file if using a readline() optional sentinel. The sentinel only means not needing to break on the END line of the PDS label header when iterating over the lines. The point of abandoning the iteration is to not read more information than necessary. This is especially poignant given than the binary data may dominate the total file size.

Support for 16 bit image formats

This ( http://pds-imaging.jpl.nasa.gov/data/mer/spirit/mer2po_0xxx/data/sol0770/edr/2p194722334esfapbcp2592r1m1.img ) file is an example of 16 bit .img data which PyPDS currently cannot decode to an image. Mars products appear to be all 16 bit as far as I can see.

The IMAGE label of the above produces this info:
{'BANDS': '1',
'BAND_STORAGE_TYPE': 'BAND_SEQUENTIAL',
'CHECKSUM': '7.44069e+07',
'FIRST_LINE': '93',
'FIRST_LINE_SAMPLE': '201',
'INTERCHANGE_FORMAT': 'BINARY',
'INVALID_CONSTANT': '0.0',
'LINES': '840',
'LINE_SAMPLES': '624',
'MAXIMUM': '254.0',
'MEAN': '141.955',
'MEDIAN': '148.0',
'MINIMUM': '46.0',
'MISSING_CONSTANT': '0.0',
'SAMPLE_BITS': '16',
'SAMPLE_BIT_MASK': '2#0000000011111111#',
'SAMPLE_TYPE': 'MSB_INTEGER',
'STANDARD_DEVIATION': '27.0024'}

Submit PyPDS to PyPI

PyPDS should in easy_install and pip installable. Period.

QUB File Support

Nuff said.

pip version > 1.4.x requires --pre flag for installation

The versioning of PyPDS does not seem to follow the guidelines for the new (since pip version 1.4) versioning rules to distinguish between release and pre-release versions.
So, with a new pip, currently it needs to be installed like so:

pip install --pre pypds

Maybe you could update that? Or put a warning somewhere about this? I just lost 20 minutes over that, while I had a pip 1.3.x and the other party a pip 1.5 and it was totally mysterious that the same command did not work on the other computer.