theovincent / xport

Python reader and writer for SAS XPORT data transport files.

Xport

Read and write SAS Transport files (*.xpt).

SAS uses a handful of archaic file formats: XPORT/XPT, CPORT, SAS7BDAT. If someone publishes their data in one of those formats, this Python package will help you convert the data into a more useful format. If someone, like the FDA, asks you for an XPT file, this package can write it for you.

What's it for?

XPORT is the binary file format used by a bunch of United States government agencies for publishing data sets. It made a lot of sense if you were trying to read data files on your IBM mainframe back in 1988.

The official SAS specification for XPORT is relatively straightforward. The hardest part is converting IBM-format floating point to IEEE-format, which the specification explains in detail.

There was an update to the XPT specification for SAS v8 and above. This module has not yet been updated to work with the new version. However, if you're using SAS v8+, you're probably not using XPT format. The changes to the format appear to be trivial changes to the metadata, but this module's current error-checking will raise a ValueError. If you'd like an update for v8, please let me know by submitting an issue.

Installation

This project requires Python v3.7+. Grab the latest stable version from PyPI.

$ python -m pip install --upgrade xport

Reading XPT

This module follows the common pattern of providing load and loads functions for reading data from a SAS file format.

import xport.v56

with open('example.xpt', 'rb') as f:
    library = xport.v56.load(f)

The XPT decoders, xport.load and xport.loads, return a xport.Library, which is a mapping (dict-like) of xport.Dataset``s. The ``xport.Dataset` is a subclass of pandas.DataFrame with SAS metadata attributes (name, label, etc.). The columns of a xport.Dataset are xport.Variable types, which are subclasses of pandas.Series with SAS metadata (name, label, format, etc.).

If you're not familiar with Pandas's dataframes, it's easy to think of them as a dictionary of columns, mapping variable names to variable data.

The SAS Transport (XPORT) format only supports two kinds of data. Each value is either numeric or character, so xport.load decodes the values as either str or float.

Note that since XPT files are in an unusual binary format, you should open them using mode 'rb'.

You can also use the xport module as a command-line tool to convert an XPT file to CSV (comma-separated values) file. The xport executable is a friendly alias for python -m xport.

$ xport example.xpt > example.csv

Writing XPT

The xport package follows the common pattern of providing dump and dumps functions for writing data to a SAS file format.

import xport
import xport.v56

ds = xport.Dataset()
with open('example.xpt', 'wb') as f:
    xport.v56.dump(ds, f)

Because the xport.Dataset is an extension of pandas.DataFrame, you can create datasets in a variety of ways, converting easily from a dataframe to a dataset.

import pandas as pd
import xport
import xport.v56

df = pandas.DataFrame({'NUMBERS': [1, 2], 'TEXT': ['a', 'b']})
ds = xport.Dataset(df, name='MAX8CHRS', label='Up to 40!')
with open('example.xpt', 'wb') as f:
    xport.v56.dump(ds, f)

SAS Transport v5 restricts variable names to 8 characters (with a strange preference for uppercase) and labels to 40 characters. If you want the relative comfort of SAS Transport v8's limit of 246 characters, please make an enhancement request.

It's likely that most people will be using Pandas dataframes for the bulk of their analysis work, and will want to convert to XPT at the very end of their process.

import pandas as pd
import xport
import xport.v56

df = pd.DataFrame({
    'alpha': [10, 20, 30],
    'beta': ['x', 'y', 'z'],
})

...  # Analysis work ...

ds = xport.Dataset(df, name='DATA', label='Wonderful data')

# SAS variable names are limited to 8 characters.  As with Pandas
# dataframes, you must change the name on the dataset rather than
# the column directly.
ds = ds.rename(columns={k: k.upper()[:8] for k in ds})

# Other SAS metadata can be set on the columns themselves.
for k, v in ds.items():
    v.label = k.title()
    if v.dtype == 'object':
        v.format = '$CHAR20.'
    else:
        v.format = '10.2'

# Libraries can have multiple datasets.
library = xport.Library({'DATA': ds})

with open('example.xpt', 'wb') as f:
    xport.v56.dump(library, f)

Feature requests

I'm happy to fix bugs, improve the interface, or make the module faster. Just submit an issue and I'll take a look. If you work for a corporation or well-funded non-profit, please consider a sponsorship.

Sponsorship

Project sponsored by:

Contributing

This project is configured to be developed in a Conda environment.

$ git clone [email protected]:selik/xport.git
$ cd xport
$ make install          # Install into a Conda environment
$ conda activate xport  # Activate the Conda environment
$ make install-html     # Build the docs website

Authors

Original version by Jack Cushman, 2012.

Major revisions by Michael Selik, 2016 and 2020.

Minor revisions by Alfred Chan, 2020.