phonedude / digital-preservation

NARA digital preservation file format risk analysis and preservation plans

U.S. National Archives and Records Administration Digital Preservation Framework

The National Archives and Records Administration is releasing its Digital Preservation Framework, which consists of a Risk and Prioritization Matrix and File Format Preservation Action Plans.

Background

The National Archives 2018–2022 Strategic Plan embraces the primacy of electronic records. Our vision is to ensure cutting-edge access to extraordinary volumes of government informa­tion and unprecedented engagement to bring greater meaning to the American experience. To do so, NARA must collaborate with other Federal agencies, the private sector, and the public to ensure records and archives thrive in a digital world.

Digital preservation is critical to this work. It becomes even more important because of the June 2019 direction (M-19-21, Transition to Electronic Records) to Federal agencies to transition business processes and recordkeeping to a fully electronic environment and to end the National Archives’ acceptance of paper records by December 31, 2022.

Our digital preservation subject matter experts, led by Director of Digital Preservation Leslie Johnston, have been hard at work to prepare the National Archives for this change. They have formalized a set of documents that describe how we identify risks to digital files and prioritize them for action, and created specific plans for the preservation of these many file formats.

NARA holds several billion files representing more than 500 file format versions. These files can be categorized into 17 general records types or categories of electronic records. The vast majority are email messages, followed by JPEG and TIFF images and plain ASCII text.

The NARA Risk and Prioritization Matrix

NARA uses the Risk and Prioritization Matrix to measure the preservation risk of digital file formats in our holdings and to assess formats we will receive in the future. By answering questions related to the ability to preserve and sustain a file format, we identify relative risk levels.

The sustainability factors each have a different level of impact (positive or negative) on a format’s risk level, with several high-impact factors having the greatest effect on the calculations.

High Impact Factors:

• Positive:
  • A high level of adoption; the availability of format documentation; the ability for a file to document itself; a lack of software dependencies; and no requirement for technical protections (such as encryption) provide the most positive impact.
• Negative:
  • Format age and required hardware and/or software dependencies have the most negative impact.

The answers to all the questions have been assigned numeric values, which are used to calculate an overall Risk Rating and a general risk level which translates to: Low Risk, Moderate Risk, and High Risk.

Additionally, NARA prioritizes formats in our holdings for preservation actions. The Prioritization Matrix is modeled on the traditional preservation model of Value/Use/Need. For our purposes, we use Need/Use/Readiness to determine our preservation priorities. The Risk Rating goes into the “Need” column, representing the Need for a preservation action. “Use” is represented by how common the format is in our holdings at the time of assessment, approximating the level of use of the format in the permanent records of the Federal Government. There is no way to map the “Value” of the holdings to individual file formats because record sets/series typically contain multiple file formats. Instead, we have replaced Value with Readiness, or the capacity for NARA to process and convert formats. We assess Readiness based on the general availability of tools for format migration that do not alter the content in unacceptable ways as well as our capacity to perform acceptable migrations.

For both the Risk and Prioritization, the lower the number, the greater the risk or need. It is possible for numbers to be negative integers.

We are sharing our current completed matrix as a template for its use by any interested organizations.

For a more technical discussion of the development and use of the Risk and Prioritization Matrix, a conference paper presented at the 2018 iPRES International Digital Preservation meeting is available at: https://osf.io/ctw3g/.

File Format Preservation Action Plans

We are also sharing our File Format Preservation Action Plans.

These plans consist of two types of documentation. The first is the Essential Characteristics documentation for multiple categories of electronic records. The 16 electronic record type categories are:

• CAD, Digital Design, or Vector Graphics
• Digital Audio
• Digital Still Image
• Email
• GIS
• Moving Image: Digital Cinema
• Moving Image: Digital Video
• Multimedia
• Presentation and Publishing
• Software and Code
• Structured Data: Calendars
• Structured Data: Databases
• Structured Data: Generic
• Structured Data: Spreadsheets
• Textual and Word Processing
• Web Records

Each Plan corresponds to an electronic record type category and contains a list of “Essential Characteristics,” also known as “Significant Properties,” which identify the characteristics of a record (its Appearance, Behavior, Context, and Structure) that should, be retained, if at all possible, in any format migration. These characteristics are important to ensure the highest fidelity format record migrations.

The second resource is the File Format Preservation Action Plan Spreadsheet. The spreadsheet covers over 500 variant versions of file formats and identifies:

• Categories of electronic records associated with the format
• Specifications, standards, and documentation where possible; some have no specification or standard available
• Proposed preservation migration actions to be taken by NARA, including no action when appropriate
• Recommended tools for processing and preservation actions

These Plans are not exhaustive nor universally applicable proposed actions and recommended or endorsed tools: these represent file formats in NARA holdings and variant versions, the current NARA risk assessment, processing capabilities, and tools in use at NARA.

How Can You Participate?

We are sharing these documents to be transparent about our approach to digital preservation and to share our current practices with Federal agencies, records managers, archivists, digital preservation professionals, researchers, private industry, other stakeholders and allied professionals, and members of the public to help us identify ways we can improve them.

We always welcome feedback on the following topics:

• What revisions can you suggest to the proposed processing and preservation actions for the formats?
  • Are the Essential Characteristics for each record type comprehensive enough for digital preservation?
  • Are the proposed preservation actions for the formats technically appropriate?
• Are there appropriate tools for processing and preservation migrations of specific formats that we do not have listed?
• Are there other formats we haven’t identified that need plans?

Please use the issues feature on this site to leave a specific comment or question or to just start a discussion. You can read more about how to contribute here. NARA staff will respond as quickly as they can.

We expect to update the matrix and plans spreadsheet on an ongoing basis in response to changing risks and new technologies and formats.