uniphil / trc-plaintext-old

Structured plain text of Honouring the Truth, Reconciling for the Future

Plain-text of Honouring the Truth, Reconciling for the Future

This repository is an effort to extract the plain-text content of the Truth & Reconciliation Commission's Executive Summary, in order to make it more accessible.

Sources

Truth & Reconciliation Commission Executive Summary (PDF, 14MB): http://www.trc.ca/websites/trcinstitution/File/2015/Findings/Exec_Summary_2015_05_31_web_o.pdf

Tech notes

The PDF is split into pages with pdftk.

Each page is run thorough pdftotext to extract the text, then through pdfimages to extract the images.

The jpgs are all inverted for some reason, so imagemagick is used to re-invert them back to normal.

The folder extracted-pages/ contains a subfolder for each page, with the PDF page and extracted text.

These steps are automated with the script extract.py:

$ python extract.py

Fixup Conventions

• Photos: available, auto-extracted from PDF. Perhaps files can be discovered and auto-renamed based on markdown + alt text.
• Citations: no solution yet. For now, using [[intro:N]] format, like [[intro:3]], where the number matches the citation number in the Report source. intro: prefix is necessary because the TRC numbers reset for each section. [[]] convention is made-up, but it should be possible to script a fix if it has to change for a better solution.
• Paragraphs spanning page breaks: flag in front matter to deal with this, clean_pagebreak.
• Real page numbers vs. TRC page numbers: No resolution at the moment, can be scripted later if needed.
• Medium-specific strings: removed, including the table of contents, which we can auto-generate.

Fixup example: Pages 1-10.

To do

• Ask TRC for plain-text sources

  • They are closing the office, and do not have capacity to provide

• Find other parallel efforts and try to join forces

  • Tweet about it
  • Contact key people who might know (who?)
  • Find more collaborators to help

• Extract plain-text

  • Split PDF into pages (so that the plain-text produced can be easily sourced back to the original document) pdftk
  • Extract text strings from each page with pdftotext
  • Extract images from each page with pdfimages (also from xpdf)
  • Fix and mark up the extracted text, one markdown document per page

• Rejoin into a giant markdown document

  • Write a python script for it

• Compile plain-text into useable formats

  • HTML:
    • Export as an HTML web page, which should be easy to link to for reference
    • Host the web page somewhere that makes sense. (where?)
  • epub (the current epub conversion has formatting issues):
  • ALL THE FORMATS

• Investigate collaborative opportunities with #ReadTheTRCReport

  • #6 mentions accessibility, including contributing closed captioning. Contribute that!
  • Other ways of integrating the source text with the video recordings?