wragge / hansard-xml

Commonwealth of Australia Hansard

These are Hansard XML files downloaded from the Parliament of Australia website by Tim Sherratt.

Harvesting history

• The original harvest was run in 2016.
• A complete reharvest was undertaken in April 2019
• A harvest of 1998 to 2005 was added in 2020.
• A reharvest of 1998 to 2005 was run in August 2023.
• A reharvest of 1901 to 19080 was run in May 2024.

Note that some file names change between harvests and some files are split or added.

The files are made available on the Parliament website under a CC-BY-NC-ND licence.

Currently this repository includes files for the House of Representatives and the Senate from 1901 to 1980 and 1998 to 2005. No XML files are currently available for 1981 to 1997. Open Australia provides access to Hansard XML files from 2006 onwards.

Browse the files or download a zip file of the lot.

May 2024 note

A reharvest of 1901 to 1980 was run in May 2024. Most if not all of the file names seem to have changed with version flags now added, eg _v4. Unfortunately the new versioned urls don't redirect to the old forms. The addition of the version flags seem to indicate that corrections and improvements are being made to the xml. I've noticed, for example, some changes to the internal xml structure of files, as well as standardisation of things like quotation marks. This process seems to be ongoing ­– some urls changed as I was in the process of harvesting them! This means that this repository might not reflect the current state of Hansard, but I'll try to reharvest at more regular intervals from now on to pick up these sorts of changes.

More info

• GLAM Workbench: Commonwealth Hansard – Jupyter notebooks to harvest and explore XML formatted versions of Hansard.
• Historic Hansard – my own version of Hansard, optimised for easy browsing.
• Documentation of the Historic Hansard site.

hansard-xml's Issues

7 invalid XML files

Hi,
First off, thanks for putting out this dataset, it has proved really useful.

I just wanted to flag that there are 7 XML files that are invalid. They are:

['https://raw.githubusercontent.com/wragge/hansard-xml/master/hofreps/1974/19741015_reps_29_hor91c1.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/senate/1966/19660824_senate_25_s32.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/senate/1966/19660914_senate_25_s32.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/hofreps/1920/19201117_reps_8_94.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/hofreps/1960/19600531_reps_23_hor27.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/hofreps/1920/19201125_reps_8_94.xml',
 'https://raw.githubusercontent.com/wragge/hansard-xml/master/hofreps/1920/19201116_reps_8_94.xml']

They seem to be due to 404 errors. Maybe they need to be rescrape or perhaps they don't exist? I tried searching for one specifically on APH but couldn't find it.

Text encoding issue with 1998-2005 xml files

It looks like there's a text encoding problem with (I think only?) the 1998-2005 XML files. I've been doing some experiments and noticed a lot of character noise for that time window.

For example, if you compare the original with this file in the repository you'll see that this:

<business.start>
<day.start>2005-02-10</day.start>
<para>—————</para>
<para>
<inline font-weight="bold">The SPEAKER (Hon. David Hawker)</inline>
took the chair at 9.00 a.m. and read prayers.
</para>
</business.start>

has become:

<business.start>
<day.start>2005-02-10</day.start>
<para>�����</para>
<para>
<inline font-weight="bold">The SPEAKER (Hon. David Hawker)</inline>
took the chair at 9.00 a.m. and read prayers.
</para>
</business.start>

I should be able to put together a PR to fix this once I wrap my head around your harvesting notebook.