diamm / diamm

The Digital Image Archive of Medieval Music

The Digital Image Archive of Medieval Music

This project is the website of the Digital Image Archive of Medieval Music. While just having the source code is likely not directly useful to others, publishing the code behind the website as open source is one component of our efforts to sustain this project in the long term.

Our Issues page tracks all known problems and feature requests for the site, along with any discussions around that issue. When an issue has been resolved it is 'closed,' but still available for review. Issues that require changes to the code of the site will usually be linked together, so a person reporting the issue can be apprised of changes arising from their report. See this report for an example.

Our development process has all 'in-progress' code committed to the 'develop' branch. New releases of the code mean these developments are merged to the 'master' branch, and a new release tag created. The version of the site on the master branch is always the version that is running the live site. A list of all changes to the site can be found by reviewing our commits.

Components and Technologies

The site is built using the Django web framework. It uses a PostgreSQL database and an instance of the Solr search engine for its data storage and search capabilities. The front-end for the 'Source' view and the 'Search' view are built using ReactJS with Redux. (Other page views are rendered using Jinja2 templates.)

The Django REST Framework forms a crucial part of the site's functionality. It provides a machine-readable representation for every entity in the database, content negotiation capabilities, and HTTP method-based security.

The site is served with the NginX web server and the gunicorn FastCGI application server. The images are served using the IIP Image Server.

All of our images are available as IIIF Image API endpoints, and we deliver IIIF Presentation API manifests for every source with images. The only caveat is that you must be authenticated with a username and password to access these, due to licensing restrictions with our partner libraries.

diamm's Issues

thumbnails key can be null, which is not valid IIIF

Have to figure out how to omit a key from a serialized instance if the value is null.

Make folio numbers in inventory go to page images

When clicking on a folio number (start or end) the view should switch to the image view. Will probably need to figure out how to do this properly.

Implement IIIF 7.3 for alternate views of a page

For images that exist with multiple wavelengths, implement http://iiif.io/api/presentation/2.0/#choice-of-alternative-resources

Transition IIIF Manifests to use Ranges

Ranges are a better way of doing a page's contents, instead of annotations for each image. The IIIF manifest generation should be modified to fit the new model.

See: http://iiif.io/api/presentation/2.0/#ranges

Add feedback form

Add a feedback model and form for registered users. Maybe it can be attached to any piece of content by generic foreign key?

Javascript initialization is slow

When loading a page the JavaScript event handlers don't attach until a few seconds later. Figure out why.

Search Facets, Filters, and Sorts

Facet/Filter by:

• Type (source, composition, etc.)
• Date (for those items where this matters)

Sort by:

• Name (how? give every object a 'name' field?
• Date

Export MARC for Sources

Migrate Images

Images should be organized in folders:

• Shelfmark
  • Image_0000

A migration script will need to be written that uses the database to do the organization of the filesystem.

Migrate to Diva 5.0

✨ It's shiny. ✨

Display item information with image scroll

The information about what is on each page is now transferred in the IIIF manifest as a 'structures' block. Inside of this block there is a range that gives the "canvases" (== pages) that a specific composition appears on in that source.

As a user scrolls, the information should appear on the blank space to the left of the image viewer. (See the Cantus Ultimus viewer for a model of how this should work.)

This could be implemented as a Diva plugin so that we can listen to the Diva page changing events.

Search partbook sets in full text

Bug: composers should be singular when there is only one composer

Import Manifest

Given a IIIF Manifest, import it and match it to pages.

Return helpful message if IIIF manifest is requested for source without images

If /manifest is requested for a source without images, return a helpful message of some form that indicates that this is not possible.

Convert IIIF range key for canvas to member

canvas and ranges are being phased out in IIIF.

Migrate comments from old site

Comments from the old site are in the database, we just need to figure out how to bring them over and attach them to the appropriate new user accounts. This will depend on first building a comment function in the new site, which will be filed as a separate issue.

Investigate Solr handler for page info

Currently the information about a particular page range is all transferred within the IIIF manifest. This bulks up the manifest size considerably and increases the download time of the manifest.

It may be more performant to transition this to an on-the-fly lookup for that range that proxies the range handler to Solr to look up the information for a particular page.

I will investigate and report back.

INSTALLED_APPS is missing the flatpages and sites app

The contrib.flatpages and contrib.sites app entries is missing from INSTALLED_APPS in the settings.py on the about-menu branch.

Build is broken

  File "/Users/ahankins/Documents/code/git/diamm/diamm/urls.py", line 39, in <module>
    from diamm.views.feedback import FeedbackView
ImportError: No module named 'diamm.views.feedback'

The feedback view is imported in urls.py but has not been committed, which breaks the build. This should be fixed ASAP.

Figure out best way to import archives' former name

Currently the archive have their 'former' name in their actual name. We should have a field to separate out the 'formerly' part of the name. Also, this name should be indexed for search.

Results pagination controls

It's all in place, just need the controls

Launch of Beta site (?)

The programme for the Med-Ren conference in Sheffield has timetabled a short session for three projects to launch their new websites, and includes DIAMM. The date is 6 July at 14.30 BST.

Add DIAMM logo to IIIF manifest

n/t

The About Menu

The about menu will be a single large drop-down ("mega dropdown") multi-column menu that will host all the bits of information for the DIAMM organization.

This is what I'm talking about:
http://designmodo.com/demo/css3megamenu/ (hover over 'earnings').

All of the pages there should be editable from the admin interface.

The headings should be:

• Project Description
• Acknowledgements
• Collaborators
• Project Team
• Partners/Sponsors
• Technical Overview
• Depositors
• Copyright and Citation
• DIAMM in Use
• Donate to DIAMM (will also be a big red button in the menubar)
• Contact

Content from these pages can be taken from the existing site, http://www.diamm.ac.uk/about/

We may also want to have a description of the website's technical capabilities, etc.

The pages should be written using Markdown, which is already built into the webapp for source descriptions. The Django staticpages module may prove to be useful for this task.

Fix relationships to people and organizations

Make the relationship to people and organizations a GenericForeignKey relationship so that it can point to both types of models.

This is for:

• SourcePerson
• SourceProvenance
• SourceCopyist?

Integrate markdown on text fields

Format search results list

Make it pretty.

Titles for MSS headings

From @mscuthbert:

This reminds me (if you're up for suggestions yet), I think that the heading of a manuscript page / search result should be in the form "Oxford, Bodleian Library, Canon. Misc. 213" and not "GB-Ob Canon. Misc. 213" and only use RISM sigla for linking when space is at a premium; given that we're using post-1980 RISM sigla now ("GB-Ob Canon. Misc 213" instead of "GB-Ob213") the space savings is far less than it used to be, compared to the clarity (and potentially higher Google search results)

Use custom metadata service for page contents information

It is not possible to capture structured data in the Open Annotation model. A separate service should be used to capture the data.

The custom service can be implemented with a seeAlso with format type application/json

Include indication in search results which sources have images

There should be an indication in the search results which sources have images and which sources do not.

Add configuration setting for IIIF thumbnails

Add a setting in the django configuration to control the width of the thumbnails.

Todo: Track contributors

Allow external contributors for different objects in the DB

• Create contributors group. Assign trusted users to this group
• Trusted users see buttons when they log in allowing them to contribute stuff
• Create contributions model.
• Contributions model stores original model, pk, fieldname, contributor, contribution, approval status, approver, date & time.
• Approve script on backend to approve publications
• Frontend shows contribution metadata: contributor, approval, dates, etc.

Figure out:

• Multiple contributions on the same field. Aggregate? Edit?

Show/Index titles for non-public sources

If a source is set to non-public, the title should still be shown (so that nothing 'disappears' on people)

Include filter for sources with images

There should be a filter option for only retrieving sources with images attached to them.

Get contribution model working

The Contribution model in models/site/contribution.py is waiting to be hooked up. People, Organizations, Compositions, and Sources can all have a "contribute an update" function where people can provide updates to records.

These contributions will then be checked by a member of the DIAMM staff and integrated into the record.

Any contribution marked 'completed' by a member of DIAMM staff can then appear in a 'contributions' section of the object in question. There is also a 'summary' field. Thus the contributor's section for a source might have entries like:

John Smith: Spelling corrections for description. DD/MM/YYYY.
Jane Smith: Correction of inventory listing. DD/MM/YYYY.
&tc.

Supercedes #1

Bug: Key error for all copyists

'type_s' key error in the detail view of copyists.

Traceback :

File "/Users/ariellegoldman/Documents/code/diamm/diamm/serializers/website/person.py" in <listcomp>
  109.         return [PersonSourceCopyistSerializer(o, context={"request": self.context['request']}).data for o in obj.solr_copyist]

CCM and RISM bibliographical abbreviations within descriptions

At the moment CCM and RISM self-refer to their bibliographies with their own system of bibliographical abbreviations. These are in the bibliography table (though in some cases where both catalogues list the same book, we rationalised and used a single abbreviation for a book, so one of the abbreviations will have been replaced with the other). The abbreviations are not currently visible to users. However there are remarks like this in the descriptions: '...with additions ca. 1600-3 (BrayJB)'. This tells the user which bibliographical entry supplied the date. We could:

• a) convert these mentions to full citations (which means we no longer respect the original text );
• b) create a link to the bibl entry hiding behind the abbreviation (so that we're not editing the RISM/CCM text);
• c) allow users to search on the abbreviations (might not work if we've 'rationalised' that abbreviation with one from the other catalogue, so this is no longer visible);
• d) ignore them and let users work it out for themselves from the bibliography list

This is of very minor importance, but might be worth thinking about. At the moment there is just one field for abbreviations (and I use that to connect to source/item/composition records rather than key numbers as abbreviations are easier to remember), so no differentiation between RISM/CCM and newly created abbreviations for new bibliographies.

Remove the Browse button in the menu

It may make a comeback later, but for now we will disable it until we figure out how browse will work in the new site.

Choose naming for libraries and archives

would the 'Archive' label not more accurately be called 'Library/Archive'? There seem to be as many libraries as archives on your list of locations.

MS listings

Any 'short title' list of MSS/hselfmarks should include the tag line following the shelf mark (perhaps in brackets or separated by a dash?); this is in the Source Table field , the point being that you don't have to switch to the full MS description to find out what type of source lies behind a shelfmark (e.g. fragment, choirbook, partbook, chant, theory treatise, etc.). The current facet listing was supposed to do this, but the layout meant it was too narrow to accommodate the information.

Browse sources

From an e-mail:

"In the new form of the website would it be helpful to be able to select country by alphabet letter at the top (as when one is searching journals on Jstor) rather than having to scroll through the whole lot?"

Case sensitivity in searches (recommendation: NO!)

in the Beta:
leiden 2515 does not find the manuscript it should refer to, but Leiden 2515 does. (screenshot)

It appears that some fields are being indexed case sensitively and others are not, and Shelfmark, etc.

(oh, and tiny annoying extra space between fragment and 14th c.)

Use range for IIIF manifests

See: http://iiif.io/api/presentation/2.1/#range

Use range for image contents.

Folio Start and End use not consistent

Causing a failure in manifest generation

Todo: User migration

Current users can sign in with their old username or e-mail. If the system detects that they were a former user, it will prompt them to update their password.

Figure out: Can this be generalized for all users (new / existing)

Fix formatting of search results in result list

Adding display information for each source result so users can see more information before clicking the link

Search box should keep the query

The search box should refresh with the query still intact.

Add descriptions to the IIIF manifests

Add the description field to the IIIF manifest.

Save searches

Searches should be saveable and re-loadable.