uw-loci / forward-cron-job

[RETIRED] WiscScan/UW-Forward integration

NB: This project has been discontinued. It remains archived here for historical reasons.

UW-Forward Collaboration

Overview

LOCI's collaboration with UW-Forward aims to provide a method for WiscScan users to share data with the UW community, as well as the general public, through the UW Library System’s Forward application. Data can be shared with UW-Forward by filling out several simple fields in WiscScan during acquisition.

This page provides a guide for WiscScan users at LOCI who are interested in sharing data in UW-Forward. In addition, other labs or projects interested in developing a similar approach to sharing data can view the code provided at the bottom of this page.

UW-Forward

Forward is the UW library system’s experimental catalog and resource discovery website. Forward allows users to search and browse the library holdings of the entire UW system through a more advanced and user-friendly interface than that of a traditional library catalog. In addition, the Forward project seeks to provide access to other types of university resources including digital collections and research data.

LOCI has developed a system that allows WiscScan users to "opt in" to sharing data in Forward. If a WiscScan user agrees to share data for a particular experiment, it will appear in Forward as a record consisting of a thumbnail image and information about the experiment (metadata). Forward does not provide direct access to the raw data file(s), but it does include instructions for users interested in obtaining raw data to contact the experimenter via email. These records are searchable and browsable like any other library materials, so users searching Forward will now be able to discover and access research data alongside books, journals, and other library holdings.

Sharing Data in UW-Forward

1. In the WiscScan "Experiment Setup Information" window, there is now a check-box at the bottom that allows users to agree to share data with Forward (see below).
2. After clicking the box next to "Check if you want to upload images to UW Forward project," another window will pop up with a brief description of what agreeing to share data with Forward entails. Click "OK" to continue.
3. For any data collected while this box is checked, WiscScan will automatically generate a thumbnail image and metadata file for Forward to use. These files will then be uploaded to a directory that Forward harvests on a regular basis.

Describing Data

The metadata that Forward displays is mapped directly from the descriptions entered into WiscScan during acquisition. In particular, the information entered into the "Project Name" and "Description" fields will help Forward users search and browse for data. There is also a new text field called "Annotations" where brief keywords can be provided (see above). These may be single words or short phrases, and multiple keywords can be entered for each experiment (separated by commas). Entering clear and detailed descriptions and annotations will make it much easier for data to be discovered by Forward users. A sample of what images and metadata look like in Forward is provided below. Note that the information in the "Title," "Description," and "Subjects" fields in Forward are mapped directly from what is entered in the "Project Name," "Description," and "Annotations" fields in WiscScan, respectively. These are the only fields that require input from the experimenter. The rest of the information Forward displays is generated from metadata that WiscScan automatically captures.

Technical Information

In order to share data with UW-Forward, an automated process was developed to transform OME-XML metadata produced by WiscScan to METS XML metadata used by Forward. This was done through the use of an XSLT stylesheet.

In addition, a cron job was created to automatically extract a thumbnail image and upload both the thumbnail and the METS XML metadata to a directory that Forward uses for harvesting. Other projects/labs using the OME-XML data model could use a similar approach for sharing data with Forward.

This cron job takes image datasets produced by an acquisition system (such as OME-TIFFs generated by WiscScan) and uploads them to the UW-Forward database. It resides on the open.microscopy.wisc.edu machine.

Specifically, the workflow with respect to WiscScan looks like this:

1. Scientists gather data, check the "Share with Forward" checkbox in the metadata dialog box.
2. This creates a METS companion file, which is copied to the user's /data/username directory on open.microscopy.wisc.edu, along with the .ome.tiff files.
3. The cron job contained in the file publish.sh runs hourly. When it detects new files in the /data directory, it does two things:
   • Create a thumbnail for each dataset by running CreateThumbnail
   • Copy the thumbnail and the METS companion file into /var/www/forward/adds directory
4. Forward queries the /var/www/forward/adds directory and links the files there to its website.

To compile the code:

mvn

To upload changes to the server:

bin/upload-to-server.sh

After that, publish.sh (which calls the Java program) should run correctly. To make it a cron job, it has been sym-linked to the /etc/cron.hourly directory.

forward-cron-job's Issues

Controlled vocabulary for Forward subject terms

Currently, users are able to enter any terms into the Annotations field in WiscScan. These terms appear as subject terms in Forward. To allow for more effective discovery by Forward users, it would be useful to implement some sort of controlled vocabulary for these terms. This could happen in WiscScan or post-acquisition. An existing controlled vocabulary (e.g., MeSH) could be used, or if that proves too difficult, an internal controlled vocabulary would be helpful to avoid different versions (e.g., singular/plural) of the same term.

Migrated-From: http://dev.loci.wisc.edu/trac/software/ticket/662

Post-acquisition Forward workflow

Currently, the workflow that experimenters use to get data into Forward happens during acquisition (mostly in WiscScan). It would be useful to have a way for experimenters to decide to submit datasets to Forward later in the data lifecycle. This would require either a standalone tool or modifications to an existing tool (e.g., OMERO) that would allow researchers to opt in to sharing data with Forward and to add descriptive information (title, description, keywords). This tool would also need to execute the OME-XML to METS/MODS XSLT stylesheet in order to create the records Forward needs. It would also need to save data to a location where the cron job that generates thumbnails and uploads the METS files and images to Forward is running.

Migrated-From: http://dev.loci.wisc.edu/trac/software/ticket/661

Direct links to OME-TIFF datasets

Currently, Forward instructs users to email the experimenter for access to full OME-TIFF datasets. It would be useful to have a way to link directly to the data from Forward. This would require creating a repository on the LOCI end since Forward will not host data.

Migrated-From: http://dev.loci.wisc.edu/trac/software/ticket/663