buda-base / buda-volume-manifest-tool

BUDA volume manifest editing tool

Vagrant scripts for BUDA platform instanciation

The base platform is built using Vagrant and VirtualBox:

1. Install Vagrant and VirtualBox.
2. Download or git clone this repository.
3. cd into the unzipped directory or git clone
4. install VirtualBox guest additions with vagrant plugin install vagrant-vbguest
5. run vagrant up to summon a local instance

Or for an AWS EC2 instance:

1. install the vbguest plugin: vagrant plugin install vagrant-vbguest
2. and run the command: vagrant up or rename Vagrantfile.aws to Vagrantfile and run vagrant up --provider=aws

This will grind awhile installing all the dependencies of the BUDA platform.

Once the initial install has completed the command: vagrant ssh will connect to the instance where development, customization of the environment and so on can be performed as for any headless server.

Similarly, the jena-fuseki server will be listening on:

http://localhost:13180/fuseki

Lds-pdi application is accessible at :

http://localhost:13280/

(see https://github.com/buda-base/lds-pdi/blob/master/README.md for details about using this rest services)

The command: vagrant halt will shut the instance down. After halting (or suspending the instance) a further: vagrant up will simply boot the instance without further downloads, and vagrant destroy will completely remove the instance.

If running an AWS instance, after provisioning access the instance via ssh -p 15345 and delete Port 22 from /etc/ssh/sshd_config and sudo systemctl restart sshd. This will further secure the instance from attacks on port 22.

buda-volume-manifest-tool's Issues

add IDs in image objects when opening existing manifest

There's no notion of ID in the manifests (it was envisionned at some point but then abandonned). I don't know if the ids are necessary for the UI but in any case, they should be regenerated when opening an existing manifest

adding rotation field

there should be a new field in all the image cards, called rotation. This should have a few constraints:

• the default value should be undefined and in that case the field should be empty
• setting it to 0 must set it to undefined and make the display empty
• the value should correspond to the rotation of the iiif spec, so:
  • a string or undefined
  • optionally starting with !, followed by an integer between 0 and 360
• because we don't want weird rotations, the field should be a select between these values:
  • undefined (labelled "no rotation")
  • "90" (labelled "rotated right")
  • "180" (labelled "rotated 180°")
  • "270" (labelled "rotated left")
  • "!0" (labelled "mirrored horizontally")
  • "!180" (labelled "mirrored vertically")

once a value is selected, the preview of the image on the left should be adjusted (perhaps?)

REFACTOR / ORGANIZE

cosmetic improvements

We could have some improvements on the top part:

• adding thumbnails (
  favicons.zip)
• title should be "BUDA Volume Manifest Tool" (both at the top and in the html)
• the top part should be sticky and be visible even if we scroll

information on available updates

The app should read a URL (to be determined) that is a json file having some information about the latest version of the app. If it doesn't match with the current version, the app should point the user to the download page of the latest version. It might be that the github API is enough for this feature.

new tag: color image

There should be a way to indicate that a color should be:

• color
• grayscale
• bitonal (black and white)

The best would be a little icon that would default to bitonal and that when clicked would have:

• color
• grayscale
• bitonal
• set default for the volume

It would be represented in the data as a new tag for each image (just like the other tags), and for the volume with a new volume-default-quality tag in the appdata.

The set default for volume would change all following un-reviewed images

when a user modifies the pagination, the "update following unchecked items" button should appear,

it's only relevant when there's a modification that's being made, but it's very important that it's highlighted in these cases, otherwise users won't understand they need to click it

add little icon on the main card when there is a note associated with the image

button for using grayscale

I just got an epiphany: for areas with very low bandwidth (which is an important part of the target audience), it would be nice to have a button (in the little viewer box) to use grayscale images, so that the images load faster (there's a iiif option to request grayscale).

Note that it shouldn't be the default because we also have a lot of binary images that would be much bigger in grayscale. If there's an easy way to detect images that are color images and have the button only appear in that case (and be the default) that's perfect, but it's probably more dev...

thumbnail for volume should be a tag

I think it shouldn't be in the normal list of tags in the json file because it has a special feature: there can be only one per volume. Other than than it can be with the other tags in the UI. I think the behavior should be that if there's already a thumbnail and the user tags an image as being the thumbnail, it removes the first tag (silently).

add more types of duplicates

instead of the 2 types of duplicates that the bvmt can handle, it would be useful to have the following 4:

• duplicates in the original: in some prints there can be the same page printed twice in a row
• intentional duplicate shots: the same physical page has been digitized multiple times, with different exposures so that difficult parts can be read more easily
• unintentional duplicate shots: the same physical page has been digitized twice, with no clear intent, probably because of a mistake in the digitization process
• file duplicates: the same image file is there twice (the exact same image, pixel by pixel, coming from the same digitization but duplicated after that)

Edit and Hide should be always visible and could be next to the three dots menu on each card

help popups

The app should have help buttons in the shape of a white question mark in a red disc. When clicked, the app should display a popup with some information that's contained in MarkDown files. The files are not written yet but would have the following characteristics:

• would have the language before the extension, ex: general.en.md, and the one corresponding to the current UI language would be selected
• would be in the doc/help/ folder at the root of the git repo (this can be changed if some locations are easier)
• would contain images, indicated like the ones in Github MarkDown ( markup), the images would be in the doc/help/assets/ folder

improve list display efficiency

with https://addyosmani.com/blog/react-window/ or something similar

use postcss

group images in UI

This is a major feature for the UI of v2. The idea is that pages can be grouped by either:

• no group
• left/right
• recto/verso (default)

In both grouped cases, the grouping is horizontal in the UI, two cards are displayed on the same line. There are two purposes to this:

• it becomes easier to check if a single page is missing
• when grouped by recto/verso, the whole group can be moved instead of just one page at a time (which is error-prone)

This means that for drag and drop and for automatic reordering of pages, the whole group should be considered instead of individual pages.

selectors in images

The idea of this feature is that we want to be able to indicate areas of images in the manifest. There are two use cases:

• one is to select a thumbnail which would be a cropped version of an image
• one is to indicate the bounding box of a typical character

Both would be recorded in the manifest in new properties inspired by iiif selectors:

{
... root of the manifest
  "thumbnail-selector" :{
     "on": "I09870012.jpg",
     "region": "100,100,300,100",
     "rotation": "0"
  },
  "shad-selector": ... same format
}

The constraint is that the thumbnail shouldn't stretch more than 1:3 or 3:1 (no constraint for the shad).

In the desktop version, there should be an error if the user doesn't select the shad. In both online and desktop version, there should be an error if the user doesn't select a thumbnail.

For thumbnail, there should be a "select area as volume thumbnail" in the menu of each card. At the beginning of the volume, there should be a card that looks a bit differently that the others and should just be named "volume thumbnail". The preview displayed in the card should be "read only" (it shouldn't have any option, and shouldn't respond to the mouse or have any button.

For the typical character selector, the system should be exactly the same, changing "thumbnail" with "typical shad".

minor issues

Now that the main features work well (thanks!), here's a list of less critical improvements:

• volume language should be bo by default
• when a "duplicate of" or "detail of" field appears, the selected file name by default should be the one before the current image (we used to have that at some point IIRC)
• when an input is focused (ex: pagination), then if the user wants to open the card menu, they need to click anywhere first to defocus the input, then click on the menu icon, I think we can reduce it to one click
• the "save" and "settings" buttons should be in the sticky bar at the top, as should the "fold checked images" button
• prompt the user for an optional commit message when saving (it should be passed to the postUpdate function)
• handle rotate argument of images (don't change the preview with it), it should be a string that conforms iiif rotation spec
• when the user is typing in a pagination field, there should be a visual clue if what they type is not valid according to the type of pagination (there's a function validate() for each type of pagination already in the prediction js file)
• during the drag of the drag & drop operation, if the user moves the mouse towards the top or bottom of the screen, the list should scroll up or down
• when selecting the text of "marginal indication", the mouse moves the whole card instead of selecting the text
• when a bvm with missing pages is open, it requests some silly iiif urls, the preview should indicate that there is not page with an icon or a different color
• the main pagination should appear next to the file name in the header of the card

issues for demo

Let's make the following changes for the screencast demo:

• users should be able to use the tool (without saving) without being logged, maybe this behavior could be an option in the configuration?
• let's make at least the "fold unckecked images" button sticky in the top bar (so that we don't have to scroll all the way up to do that)
• let's make the tags json file an array instead of an object, so that we can reorder them properly without changing their ID (they should keep their current ID, in an "id" field perhaps?
• let's make the tag selection popup wider, it currently doesn't display the full name of some tags, cf:

batch reordering

We don't often have this kind of case, but we recently stumbled upon an interesting one for which I am unsure what the best UX would be, here's the case: in volume bdr:V4CZ45315_I2KG209220:

• images I2KG2092200098.jpg through I2KG2092200407.jpg are 1b to 124a
• images I2KG2092200003.jpg through I2KG2092200097.jpg are 124b to 181a
  (numbers might be a bit inaccurate but that gives an idea of the problem)

So ideally we would in a way select the 97 first pages, and move them at the end... I think one way to do that in terms of ux would be to allow the selection of some images (using a checkbox), and them when drag&dropping one of them, all of them would move... not sure how realistic or ideal that is but is it a big development?

update image list

This is a feature that would be present both online and locally. What I call "image list" is:

• for the online version, the image list like this one
• for the local version, the image list is just the list of files in the folder

This feature is that an action (a button or just every time a manifest is open) triggers the app to fetch the image list (through an API online or by scanning the files locally). The the app compares the images present in the manifest with the images present in the image list. Then:

• for each image that is present in the manifest and not in the image list, the image card is displayed on a red background and the user can only click the "remove" button. An image in that state in the manifest has in its json: "error": "not-in-image-list"
• for each image in the image list not present in the manifest, it creates new cards at the beginning of the manifest, in alphabetical order, in non-reviewed mode

size of preview

On larger screens, the UI block that gets stretched is the one with the tags, but it should be the preview instead. Or perhaps it should just be larger by default, perhaps something like 500px instead of 300

hide preview of validated images

Once the pagination of an image has been validated, the preview doesn't need to appear anymore by default (but it should still be available when clicking on a button)

format all UI data into manifest shape

duplicate 'of-file' options only show files rendered

We should show all file names, even if they aren't rendered to the screen yet. This is a problem after implementing infinite scroll

ease section handling

Instead of having the sections hidden in the volume settings, it would be easier if sections was a dropdown that by default is empty, and when it's open it displays the list of sections, with a form at the top so that a new section can be added easily. A bit like what github does with branches:

"insert missing before" button

This feature is an "insert missing before" button that would appear at each place where there is a gap in the pagination. It would appear in the form of a red box asking users to select one of the following:

• add missing from scans
• add missing from artefact
• pagination gap in the original

In the first two cases it adds all the missing images with the relevant tag (either missing from tag or from artefact tag) and automatically fills the pagination.

In the third case it doesn't add anything in the image list, but marks the error as resolved (in a box that becomes green instead of red). In that case the information should be kept in appData of the manifest, so that the error doesn't appear the next time.

default to hide when tagging bdrc scanrequest tag

When the bdrcscanrequest tag is selected, "hide" should also be selected (it can be deselected manually). Also, when looking at the data of the volume, the tool should detect scan request images automatically and tag them by default (and thus also hide them)

add input to fetch volume id if no query param

focus on first non-reviewed image

When opening a manifest, the app should scroll down the list to the first unchecked image

bug in the simple pagination

when selecting simple pagination (not folio) for a volume, setting a pagination to 1 and trying to predict the subsequent ones doesn't work, with an error in the console

user preferences

The app should be able to write a json file with user preferences in the user directory of the machine. The information should have:

{
   "for-bvmt-version": "0.1.0",
   "user": {
      "gender": "F/M",
      "younger-than-18": false,
      "DPE": "any string",
      "FR": "any string"   
   },
   "default-program-info": {
      "context of discovery": "any string",
      "digitization-setting": "any string",
      "region digitized": "Central/Kham/Amdo",
      "digitized-location-address": "any string",
      "context-of-discovery": "any string"
   },
   "recently-opened-folders": [
      "/path/to/folder1",
      "/path/to/folder2"
   ]
}

and any other info necessary. See issue about bibliosheet for more on these values.

move tag form to main card

changes from last time

Here's a list of changes in the data structure (exemplified in the manifest-simple.js file) that I think make it more coherent and easier to work with:

• new fields in the volume settings (the values are IDs and must have a label in English that can be translated into Tibetan):
  • viewing-direction with 3 values: top-to-bottom (default), left-to-right and right-to-left
  • status with 3 values: editing (default), released and withdrawn
• new string handling: now all strings must be in the lang/langtag for (object with @value and @language), and almost all string fields (except the section titles I think) must be arrays, even if there's just one string. Because each string must have a language, it would be nice to have a dropdown list of languages that the user can choose from (it can be just English and Tibetan for now) on each string field.
• instead of having duplicate-of, etc. fields, the various types of duplicate, details, etc. are now tags, and there's just one of field. The tags that necessitate an of field are indicated in the tags.json file
• the shape of the pagination object has changed but is more simple now
• the shape of the pagination field in the image list also changed, but should be easier to work with, its keys are the ids of the main pagination object
• the data that is not really in the manifest per se but is useful for the app to keep track of is now in an appData object that's not checked by the API
• the default language for the manifest strings (section names and image info indications) can be kept in the appData (and should be Tibetan by default)
• the default language for the other fields should depend on the user and will be read in the user profile in our database (our dev is almost done doing that, so don't bother for now and use English by default)
• the type field of the main pagination object should be one of the values of the pagination_types object in pagination-predictions.js (there are just two values for now)

The API should now be working (some checks are commented to ease testing), you can send it any test data you need, I'll remove it later.

The main feature that is missing other than the new data organization is the automatic reordering which should be relatively easy with the new code. Don't hesitate to ask any question!

Optimize UI updates with longer lists

Right now UI seems to lag a bit with longer lists. Look to use Pure Components, etc

hiding / delete

This is more a clarification than just an issue. In the specs I was using "hide" in two contexts, but I think it needs to be clarified. Here are some new concepts that we could use:

• delete: (manifest only) a deleted image is an image that will be indicated in the manifest as being deleted (although it will still appear in it), this will drive the rest of our system so that it doesn't display the image
• fold: (UI only) an image is folded when the UI has just its name in the bar but doesn't display anything else (no preview / thumbnail, no tags, etc.), this says nothing about the status in the manifest and is just there to allow the UI to be faster by not displaying preview/thumbnail for images that have been validated already
• hide: (UI only) an image is hidden when the image doesn't display it at all, this doesn't mean that it won't appear in the manifest

A folded image could look like

with perhaps an indication of the pagination and a button to unfold it.

The two buttons at the top of the interface could be renamed into the now self-obvious:

• fold checked images
• hide deleted images

preview split

There should be two new buttons in the preview menu (the three dots in the corner of the image):

• split horizontally (this one would be only for Chinese / Japanese publications so we may not want it yet)
• split vertically

When clicked, the preview is split (with a little white space between the two parts). The two parts are independent and have their own set of buttons.

If the "set preview" button is clicked, then all the other previews are done in the same way (split or not, with the same coordinate for the preview parts)

local version

This is the biggest change for v2: the app should be installable locally. The changes when the app is local are:

• instead of having to type a RID to open a manifest, the user is presented with a button "open volume folder" which opens a folder selector
• instead of reading the image list from an API, the app lists the files in the folder
• instead of reading/writing the bvm from an API, the app reads/writes a bvm.json file in the folder
• instead of reading the images from an API, the app reads them from a local folder

Support for Windows 10 Home edition is ok