fieldmuseum / dams-netx Goto Github PK

Tool for migrating data and media between Collections Management Sys. (EMu) and DAMS (NetX)

License: Other

Python 100.00%

dams netx scripts

dams-netx's Introduction

DAMS-NetX

Scripts for migrating media-files and corresponding data between a museum's Collections Management System (EMu) and DAMS (NetX).

See data/README.md for examples of how to setup NetX-EMu mappings and properties config files.

NetX Folder structures

A given Multimedia asset will be pointed at its Netx destination folder based on these fields:

emultimedia.SecDepartment_tab
- 1st Security Department value is the initial NetX folder
- Subsequent Department values are pathAdd NetX folders

NetX Assets

File names

EMu MulIdentifier -> NetX "EMu File Name" (NetX attribute)
EMu AudIdentifier -> NetX "Filename" (name of asset file in NetX)

Attributes

Multimedia fields from the NetX emultimedia" export are mapped in syncedMetadata.xml
Reverse-attached Event fields:
- "EveEvent" is a concatenation of:
  - eevents.EveEventNumber
  - eevents.EveTypeOfEvent
  - eevents.EveEventTitle (not all Events have EveShortName)
- "EveEventURLs" is a concatenation of:
  - "https://pj.fieldmuseum.org/event/" + eevents.AdmGUIDValue
Reverse-attached Catalogue fields:
- ecatalogue.CatDepartment
- ecatalogue.CatCatalogue

prep_emu_media.py script - rename and copy EMu files to NetX folder structure

How to Prep Media-Files:

In EMu, run the "NetX emultimedia" Multimedia export
python3 prep_emu_media.py [path/to/input-emu-netx-export.xml] [path/to/output.csv] [LIVE/test]

This script takes an emultimedia XML export and copies each record's main Multimedia file into the appropriate folder location for NetX IO to import files to NetX. The appropriate folder is defined by the SecDepartment values from EMu. The newly copied file's name is the AudIdentifier value from EMu. Once complete, NetX IO can be run in watchedFolder mode to ingest the organized, renamed files to NetX -- a prerequisite to running NetX DSS. (see example below)

Output:

Files renamed to AudIdentifier-values (with original file extension).
File paths defined by SecDepartment values in a folder-structure that follows the DEPARTMENT_CSV hierarchy.

DESTIN_PATH_MEDIA and DEPARTMENT_CSV should be defined in your .env file. See example here.

...like so:

Filename | File path
-|-
123-abc-987-def.jpg | Multimedia/Geology/Paleobotany/

How to Prep Media Record data:

python3 prep_emu_xml.py [path/to/input-emu-netx-export.xml]

This script takes the same emultimedia XML export as prep_emu_media.py, and reshapes it into the corresponding DSS XML input-file.

Output

The DSS XML input-file dss_prepped.xml is stored at the DESTIN_PATH_XML location defined in .env For NetX, ingest this to the NetX front-end synced-metadata folder for the DSS autotask to pick up on its next run.

IIIF manifest generation with mongo_to_IIIF.py

If EMu Multimedia records are indexed in a MongoDB database, this script can generate a IIIF manifest for a given EMu IRN. The IIIF manifest outputs as a JSON file as well as to the console. For more info about IIIF setup, see iiif.io

How to Setup mongo_to_IIIF.py

In the env file, define the following 5 MongoDB and IIIF variables:

MONGO DB INFO
- MONGO_DB - add user credentials and login string to log into your MongoDB setup
IIIF INFO - default values in .env.example can work if you're testing locally within this repo
- IIIF_SCHEMA - path to a IIIF schema json file (e.g. "data/config/iiif_schema.json" within this repo)
- IIIF_OUT_HOST - path to the domain where an output IIIF manifest JSON file will be hosted (e.g. while testing, could use the path to raw content in your GitHub repo: "https://raw.githubusercontent.com/[your_gh_id]/dams-netx/main")
- IIIF_OUT_PATH - writeable path to a sub-directory on that domain &/or in this repo (e.g. "data/iiif")
- IIIF_OUT_FILE - name of the output manifest JSON file (e.g. "manifest.json")
Run the script for a given EMu irn present in MongoDB: python3 mongo_to_IIIF.py [EMu IRN]
- e.g.: python3 mongo_to_IIIF.py 2441464
Output IIIF Manifest JSON will be displayed in the console, and output to the path & file defined in your .env
- e.g., see example output manifests in data/iiif

Related Repo's

dams-beam - to cross-check and manage media from BEAM
EMu-xml-to-json - to prep EMu XML for input to dams-netx (among other things)

dams-netx's People

Contributors

Stargazers

Watchers

dams-netx's Issues

EMu export 1: setup for main initial Multimedia ingest

EMu export 5: (eaudit) unpublished media

Added 'NetX_eaudit_unpublished_MM' export in EMu

example export XML file here:
https://github.com/fieldmuseum/dams-netx/tree/main/data/xml_good_examples/NetX_audit_unpublished_MM/2022-9-30

still to do: script to cross-check/remove from NetX (draft here)

Test-feedback -- update DSS

Feedback from test-rounds 1 & 2 includes updates for DSS:

add fields to DSS (ChaMediaForm + Dates/Descrips [DONE]
add conditional field for Date Created (where Descrip = "Created", add date) [TO DO]

future update - add way to map EMu depts to NetX folders

e.g. -- add 3rd column to this csv for original EMu SecDept value vs NetX folder names (currently 1st + 2nd cols):
(e.g. "Education" vs "Learning")

https://github.com/fieldmuseum/dams-netx/blob/main/data/config/SecDepartment_hierarchy.csv

In NetX - attrib history + return-trip

- switch on history for attributes
- map data & media return-trip

setup script to reshape XML for DSS

input = EMu Multimedia XML
output = DSS XML to update all initially-ingested mm

Schedule 3: cronjobs - netx-io (file ingest)

scheduled & running daily, 6 days/week

Add media sets for permission tests

...specifically, test-groups with limited access to:

high-res media
sensitive media
media owned by certain depts/collections

Schedule 2: cronjobs - dams-netx scripts

scheduled & running daily, 6 days/week

setup script for prepping pathAdd CSV

inputs = 2nd & 3rd EMu Exports for Cat/Event MM
output 1 = pathAdd CSV for Catalog-MM
output 2 = pathAdd CSV for Event-MM

[Stage 2] DAMS <-> EMu: setup texcdp & NetX-API functions

Add emu_api functions

EMu export 3: setup for tertiary Event-media prep

Add an XML export in EMu / Multimedia to pull rev-attached Event data

Schedule 1: EMu exports

scheduled & running

future update - figure out searching by date for netx-api fxns

currently, to search by date fields with the netx api call getAssetsByQuery, the query-value needs to be in "milliseconds since Epoch" -- https://currentmillis.com

allow search-input of dates formatted "YYYY-M-D" -- e.g:

netx_api.netx_get_asset_by_field( search_field = 'modDate', search_value = '2022-5-10')
convert search_value from YYYY-M-D to milliseconds - consider https://stackoverflow.com/a/11111177

Check logs/errored files

Logging added to prep_emu_media & netx_add_folder

Documentation added here

setup script for initial ingest file-prep

output 1 = prepped files/folders for netx-io
output 2 = prepped XML for DSS

Add unit-tests & input-validation

unit tests:

for functions in prep_emu_media.py
for functions in prep_emu_xml.py
for utils...

input validation:

check that EMu-exports (mm, eve, cat) have same length (# of records)

for functions in prep_emu_media.py
for functions in prep_emu_xml.py
for utils here