dina-web-nrm / mediaserver-module

Mediaserver core module (source).

mediaserver-module

Mediaserver core module

Background & Purpose

Restful-API -> documentation according to apiary.

The Mediaserver (aka. 'attachment server') handles media files and their metadata.

The functionality is implemented using the RESTful-architecture style

1. The Mediaserver can be used as a standalone server (no coupling to an external system).
2. The Mediaserver can be used as an integrated part of the collections management system (coupled).

The Mediaservers has the basic CRUD responsibilities:

1. Storing : Storing binaries to the filesystem and Storing the metadata to a database.
2. Retrieving: Retrieving the binaries and retrieving the metadata for the binaries.
3. Update: Updating the binaries and updating the metadata for the binaries.
4. Delete: Deleting the binaries and updating the metadata for the binaries.

Constraints

1. The Mediaserver stores the binary-files to the local file system, to the file system where the Mediaserver is installed.

Functional Requirements

The Mediaserver should provide services for other systems.

1. To store media files (Create)
2. To edit media files (Update)
3. To search on metadata, fetch mediafiles (Retrieve)
4. To delete media files (Delete)

Storing the media files

The mediafiles are stored in the filesystem, which has a depth of 3 levels. Only one mediafile is stored, no derivates for images are created at the same time.

All medifiles are stored using UUID as names.

UUID, " Universally unique identifier, 128-bit number." ( http://tools.ietf.org/html/rfc4122)

directory-structure

The directories range from '0' to 'F' (hexadecimal) with a depth of 3 layers, which results in 4096 directories ( 16^3 ). To illustrate, 10 000 000 (ten million) mediafiles would be, with an even spread, would be divided into 2441 mediafiles per directory.

All media files are processed in the same way - they are streamed to the filesystem and streamed back to their client.

The mediafile will hold their own 'media-type'/'mime-type' ( stored in the database )

• The principle is the following :
• • if the generated UUID is 'ab30899c-58a0-4305-85a6-bbfa14f89b92'
• Then the file is stored in the following directory ( subdirectory is made up by the first 3 chars ):
• • directory = /opt/data/mediaserver/newmedia/a/b/3
• • directory and file = /opt/data/mediaserver/newmedia/a/b/3/ab30899c-58a0-4305-85a6-bbfa14f89b92

Metadata Terminology

The terms “metadata" and “tags” are distinguished in the following way:

1. 'metadata' are immuatable, data about the file.
2. 'tags’ are mutable, data about the file content.

'metadata':

1. original filename
2. mime type
3. owner
4. visibility (i.e 'private' or ‘public')
5. md5hash

• Saving the md5hash for every media file facilitates finding duplicates.

6. Exif-metadata Exif-metadata for media files where type is an image are stored in a separate table.

• Exif-metadata is stored in a table of its own.

• there is a configurable Boolean parameter in the database, this parameter is a flag set to 'true' or 'false'.

'tags':

The Mediaserver sets no constraint on the keys that are used.

This gives an external module using the Mediaserver freedom to define its own keys and constrain others keys.

Generics tags are supported and saved as a text-string in the database.

• ':' is used as the delimiter between key and its value.
• '&' is used as the delimiter between key/value-pairs

i.e setting key='country' with value='sweden' and key='view' with value='dorsal'.

is saved as -> 'country:sweden&value=dorsal'

User Guide

The Mediaserver is database- and application server agnostic.

The guiding principle is 'ease of installation and management'.

Verify the system.

base url (default) : http://localhost:8080/MediaServerResteasy/

Test to post a binary-file from the interface (action='media/load')

If all goes well: the server returns a UUID

The log from the wildfly-server : class se.nrm.bio.mediaserver.rs.MediaResourceForm

00:42:13,996 INFO  [se.nrm.bio.mediaserver.rs.MediaResourceForm] (default task-3) in POST -> /load using multiform 
00:42:14,005 INFO  [se.nrm.bio.mediaserver.rs.MediaResourceForm] (default task-3) writing to file /opt/data/media/9/f/e/9fe963f5-0125-45e3-bec1-23885031e952

Test to post a base64-file using curl

directory: docs/demo-data

BASE_URL="http://localhost:8080/MediaServerResteasy/media"
echo "Base URL is " $BASE_URL

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @corvuxcorax.b64 $BASE_URL | json_pp

The log from the wildfly-server : class se.nrm.bio.mediaserver.rs.MediaResourceForm

00:51:45,691 INFO  [se.nrm.bio.mediaserver.rs.MediaResourceForm] (default task-4) in POST -> /media using multiform 

---

How to install

'turn-key' docker-project at dw-media
'turn-key' docker-project at media-docker

Basic steps are as follows:

1. git clone
2. install and populate the chosen database-engine, use the liquibase-script
3. install the Application server (AS), Wildfly 8.x
4. Set up a datasource/datapool/JNDI-handle ( JNDI: java:/MediaDS), same JNDI-name as in the persistence.xml
5. set up the filesystem-path for the media files
6. cd '/mediaserver-module' ( root pom ) :

7. prompt>mvn clean package wildfly:deploy

### NB: if you fail on setting up the datasource you will get the following error
* [ERROR] Failed to execute goal org.wildfly.plugins:wildfly-maven-plugin:1.0.2.Final:deploy (default-cli) on project mediaserver-ear: *

Adding a datasource to wildfly

same info here: https://gist.github.com/Inkimar/d81639a9cd41e96903bfbaa9d07decff

Eclipse modules

0. mysql module
1. eclipse module https://docs.jboss.org/author/display/WFLY8/JPA+Reference+Guide#JPAReferenceGuide-UsingEclipseLink

How to connect to an external system

A link-table in the database maps the ID from the external system to one or many media files.

RESTful-API

Documentation according to apiary

1. @POST mediafile
2. @GET the metadata of the mediafile
3. @GET the mediafile itself
4. @GET a derivate of an image-file ( i.e : height = 150)
5. @UPDATE the mediafile
6. @DELETE the mediafile

How to add supported licenses

Licenses are stored in a separate license-table

This gives the administrator full control of what licenses are permitted in the system.

Maintenability , Some configuration done in the database

The database table ADMIN_CONFIG A table ( key/value) in the schema is used for managing settables :

For instance the below ( key/value ) is set in the database.

1. is_exif = [true||false]
2. path_to_files = '/opt/data/mediaserver/newmedia'

Wildfly and large files

example to increase the possibility to post 4.2GB

1. /opt/jboss/wildfly/bin/jboss-cli.sh
2. [standalone@localhost:9990 /] connect
3. [standalone@localhost:9990 /] /subsystem=undertow/server=default-server/http-listener=default/:write-attribute(name=max-post-size,value=4200000000)
4. [standalone@localhost:9990 /] :reload
5. verify
6. [standalone@localhost:9990 /] /subsystem=undertow/server=default-server/http-listener=default/:read-attribute(name=max-post-size)

mediaserver-module's Issues

UUIDs in db schema should be char(32) not varchar(255)

All 128bit UUIDs in hex are 32 chars long:
ab30899c58a0430585a6bbfa14f89b92

So it is a fixed length field.

In https://github.com/DINA-Web/mediaserver-module/blob/master/db/src/main/dbschema/db.changelog-0.5.xml for various tables (MEDIA, IMAGE, SOUND, VIDEO, etc) is defined as:
<column name="UUID" type="varchar(255)" autoIncrement="false">
I think it should be
<column name="UUID" type="char(32)" autoIncrement="false">

Suggest more scalable directory structure

"The directories range from '0' to 'F' (hexadecimal) with a depth of 3 layers, which results in 4096 directories ( 16^3 ). To illustrate, 10 000 000 (ten million) mediafiles would be, with an even spread, would be divided into 2441 mediafiles per directory."

Suggest each directory going from 00-ff (256), as modern Linux file systems can handle this number of directories no problem. And suggest depth of 5, as the example of 10m media files is not a large amount for many sites.
The example for ab30899c-58a0-4305-85a6-bbfa14f89b92 would now be:
ab/30/89/9c/58/a0430585a6bbfa14f89b92

Also, do not store the dashes in the UUID: they are only inserted to make it easier for humans to read. Also suggest not using them in URLs, etc.

OK, multiple issues, my bad.

SQLite3 support

How difficult would it to modify to allow SQLite3 backend instead of MySql? Advantage is that a site would not need to have a MySql server instance running.