cchiama / mod-data-import

mod-data-import

Copyright (C) 2018-2019 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

• Introduction
• Compiling
• Docker
• Installing the module
• Deploying the module

Introduction

FOLIO data import module.

Compiling

   mvn install

See that it says "BUILD SUCCESS" near the end.

Docker

Build the docker container with:

   docker build -t mod-data-import .

Test that it runs with:

   docker run -t -i -p 8081:8081 mod-data-import

Installing the module

Follow the guide of Deploying Modules sections of the Okapi Guide and Reference, which describe the process in detail.

First of all you need a running Okapi instance. (Note that specifying an explicit 'okapiurl' might be needed.)

   cd .../okapi
   java -jar okapi-core/target/okapi-core-fat.jar dev

We need to declare the module to Okapi:

curl -w '\n' -X POST -D -   \
   -H "Content-type: application/json"   \
   -d @target/ModuleDescriptor.json \
   http://localhost:9130/_/proxy/modules

That ModuleDescriptor tells Okapi what the module is called, what services it provides, and how to deploy it.

Deploying the module

Next we need to deploy the module. There is a deployment descriptor in target/DeploymentDescriptor.json. It tells Okapi to start the module on 'localhost'.

Deploy it via Okapi discovery:

curl -w '\n' -D - -s \
  -X POST \
  -H "Content-type: application/json" \
  -d @target/DeploymentDescriptor.json  \
  http://localhost:9130/_/discovery/modules

Then we need to enable the module for the tenant:

curl -w '\n' -X POST -D -   \
    -H "Content-type: application/json"   \
    -d @target/TenantModuleDescriptor.json \
    http://localhost:9130/_/proxy/tenants/<tenant_name>/modules

Maximum upload file size and java heap memory setups

mod-data-import provides the ability to uplaod a file of any size. The only limitation is related to the current implementation of the RMB and the size of the heap in the java process. Currently, before saving the file, it is read into memory, respectively, it is necessary to have the size of the java heap equal to the file size plus 10 percent.

Example

File Size	Java Heap size
256mb	270+ mb
512mb	560+ mb
1GB	1.1+ GB

Module properties to set up at mod-configuration

• data.import.storage.type - type of uploaded data storing. Default value is LOCAL_STORAGE. Need to add another implementations for storing
• data.import.storage.path - path where uploaded file will be storing

Issue tracker

See project MODDATAIMP at the FOLIO issue tracker.

Additional information

The raml-module-builder framework.

Other modules.

See project MODDATAIMP at the FOLIO issue tracker.

Other FOLIO Developer documentation is at dev.folio.org