ocfl / ocfl-java Goto Github PK

Allow users to configure what they want to happen when unsupported extensions are encountered:

1. Fail on unknown repo extensions
2. Warn on unknown repo extensions
3. Fail on unknown object extensions
4. Warn on unknown object extensions

Refactor the storage layout code, and implement the existing specs:

OCFL/extensions#12
OCFL/extensions#16

When using the mutable head extension, revision number markers remain in the revisions directory indefinitely. This becomes problematic if someone creates a huge number of revisions (see fcrepo/fcrepo#2024). One solution to this problem is to attempt to delete non-current revision markers after creating a new revision so that the directory only ever contains at most 2 revision markers. I think this should be safe, but requires further investigation.

The cloud implementation currently does not cache any files other than object inventories. If possible, it might be desirable to cache object files on the local filesystem to decrease cloud accesses.

When there is an existing ocfl_layout.json that does not point to a known storage layout, then the failure message needs to be more clear about what's going on.

The mutable HEAD uses rX directories to manage changes. These directories are carried forward when the mutable HEAD is committed. It would be ideal if only the contents of these directories was committed, but this complicates the atomicity of installing the mutable HEAD as a proper version.

OCFL/extensions#17

• Update extension directory name
• Change the HEAD version directory name from HEAD to head

Currently, intermediary directories between the storage root and object root are left on disk when an object is purged. If they are empty, they should be removed.

The batch delete api supports deleting a maximum of 1,000 objects per request: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

Currently, ocfl-java does not take this into account, and will happily create a single request to delete any number of object.s

Per spec changes: https://github.com/OCFL/spec/pull/432/files

Based on the javadocs and the method parameters, it would seem that this method should be named something like putFile() instead of getFile().

https://github.com/UW-Madison-Library/ocfl-java/blob/31129b8ccdd1b9f52a9cbfcfdb6353ec90944e47/ocfl-java-api/src/main/java/edu/wisc/library/ocfl/api/OcflObjectReader.java#L41

The cloud implementation works but is not well optimized. Need to run performance tests, and investigate async clients vs thread parallelization.

For our project I have been trying to see if we can use this library to implement OCFL, and I ran into a small bug causing:

Exception in thread "main" edu.wisc.library.ocfl.api.exception.RepositoryConfigurationException: OCFL root is missing its namaste file, eg. 0=ocfl_1.0.

I think I traced the cause to DefaultOcflStorageInitializer.java#L92 which will always result to false

ocfl-java currently only interacts with objects within the context of an OCFL repository. Should it also support interacting with ad hoc OCFL objects outside of a repository?

I created the ocfl-1.1 branch to start working on adding support for OCFL 1.1.

Outstanding work:

• Update fixity validation OCFL/spec@77ab30d
• Update validation code for object id changes OCFL/spec@89330e1
• Update content directory validation OCFL/spec@0cd230e
• Update manifest validation codes OCFL/spec@9509774
• Update version directory validation codes OCFL/spec@c4c3d8e
• Enforce monotonically increasing versions OCFL/spec@5983ac8 and OCFL/spec@dd27202
• Update extension validations OCFL/spec#584 and OCFL/spec#583
• Update codes for multiple namaste files OCFL/spec#581
• Add 1.1 spec txt
• Thorough testing

Currently, new inventories are first serialized to a file and then their digest is calculated. This can be optimized slightly by calculating the digest while the inventories are being written to disk.

Once the OCFL extensions "spec" has a 1.0 release, include a copy of it in the storage root.

For large video or PDF files I need to response to range requests.
OcflObjectReader should allow access to physical Path on the underlying filesystem to allow efficient handling of those requests.

Change the S3 db lock to not lock a row or hold a connection open. Instead, write a record that indicates the lock is held. Fail if the record already exists and indicates a hold. Partial solution for the problem of not releasing a lock, holds are only valid for an hour (or a configurable duration).

It would be great to be able to use ocfl-java to connect with other, non-AWS services that expose an S3 API.

Per spec change: https://github.com/OCFL/spec/pull/438/files

There should be some sort of validation api that validates an ocfl object's structure and perhaps optionally does a fixity check.

Shorten staging directory names so that there isn't the possibility of creating a directory name that's longer than the filesystem supports.

The S3 transfer manager was finally released:

https://aws.amazon.com/blogs/developer/introducing-crt-based-s3-client-and-the-s3-transfer-manager-in-the-aws-sdk-for-java-2-x/
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html

The S3 code needs to be updated to use it. This should greatly improve S3 performance.

Tasks before releasing 1.5.0

• #78
• Update all dependencies
• Review String.split() usage
• Update contact info in pom

Consider adding support for multi-object transactions.

1. Modified objects would need to be write locked
2. Need a way to stage version changes
3. Need a way to access staged versions
4. Cache update logic needs to change
5. Should transactions expire, and what happens if they do?

The validation code incorrectly skips validating the structural contents of a version's content directory if the version does not contain an inventory file.

Add a method for manually clearing the inventory cache.

https://github.com/adobe/S3Mock is really buggy. Try switching to https://github.com/findify/s3mock

The extensions directory should be skipped when iterating over all of the objects in a repository.

How should the logs directory be used? Currently, nothing is written to the logs directory. How do people want to use it?

Issue raised by @benc. May want to support unicode normalization for logical paths and object ids. For example, ṩ could be represented as \u1E69, \u0073\u0323\u0307, or \u0073\u0307\u0323. Without normalization, it would difficult to find an object or file that uses such a character without knowing the representation that was used.

For logical paths, strings could be normalized for comparison purposes only and written to the inventory in the same form as they were received.

Object ids would need to be normalized as part of the storage layout.

It's unclear how much of an issue this is. If a user consistently encodes their strings, they will not experience any issues.

https://www.unicode.org/reports/tr15/
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/text/Normalizer.html

Within a version, logical paths MUST be unique and non-conflicting, so the logical path for a file cannot appear as the initial part of another logical path.

Update the copy of the ocfl spec that's included in the repo root

The testing and surfacing of this limitation came through the script at the bottom run against a Fedora 6 repository.
The issue, which may potentially require no additional action beyond documenting in the README what the limitation is for file lengths on Ubuntu, is that files with names with 323 characters fails when persisting to OCFL. I have not narrowed down the exact limitation length.

• System

$ uname -a
Linux silver 5.3.0-42-generic #34-Ubuntu SMP Fri Feb 28 05:49:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.10
Release:	19.10
Codename:	eoan

Testing script

clear;x=0; r="http://localhost:8080/rest/"; while true; do x=$(($x + 1));echo $x; r=`curl -XPOST $r`; echo "r = $r"; done 

Script output

Add support for rolling an object back to a previous version. The end result should be that the head version is set to the previous version and all later versions must be deleted.

Include copies of the extension specs when they are used:

• OCFL/extensions#12
• OCFL/extensions#16
• OCFL/extensions#17

Comparing the head version of ObjectVersionId leads to a NPE. ObjectVersionId#equals() should check versionNum for null values.

@Test
public void equalsVersionId() {
    assertEquals(ObjectVersionId.head("a"), ObjectVersionId.head("a"));
    assertNotEquals(ObjectVersionId.head("a"), ObjectVersionId.head("b"));
}

Review the ObjectDetails and VersionDetails APIs to make sure that they are exposing information sensibly. For example, VersionDetails should probably have a method for getting its version number, and there might be a better alternative to ObjectDetails.getVersionMap().

Writing a file with the same hash leads to a IllegalArgumentException due to the fact that the source directory is empty. Before copying, the MutableOcflRepository should check if the hash is already present in the inventory.json.

java.lang.IllegalArgumentException: Source must exist and be a directory: /tmp/ocfl3291760920107648701/ocfl-work/e03731555a6eece977a708aaf6a3c780-3248953293/content/r1

	at io.ocfl.core.util.FileUtil.moveDirectory(FileUtil.java:90)
	at io.ocfl.core.storage.filesystem.FileSystemStorage.moveDirectoryInto(FileSystemStorage.java:266)
	at io.ocfl.core.storage.DefaultOcflStorage.moveToRevisionDirectory(DefaultOcflStorage.java:756)
	at io.ocfl.core.storage.DefaultOcflStorage.storeNewMutableHeadVersion(DefaultOcflStorage.java:694)
	at io.ocfl.core.storage.DefaultOcflStorage.storeNewVersion(DefaultOcflStorage.java:253)
        ...

public class OCFLTestCase {

    @Test
    public void testMutable() throws IOException {
        Path tempDirectory = Files.createTempDirectory("ocfl");
        Path repoDirectoryPath = tempDirectory.resolve("ocfl-repo");
        Path workDirectoryPath = tempDirectory.resolve("ocfl-work");

        Files.createDirectory(repoDirectoryPath);
        Files.createDirectory(workDirectoryPath);

        MutableOcflRepository repository = new OcflRepositoryBuilder()
            .defaultLayoutConfig(new HashedNTupleLayoutConfig())
            .storage(storage -> storage.fileSystem(repoDirectoryPath))
            .workDir(workDirectoryPath)
            .buildMutable();

        String objectId = "object_1";
        ObjectVersionId head = ObjectVersionId.head(objectId);
        repository.stageChanges(head, new VersionInfo(), (updater) -> {
            updater.writeFile(new ByteArrayInputStream(new byte[] { 1 }), "info_1.txt");
        });
        repository.commitStagedChanges(objectId, new VersionInfo());

        repository.stageChanges(head, new VersionInfo(), (updater) -> {
            updater.writeFile(new ByteArrayInputStream(new byte[] { 1 }), "info_2.txt");
        });
        repository.commitStagedChanges(objectId, new VersionInfo());
    }

}

Changing one of the byte[] { 1 } to byte[] { 2 } will work as expected.

Currently, optimistic locking only works using version ids. Should it work with mutable HEAD revisions as well?