ocfl / ocfl-java Goto Github PK
View Code? Open in Web Editor NEWA Java OCFL implementation
License: MIT License
A Java OCFL implementation
License: MIT License
Allow users to configure what they want to happen when unsupported extensions are encountered:
Refactor the storage layout code, and implement the existing specs:
When using the mutable head extension, revision number markers remain in the revisions
directory indefinitely. This becomes problematic if someone creates a huge number of revisions (see fcrepo/fcrepo#2024). One solution to this problem is to attempt to delete non-current revision markers after creating a new revision so that the directory only ever contains at most 2 revision markers. I think this should be safe, but requires further investigation.
The cloud implementation currently does not cache any files other than object inventories. If possible, it might be desirable to cache object files on the local filesystem to decrease cloud accesses.
When there is an existing ocfl_layout.json
that does not point to a known storage layout, then the failure message needs to be more clear about what's going on.
The mutable HEAD uses rX
directories to manage changes. These directories are carried forward when the mutable HEAD is committed. It would be ideal if only the contents of these directories was committed, but this complicates the atomicity of installing the mutable HEAD as a proper version.
HEAD
to head
Currently, intermediary directories between the storage root and object root are left on disk when an object is purged. If they are empty, they should be removed.
The batch delete api supports deleting a maximum of 1,000 objects per request: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
Currently, ocfl-java does not take this into account, and will happily create a single request to delete any number of object.s
Per spec changes: https://github.com/OCFL/spec/pull/432/files
Based on the javadocs and the method parameters, it would seem that this method should be named something like putFile()
instead of getFile()
.
The cloud implementation works but is not well optimized. Need to run performance tests, and investigate async clients vs thread parallelization.
For our project I have been trying to see if we can use this library to implement OCFL, and I ran into a small bug causing:
Exception in thread "main" edu.wisc.library.ocfl.api.exception.RepositoryConfigurationException: OCFL root is missing its namaste file, eg. 0=ocfl_1.0.
I think I traced the cause to DefaultOcflStorageInitializer.java#L92 which will always result to false
ocfl-java
currently only interacts with objects within the context of an OCFL repository. Should it also support interacting with ad hoc OCFL objects outside of a repository?
I created the ocfl-1.1 branch to start working on adding support for OCFL 1.1.
Outstanding work:
fixity
validation OCFL/spec@77ab30dCurrently, new inventories are first serialized to a file and then their digest is calculated. This can be optimized slightly by calculating the digest while the inventories are being written to disk.
Once the OCFL extensions "spec" has a 1.0 release, include a copy of it in the storage root.
For large video or PDF files I need to response to range requests.
OcflObjectReader should allow access to physical Path on the underlying filesystem to allow efficient handling of those requests.
Change the S3 db lock to not lock a row or hold a connection open. Instead, write a record that indicates the lock is held. Fail if the record already exists and indicates a hold. Partial solution for the problem of not releasing a lock, holds are only valid for an hour (or a configurable duration).
It would be great to be able to use ocfl-java to connect with other, non-AWS services that expose an S3 API.
Per spec change: https://github.com/OCFL/spec/pull/438/files
There should be some sort of validation api that validates an ocfl object's structure and perhaps optionally does a fixity check.
Shorten staging directory names so that there isn't the possibility of creating a directory name that's longer than the filesystem supports.
The S3 transfer manager was finally released:
https://aws.amazon.com/blogs/developer/introducing-crt-based-s3-client-and-the-s3-transfer-manager-in-the-aws-sdk-for-java-2-x/
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html
The S3 code needs to be updated to use it. This should greatly improve S3 performance.
Tasks before releasing 1.5.0
String.split()
usageConsider adding support for multi-object transactions.
The validation code incorrectly skips validating the structural contents of a version's content directory if the version does not contain an inventory file.
Add a method for manually clearing the inventory cache.
https://github.com/adobe/S3Mock is really buggy. Try switching to https://github.com/findify/s3mock
The extensions
directory should be skipped when iterating over all of the objects in a repository.
How should the logs directory be used? Currently, nothing is written to the logs directory. How do people want to use it?
Issue raised by @benc. May want to support unicode normalization for logical paths and object ids. For example, ᚊ
could be represented as \u1E69
, \u0073\u0323\u0307
, or \u0073\u0307\u0323
. Without normalization, it would difficult to find an object or file that uses such a character without knowing the representation that was used.
For logical paths, strings could be normalized for comparison purposes only and written to the inventory in the same form as they were received.
Object ids would need to be normalized as part of the storage layout.
It's unclear how much of an issue this is. If a user consistently encodes their strings, they will not experience any issues.
https://www.unicode.org/reports/tr15/
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/text/Normalizer.html
Within a version, logical paths MUST be unique and non-conflicting, so the logical path for a file cannot appear as the initial part of another logical path.
Update the copy of the ocfl spec that's included in the repo root
The testing and surfacing of this limitation came through the script at the bottom run against a Fedora 6 repository.
The issue, which may potentially require no additional action beyond documenting in the README what the limitation is for file lengths on Ubuntu, is that files with names with 323 characters fails when persisting to OCFL. I have not narrowed down the exact limitation length.
$ uname -a
Linux silver 5.3.0-42-generic #34-Ubuntu SMP Fri Feb 28 05:49:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan
Testing script
clear;x=0; r="http://localhost:8080/rest/"; while true; do x=$(($x + 1));echo $x; r=`curl -XPOST $r`; echo "r = $r"; done
Add support for rolling an object back to a previous version. The end result should be that the head version is set to the previous version and all later versions must be deleted.
Include copies of the extension specs when they are used:
Comparing the head version of ObjectVersionId leads to a NPE. ObjectVersionId#equals() should check versionNum for null values.
@Test
public void equalsVersionId() {
assertEquals(ObjectVersionId.head("a"), ObjectVersionId.head("a"));
assertNotEquals(ObjectVersionId.head("a"), ObjectVersionId.head("b"));
}
Review the ObjectDetails and VersionDetails APIs to make sure that they are exposing information sensibly. For example, VersionDetails should probably have a method for getting its version number, and there might be a better alternative to ObjectDetails.getVersionMap()
.
Writing a file with the same hash leads to a IllegalArgumentException due to the fact that the source directory is empty. Before copying, the MutableOcflRepository should check if the hash is already present in the inventory.json.
java.lang.IllegalArgumentException: Source must exist and be a directory: /tmp/ocfl3291760920107648701/ocfl-work/e03731555a6eece977a708aaf6a3c780-3248953293/content/r1
at io.ocfl.core.util.FileUtil.moveDirectory(FileUtil.java:90)
at io.ocfl.core.storage.filesystem.FileSystemStorage.moveDirectoryInto(FileSystemStorage.java:266)
at io.ocfl.core.storage.DefaultOcflStorage.moveToRevisionDirectory(DefaultOcflStorage.java:756)
at io.ocfl.core.storage.DefaultOcflStorage.storeNewMutableHeadVersion(DefaultOcflStorage.java:694)
at io.ocfl.core.storage.DefaultOcflStorage.storeNewVersion(DefaultOcflStorage.java:253)
...
public class OCFLTestCase {
@Test
public void testMutable() throws IOException {
Path tempDirectory = Files.createTempDirectory("ocfl");
Path repoDirectoryPath = tempDirectory.resolve("ocfl-repo");
Path workDirectoryPath = tempDirectory.resolve("ocfl-work");
Files.createDirectory(repoDirectoryPath);
Files.createDirectory(workDirectoryPath);
MutableOcflRepository repository = new OcflRepositoryBuilder()
.defaultLayoutConfig(new HashedNTupleLayoutConfig())
.storage(storage -> storage.fileSystem(repoDirectoryPath))
.workDir(workDirectoryPath)
.buildMutable();
String objectId = "object_1";
ObjectVersionId head = ObjectVersionId.head(objectId);
repository.stageChanges(head, new VersionInfo(), (updater) -> {
updater.writeFile(new ByteArrayInputStream(new byte[] { 1 }), "info_1.txt");
});
repository.commitStagedChanges(objectId, new VersionInfo());
repository.stageChanges(head, new VersionInfo(), (updater) -> {
updater.writeFile(new ByteArrayInputStream(new byte[] { 1 }), "info_2.txt");
});
repository.commitStagedChanges(objectId, new VersionInfo());
}
}
Changing one of the byte[] { 1 } to byte[] { 2 } will work as expected.
Currently, optimistic locking only works using version ids. Should it work with mutable HEAD revisions as well?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. đđđ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google â¤ď¸ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.