nixxcode / jvm-brotli Goto Github PK

It's good practice to tag all commits that serve as the basis for published artifacts.
This way it's easy to see exactly which state of the project a release is based on and which commits are still unreleased.
Also, github nicely shows a list of all releases on the "Releases" tab... :)

The current publish process is very convoluted and error-prone. It involves grabbing the packaged jars containing native libs, signing them manually, then stuffing them into a local "staging" repository one by one before pushing them out to the Nexus staging repo.

Since the native libraries likely won't change anywhere near as often as jvm-brotli code, it would make more sense to simply move the pre-compiled libraries inside their corresponding natives folder and have maven copy them into the classpath when the release profile is run.

This way, the release could be done by maven as a single operation, and the artifacts would be signed automatically. This would drastically cut down on release complexity

Mojave 10.14.6

java.lang.RuntimeException: java.io.FileNotFoundException: File /lib/darwin-x86-amd64/libbrotli.dylib was not found inside JAR. at com.nixxcode.jvmbrotli.common.BrotliLoader.loadBrotli(BrotliLoader.java:85) at com.nixxcode.jvmbrotli.common.BrotliLoader.isBrotliAvailable(BrotliLoader.java:63) at Test$Companion.main(Test.kt:15) at Test.main(Test.kt)

At present, the developer must invoke BrotliLoader.loadBrotli() to load the appropriate native library. This should be done internally by the callable (public) Java methods.

To compensate for the removal of manual loading, a boolean method similar to Brotli.isAvailable() should be implemented to give the developer a way to test for Brotli availability.

UPDATE: The design choice on this issue has changed. Given jvm-brotli's platform dependent nature, the user should always be calling BrotliLoader.isBrotliAvailable() before attempting to "enable" the use of Brotli in their own application.

This means the native library loading can remain tied to this method. It is much simpler than forcing each and every object in the library to check for loaded status.

The difference from previous version, is that the new method returns a boolean, indicating if Brotli is available to use. When called for the first time in a JVM instance, it attempts to load the native lib. If it fails, it prints an exception explaining why. Regardless of failure or success, any further calls to this method will only return true or false, no further load attempts will be made beyond the first call.

This project has unclear licensing.

You have mention the licensing of other code in your README.

But you haven't properly declared a license for your project.

I see you have a <licenses> section in your pom.xml, which is a good first step.

But most projects that care about licensing (like Eclipse Foundation or Apache Foundation), require a few more things.

Create license file in your project root on github (names like LICENSE, or LICENSE.txt, or LICENSE.md are common).
This should contain the license (or licenses if multi-licensed) for your project.

Then create a notice file in your project root on github (names like NOTICE, or NOTICE.txt, or NOTICE.md are common)
This is used to indicate licensed content that you are using, just like you have defined in your README.

Bonus, is that these 2 files are also used by github and your project will start advertising their license properly in the github metadata.

Finally, those 2 files should be present in your binary artifacts as well.
Usually in the META-INF/ directory.

One tip, use the SPDX-License-Identifier: at the end of your LICENSE file correctly, and you'll get even better license integration across the various tooling out in the wild. See https://spdx.org/licenses/

Just wanna know if this library Production ready on Windows and Linux (CentOS) amd64?

Should benchmark old IO stream methods (BrotliInputStream and BrotliOutputStream) against new IO byte channels. (BrotliEncoderChannel and BrotliDecoderChannel)

It would be useful to know which of these I/O methods is faster in particular usage scenarios such as:

• Encoding/decoding small/medium/large files (Disk I/O)
• Encoding/decoding small/medium/large data streams (Memory I/O only, no disk usage)

my code like this:
private static final int BUFF_SIZE = 4096;

private static void checkBrotliEnvironment() {
    if (!BrotliLoader.isBrotliAvailable()) {
        throw new RuntimeException("brotli not support, check about GNI environment");
    }
}

public static byte[] brotli(byte[] data) {
    checkBrotliEnvironment();
    return doBrotli(data, null);
}

private static byte[] doBrotli(byte[] data, @Nullable Parameters parameters) {
    try (ByteArrayOutputStream bos = new ByteArrayOutputStream(max(BUFF_SIZE, data.length / 2));
            BrotliOutputStream out = parameters == null ? new BrotliOutputStream(bos)
                                                        : new BrotliOutputStream(bos, parameters)) {
        out.write(data);
        out.flush();
        return bos.toByteArray();
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

public static byte[] unBrotli(byte[] data) {
    checkBrotliEnvironment();
    try (BrotliInputStream brotliInputStream = new BrotliInputStream(new ByteArrayInputStream(data))) {
        return toByteArray(brotliInputStream);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

private static byte[] toByteArray(InputStream input) throws IOException {
    ByteArrayOutputStream output = new ByteArrayOutputStream();
    copy(input, output, new byte[BUFF_SIZE]);
    return output.toByteArray();
}

private static long copy(InputStream input, OutputStream output, byte[] buffer)
        throws IOException {
    long count = 0;
    int n;
    while (-1 != (n = input.read(buffer))) {
        output.write(buffer, 0, n);
        count += n;
    }
    return count;
}

when i call brotli to compress a byte array , it seems working fine(no exception)
but error occurs when i call unBrotli (step 1 output as param).
message like this:

'java.lang.RuntimeException: java.io.IOException: unexpected end of input
Caused by: java.io.IOException: unexpected end of input
at com.nixxcode.jvmbrotli.dec.Decoder.fail(Decoder.java:50)
at com.nixxcode.jvmbrotli.dec.Decoder.decode(Decoder.java:90)
at com.nixxcode.jvmbrotli.dec.BrotliInputStream.read(BrotliInputStream.java:81)
at com.nixxcode.jvmbrotli.dec.BrotliInputStream.read(BrotliInputStream.java:73)'

What's wrong with the way I use it？Please help Me
@nixxcode

On Gradle it does not seem to be downloading the native library. I had to include following dependency explicitly to get it working on my machine

    implementation group: 'com.nixxcode.jvmbrotli', name: 'jvmbrotli', version: '0.2.0'
    implementation group: 'com.nixxcode.jvmbrotli', name: 'jvmbrotli-darwin-x86-amd64', version: '0.2.0'

However, the example project works fine on my machine.

Is it expected to work with Gradle the same way it works with Maven?

I am running on JDK 11 on CentOS 7 from a docker container and maven.

java.lang.UnsatisfiedLinkError: 'java.nio.ByteBuffer com.nixxcode.jvmbrotli.enc.EncoderJNI.nativeCreate(long[])'
	at com.nixxcode.jvmbrotli.enc.EncoderJNI.nativeCreate(Native Method)
	at com.nixxcode.jvmbrotli.enc.EncoderJNI.access$000(EncoderJNI.java:15)
	at com.nixxcode.jvmbrotli.enc.EncoderJNI$Wrapper.<init>(EncoderJNI.java:40)
	at com.nixxcode.jvmbrotli.enc.Encoder.<init>(Encoder.java:78)
	at com.nixxcode.jvmbrotli.enc.BrotliOutputStream.<init>(BrotliOutputStream.java:31)
	at com.nixxcode.jvmbrotli.enc.BrotliOutputStream.<init>(BrotliOutputStream.java:36)

OS Info

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

mvn -v
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T15:06:16Z)
Maven home: /usr/local/apache-maven-3.6.2
Java version: 11.0.6, vendor: Azul Systems, Inc., runtime: /opt/tools/Linux/jdk/openjdk_11.0.6_11.37.18_x64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.19.76-linuxkit", arch: "amd64", family: "unix"

solved

It would be really nice if you could support apple silicon (ARM, aarch64) architecture.
In this case it would be much more easy to work on a java project on the new apple devices which makes use of jvm-brotli without the need of complex project workarounds using amd64 emulation of jvm and brotli.

Right now, all native library load failures simply throw a runtime exception. The loader method doesn't have special case handling for instances of unsupported JVM platform.

It would be helpful to clearly tell the user when the exception is being thrown due to lack of support.

Really appreciate all the hard work. I would be happy to support this project.
How can I get you a coffee? Please add the "Sponsor" tab in this repository.

Cheers!

This project looks awesome but, as a prospective user, I'd be interested in your take on https://github.com/MeteoGroup/jbrotli, which looks like it is (or was) trying to solve the same problem, but appears to be somewhat unmaintained.

Have you looked at MeteoGroup/jbrotli?
How does this project differ from it?
Are there any trade-offs to be aware of?

Hi,

Our Application generally seems to work fine with Windows 10 version, however on one particular machine having windows 10, the BrotliLoader.isBrotliAvailable method returns "true", however when trying to decode a Brotli encoded string, it throws following error:

com.nixxcode.jvmbrotli.enc.EncoderJNI.nativeCreate([J)Ljava/nio/ByteBuffer;

Platform:
OS Name: Microsoft Windows 10 Enterprise 2016 LTSB
OS Version: 10.0.14393 N/A Build 14393

Any insight will be really useful, could it possibly be to do something with linkng loaded Brotli library to native library.

we have following jars available in classpath to BrotliLoader:

jvmbrotli-0.2.0.jar
jvmbrotli-linux-x86-amd64-0.2.0.jar
jvmbrotli-win32-x86-0.2.0.jar
jvmbrotli-win32-x86-amd64-0.2.0.jar

Any quick pointers to fix this will be highly appreciated.

Currently, there is some functioning example code located under the jvmbrotli test directory, however, It would be much more helpful if this code lived in its own example repository that users could clone, build and run "out of the box".

Not only would this help new users getting started, it would also be a quick and easy way for them to test if their environment is supported.