indeedeng / util

Useful Java utilities

indeedeng/util

util-core

General Java utilities and helper classes

util-varexport

Utility that enables you to expose runtime variables from a running Java application

util-urlparsing

Utility to efficiently parse key value pairs from query strings in URLs. Also includes fast number parsing and url decoding utilities.

util-compress

Utility for compressing and uncompressing data. Includes snappy and gzip.

util-io

Utility for performing IO, including many interfaces implemented elsewhere.

util-mmap

Utility for performing mmap operations and direct memory access.

util-serialization

Utility for serializing and deserializing data to and from binary and string formats.

Code of Conduct

This project is governed by the Contributor Covenant v 1.4.1

License

Apache License Version 2.0

Contributing

util's Issues

Clarify whether individual methods are threadsafe

The blog post introducing util-mmap suggests using AtomicSharedReference for thread safety. It looks like this is just a reference counter so that it can unmap files safely when no longer used, and that individual methods in MMapBufferReader and DirectMemory are threadsafe with no need for extra synchronization. Is that correct?

Change parent pom to `com.indeed#common-parent` for publish to work

Looks like existing parent (oss-parent) defines settings that prevent actual publishing of the new version. Parent pom should be changed to common-parent instead.

Fix unit tests

Tried building this on my mac today and realized the unit test is failing.

Running com.indeed.util.varexport.VarExporterTest
Tests run: 34, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec <<< FAILURE!
testExportTwoLevelMap(com.indeed.util.varexport.VarExporterTest)  Time elapsed: 0.008 sec  <<< FAILURE!
java.lang.AssertionError: 
Expected: iterable containing ["map#nullvalue=null", "map#m1={2\=two, 1\=one}", "map#m2={two\=2, one\=1}"]
     got: <[map#nullvalue=null, map#m1={1\=one, 2\=two}, map#m2={one\=1, two\=2}]>

    at org.junit.Assert.assertThat(Assert.java:778)
    at org.junit.Assert.assertThat(Assert.java:736)
    at com.indeed.util.varexport.VarExporterTest.testExportTwoLevelMap(VarExporterTest.java:680)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

WallClocks should support nano time

There are many cases where we want access to the systems nano time. The WallClock API currently only supports getting millisecond level granularity while offering an interface to code to so that the implementation can be swapped out for unit tests. Lets add nanosecond support.

Stablize travis build

Looks like the openjdk7 build fails for this project every now and then. Lets figure out why and fix it if possible.

An illegal reflective access operation has occurred

I am using util/mmap from maven.
When I use :
Stat.stat(file.getAbsolutePath()).getcTime()

I have these following warning:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.indeed.util.mmap.Stat (file:) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.indeed.util.mmap.Stat
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

i use it with openjdk11.

ByteBuffer

How to get a ByteBuffer from a memory ?

util-varexport: only traverse methods once

Copied from an Indeed internal improvement request:

VarExporter.export traverses and exports @Export-annotated methods more than once in some scenarios. Make it stop doing that.

In particular, this code collects a set of all superinterfaces and calls getMethods on each of them (if declaredFieldsOnly is false). That results in O(NumberOfInterfaces) export attempts per method in the worst case:

        Set<Class<?>> classAndInterfaces = Sets.newHashSet();
        getAllInterfaces(c, classAndInterfaces);
        classAndInterfaces.add(c);
        for (Class<?> cls : classAndInterfaces) {
            for (final Method method : (declaredFieldsOnly ? cls.getDeclaredMethods() : cls.getMethods())) {
                Export export = method.getAnnotation(Export.class);
                if (Modifier.isStatic(method.getModifiers())) {
                    loadMemberVariable(method, export, c, true, prefix, null);
                } else {
                    loadMemberVariable(method, export, obj, true, prefix, null);
                }
            }
        }

Windows support

Being that you have numerous libraries that depend on this (lsmtree in my case) it would be beneficial to provide a windows library.

Non threadsafe access to a TreeSet in Var Exporter

the addChildVariables method adds to a treeSet in a non threadsafe way.

Memory holding issue

Recently I switched from ByteBuffer to this library so as able to work with files > 2GB in library for reading *.bw files (0.3.6 -> 0.4.x changes in https://github.com/JetBrains-Research/big) . Unfortunately I've faced with memory issues, looks like MMapBuffer.close() doesn't actually mark memory as free. Finally my process is killed by linux OOM killer

[23946008.796781] Killed process 12309 (java) total-vm:182699968kB, anon-rss:174968140kB, file-rss:0kB

I'm pretty sure than I don't have other memory leaks in my application. It works with ByteBuffer based big library version. My application sequentially performs some calculations. Each iteration (calculation) opens MMapBuffer(Path path, FileChannel.MapMode mapMode, ByteOrder order) ~30 files (~200 mb) in parallel (each file in own thread), than does multiple reads of different parts of each file and then closes related MMapBuffer. Using your library memory consumption is continuously increasing according to top. Also lsof shows me that during each operation I have 30 memory mapped files, in the end of each iteration I get expected 0 mem mapped files. I added some logging, so I'm sure that each opened MMapBuffer will be closed in the end of each iteration. I cannot figure out what is going on and why memory consumption grows. My code finishes on machine with 378 GB RAM and killed by OOM killer on machine with 190 GB RAM. As I mentioned before ByteBuffer based impl successfully works on machine with 190 GB RAM and consume significantly less memory although memory mapped files are unmapped by JVM in some random moments and normally stays mapped after all iterations have completed.

Do you have any recommendations how to fix or identify this memory leak? Big library doesn't use madvise features or tracked buffers. I really wan't to use your library to deal with large files, but current issue stops me from such switch.

Android support

After heavy searching, asking here and here on SO it looks like this library might solve my problem of unmapping through a native binding - thanks :) !

Do you know if it works on (most) Android devices?

And as you see in the first question above that there seems to be a virtual memory limit on Android, are there any plans to implement kind of a mmap-cache which does mapping and unmapping under the hood? E.g. I have a 3gb file and map just 10 of the most recent accessed segments (e.g. with size of 0.1gb) and e.g. unmap the oldest automatically to be always under the ~1.3-1.5gb limit.

BTW: Some links in the readme should move to github e.g. MMapBuffer.java?

Add support for tagging varexport variable to allow cross-cutting concerns

Feature Request: Allow tagging in VarExporter, so that tags can be applied and visited for cross-cutting concerns (e.g. monitoring).

Example using @export annotation:

  @Export(name="foo", tag={"extmonitor"})
  public long getFoo();

Example of using the tag:

VarExporter.withTag("extmonitor").visitVariables(new Visitor() {
  public void visit(Variable var) {
      sendToExternalMonitoringService(var.getName(), var.getValue());
  }
});

Renaming the default branch from "master" to "main" 

The default branch name for repositories created on GitHub is now main. See github guidance

Dependency to old (16.0.1) Guava version

Current code relies on couple of methods, types included in Guava 16.0.1, but removed from 18.0.
Most important type that is deprecated is InputSupplier.
This prevents any code using these libraries from using later Guava versions.
It is possible to replace such usage with alternatives already available in 16.0.1; I will provide a patch.

Add CONTRIBUTING.md

To better inform followers on how to contribute to this library, lets add a CONTRIBUTING.md

Save the namespace that a Variable is created in

Currently there is no way to figure out the namespace that a variable was created in. It would be nice to be able to know what the namespace was.

Log4j dependency

I would suggest to not directly depend on Log4j directly. It is better to use a generic logging API instead. I suggest to switch to slf4j.