Coder Social home page Coder Social logo

electrostat-lab / jme-alloc Goto Github PK

View Code? Open in Web Editor NEW
6.0 2.0 1.0 17.96 MB

A direct dynamic memory allocation API for jMonkeyEngine lwjgl-2 and android games

Home Page: https://hub.jmonkeyengine.org/t/jme-alloc-project/46356

License: BSD 3-Clause "New" or "Revised" License

Java 57.59% Shell 9.93% C 19.48% Groovy 13.00%
cpp desktop gradle java jmonkeyengine3 allocation-api gradle-build lwjgl2 shell android-api

jme-alloc's Introduction

jme-alloc project

A direct dynamic memory allocation api for jMonkeyEngine lwjgl-2 and android games.

To build locally, use:

┌─[pavl-machine@pavl-machine]─[/home/twisted/GradleProjects/jme-alloc]
└──╼ $./gradlew clean && 
      ./gradlew :jme3-alloc:compileJava && \
      ./gradlew :jme3-alloc-native:compileX86_64 && \
      ./gradlew :jme3-alloc-native:copyNatives && \
      ./gradlew :jme3-alloc:assemble

To test locally, use:

┌─[pavl-machine@pavl-machine]─[/home/twisted/GradleProjects/jme-alloc]
└──╼ $./gradlew :jme3-alloc-examples:run

For more about, building, testing and contributing, visit:

CONTRIBUTING.md

For quick use:

/* select your platform here */
final String platform = "desktop"
final String binaryType = "debug"
final String version = "1.0.0-pre-gamma-1"

repositories {
    mavenCentral()
}
dependencies {
    implementation "io.github.software-hardware-codesign:jme3-alloc-${platform}-${binaryType}:${version}"
}

API:

Build-system:

  • Separate jvm and native modules.
  • Generating header files for java sources.
  • Packaging java and natives in a jar.
  • Github-actions.
  • Handling different variants build (linux-x86-64).
  • Handling different variants build (linux-x86).
  • Handling different variants build (windows-x86-64).
  • Handling different variants build (macos-x86-64).
  • Handling different android build architectures (aarch-64, arm32, intel-x86-64, intel-x86).
  • Handling different variants build (windows-x86).
  • Handling different variants build (macos-x86).

Documentation:

  • JavaDocs.
  • NativeDocs.
  • Running Examples.
  • Contribution guide.
  • Documentation website for different releases (not only the latest).

jme-alloc's People

Contributors

ali-rs avatar pavly-gerges avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

ali-rs

jme-alloc's Issues

[NativeBinaryLoader] Allow loading natives via JME NativeLibraryLoader

It would be nice to add an option to be able to extract and load natives via JME NativeLibraryLoader. By default, JME (in the master branch) will export to System.getProperty("java.io.tmpdir") instead of extracting to the working directory (user.dir) if no custom extraction folder is specified.

[Bug] Error when running the staging repository from branch 'maven-publishing'

Error when including the dependency io.github.software-hardware-codesign:jme3-alloc-desktop:

└──╼ $./gradlew :jme3-alloc-examples:build
Starting a Gradle Daemon, 2 busy and 13 stopped Daemons could not be reused, use --status for details
> Task :jme3-alloc-examples:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jme3-alloc-examples:compileJava'.
> Could not resolve all files for configuration ':jme3-alloc-examples:compileClasspath'.
   > Could not resolve io.github.software-hardware-codesign:jme3-alloc-desktop:1.0-pre-alpha-15.
     Required by:
         project :jme3-alloc-examples
      > Could not resolve io.github.software-hardware-codesign:jme3-alloc-desktop:1.0-pre-alpha-15.
         > Could not parse POM https://s01.oss.sonatype.org/content/groups/staging/io/github/software-hardware-codesign/jme3-alloc-desktop/1.0-pre-alpha-15/jme3-alloc-desktop-1.0-pre-alpha-15.pom
            > Missing required attribute: artifactId

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 46s
3 actionable tasks: 1 executed, 2 up-to-date

[Core-Bug] Writing on a buffer after destruction leads to a jvm monitor crash

Writing on a buffer after destroying it will lead to a jvm crash with the following logs, despite the ability to print the buffer capacity, limit and the current position.

The jvm crash log:

---------------  P R O C E S S  ---------------

VM state: at safepoint (shutting down)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007ff5180117d0] Threads_lock - owner thread: 0x00007ff518125c60

Heap address: 0x0000000746e00000, size: 2962 MB, Compressed Oops mode: Zero based, Oop shift amount: 3

CDS archive(s) mapped at: [0x0000000800000000-0x0000000800be3000-0x0000000800be3000), size 12464128, SharedBaseAddress: 0x0000000800000000, ArchiveRelocationMode: 0.
Compressed class space mapped at: 0x0000000800c00000-0x0000000840c00000, reserved size: 1073741824
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0, Narrow klass range: 0x100000000

GC Precious Log:
<Skipped>

Heap:
 garbage-first heap   total 194560K, used 5065K [0x0000000746e00000, 0x0000000800000000)
  region size 2048K, 2 young (4096K), 0 survivors (0K)
 Metaspace       used 841K, committed 1024K, reserved 1056768K
  class space    used 78K, committed 192K, reserved 1048576K

A workaround is to nullify the java.nio.ByteBuffer reference internally from jni via the invocation api after destroying its memory; because this memory is now owned by another allocated object, so accessing it using this reference and writing on it might be dangerous if it is not thread protected.

[Feature-Request] Buffer Partitioning Utility

Provides an advanced buffer partitioning as a separate library.

For example:

#include<stdio.h>
#include<stdlib.h>

typedef struct {
    void* start_address;
    void* end_address;
    size_t offset;
    size_t size;
    size_t pointer_location;

    void (*invalidate)(void*);

} MemoryPartition;

static inline void invalidate(MemoryPartition* partition) {
    partition->start_address += partition->offset;
    partition->end_address = partition->start_address + partition->size;
    partition->pointer_location = partition->offset + partition->size;
}

static inline MemoryPartition create(void* buffer, size_t offset, size_t size) {
    MemoryPartition partition = {
        buffer + offset,
        (buffer + offset) + size,
        offset,
        size,
        offset + size,
        &invalidate
    };

    return partition;
}

int main() {
    const size_t size = 4;
    void* buffer = (void*) malloc(size * 3);
    printf("Allocated buffer start address = %p\n", buffer);
    
    const MemoryPartition partition0 = create(buffer, 0, size);
    printf("Partition0 start address = %p\n", partition0.start_address);
    printf("Partition0 end address = %p\n", partition0.end_address);
    
    const MemoryPartition partition1 = create(buffer, partition0.pointer_location + 1, size);
    printf("Partition1 start address = %p\n", partition1.start_address);
    printf("Partition1 end address = %p\n", partition1.end_address);
    
    const MemoryPartition partition2 = create(buffer, partition1.pointer_location + 1, size);
    printf("Partition2 start address = %p\n", partition2.start_address);
    printf("Partition2 end address = %p\n", partition2.end_address);

    /* add some data to the partitions */
    *((int*) partition0.start_address) = 44;
    *((int*) partition1.start_address) = 55;
    *((int*) partition2.start_address) = 255;
    
    printf("%s\n", "ls part: Print partitions data: ");
    printf("%i\n", *((int*) partition0.start_address));
    printf("%i\n", *((int*) partition1.start_address));
    printf("%i\n", *((int*) partition2.start_address));

    return 0;
}

[CI/CD] A debug-test job

Add a debug-test job to test debug binaries by compiling, assembling, and testing native binaries as debug.

[Incremental-building-units] Implement an incremental building strategy

The current native build system isn't incremental.

Here is a simple strategy to implement an incremental building system:

  • Use this code first and compile each source C file into a static object file:
└──╼ $gcc -c  test-source.c -o test-static-object.o
  • Each file in the unix system has a LDM (last-date-of-modification) including the source C files and the object code files.
  • In the subsequent build, comparing these metadata (the last date of modification) of both the source code and the static object output will determine if the build will execute or not, here is a simple low-level similar approach:
#!/bin/bash
echo "Test ldm (Last Date of Modification) for incremental building units"
source="./source.c"
function getFormatedFileLdm() {
    local target=$1
    local format=$2
    stat ${target} --format=${format}
    local ldm=$?
    return $ldm
}
function getFileLdm0() {
    local target=$1
    getFormatedFileLdm ${target} %Y
    local ldm=$?
    return $ldm
}
function saveFileLdm() {
    local target="${1}-ldm.txt"
    local ldm=$2
    echo "${ldm}" > "${target}"
    return $?
}
function loadFileLdm() {
    local target="${1}-ldm.txt"
    cat $target
    return $?
}
function isModified() {
    local ldm0=$1
    local ldm1=$2
    let elapsed_ldm=$ldm1-$ldm0   
    if (( $elapsed_ldm > 0 )); then
        echo "Compiling now ...."
        gcc $source --debug -o "${source}.o"
    elif (( $elapsed_ldm == 0 )); then
        echo "File not changed, leaving now ...."
    else 
        echo "State undefined ...."
    fi
}
# prepare LDMs (Last-Dates-Of-Modifications)
current_ldm=`getFileLdm0 $source`
last_ldm=`loadFileLdm $source`
# test if the file is modified 
isModified $last_ldm $current_ldm
# update the file ldm with current new ldm
saveFileLdm $source $current_ldm
  • The compilation should end by a shared position-independent shared object file:
└──╼ $gcc -shared -fPIC test-static-object.o -o test-shared-object.so

[Docs] NativeBufferUtils#memoryMove() doesn't manipulate bytes

NativeBufferUtils#memoryMove(ByteBuffer, ByteBuffer, long) uses the GNU libc void * memmove ( void *to, const void *from, size_t size ) from string.h which according to the documentation:

memmove copies the size bytes at from into the size bytes at to, even if those two blocks
of space overlap. In the case of overlap, memmove is careful to copy the original values
of the bytes in the block at from, including those bytes which also belong to the block
at to.
The value returned by memmove is the value of to.

The workaround is to remove the function memoryMove and internally implement the memmove() for the java side memoryCopy().

[GC] Attach DirectBuffers to GC

Example to utilizing the java.lang.ref.PhantomReference<T>.

Note: the following is a pseudo-code intended to show off the interface and the implementation design.

  1. GarbageCollectibleBuffer: A GC PhantomReference wrapping a native buffer.
public class GarbageCollectibleBuffer extends PhantomReference<Buffer> {
      private final Buffer referent;
      private final long address;
      
      private GarbageCollectibleBuffer(final Buffer referent, final ReferenceQueue<? super Buffer> queue) {
              super(referent, queue);
              this.referent = referent;
              this.address = NativeBufferUtils.getMemoryAddress(buffer);
      }

      public static GarbageCollectibleBuffer from(final Buffer buffer, final ReferenceQueue<? super Buffer> queue) {
             if (!buffer.isDirect()) {
                throw new UnSupportedBufferException("Target Buffer isnot a direct Buffer!");
             }
             return new GarbageCollectibleBuffer(buffer, queue);
      }
      
      public static GarbageCollectibleBuffer allocateDirect(final long size, final ReferenceQueue<? super Buffer> queue) {
              final ByteBuffer buffer = NativeBufferUtils.clearAlloc(size);
              return GarbageCollectibleBuffer.from(buffer, queue);       
      }
      
      public void deallocate() {
            NativeBufferUtils.destroy(referent);
      }
      
      public Buffer getReferent() {
           return referent;     
      }
      
      public long getDirectBufferAddress() {
           return address;
      }
}
  1. GarbageCollectableBuffers: A collection of direct collectible buffers.
public final class GarbageCollectibleBuffers {
      private static ReferenceQueue<Buffer> COLLECTIBLES = new ReferenceQueue<>();
      private final List<GarbageCollectableBuffer> buffers = new ArrayList<>();
      
      private GarbageCollectibleBuffers() {
      }
      
      public static GarbageCollectibleBuffer allocate(final long size) {
            final GarbageCollectibleBuffer collectible = GarbageCollectibleBuffer.allocateDirect(size, COLLECTIBLES);
            buffers.add(collectible);
            return collectible;
      }
      
      public static GarbageCollectibleBuffer register(final Buffer buffer) {
            if (!buffer.isDirect()) {
                   throw new UnSupportedBufferException("Buffer isn't a direct buffer!");
            }
            final GarbageCollectibleBuffer collectible = GarbageCollectibleBuffer.from(buffer, COLLECTIBLES);
            buffers.add(collectible);
            return collectible;
      }

      public static void deallocate(final Buffer buffer) {
            if (!buffers.contains(buffer)) { 
                  throw new UnSupportedBufferException("Buffer is not a direct collectible!");
            } 
            NativeBufferUtils.destroy(buffer);
            buffers.remove(collectible);
      }
      
      public static MemoryScavenger startMemoryScavenger() {
          return MemoryScavenger.start(COLLECTIBLES);
      }
}
  1. MemoryScavenger: A memory cleaner to help GC scavenge the native memory.
public final class MemoryScavenger extends Thread {
       private final ReferenceQueue<? super Buffer> queue;
        
       private MemoryScavenger(final ReferenceQueue<? super Buffer> queue) {
            super(MemoryScavenger.class.getName());
            setDaemon(true);
            this.queue = queue;
       }
       
       public static MemoryScavenger start(final ReferenceQueue<? super Buffer> queue) { 
            final MemoryScavenger scavenger = new MemoryScavenger(queue);
            scavenger.start();
            return scavenger;
       }
       
       @Override
       public void run() {
            for (;;) {
                // blocks until an object is available in the queue before returning and removing it
                // object references are added to the queue by the GC as a part of post-mortem actions
                GarbageCollectibleBuffer collectible = (GarbageCollectibleBuffer) queue.remove();
                collectible.deallocate();
            }
       }
}

------------------------------------------------ JME ------------------------------------------------

  1. Implementing the above interface in jme would be super-easy:

LwjglBufferAllocator: A direct buffer allocator/deallocator for LWJGL-2.

public class LwjglBufferAllocator implements BufferAllocator {
        static {
            GarbageCollectibleBuffers.startScavenger();
        }
        
        @Override
        public ByteBuffer allocate(final int size) {
            return GarbageCollectibleBuffer.register(size).getReferent();
        }
        
        @Override
        public void destroyDirectBuffer(final Buffer buffer) {
            NativeBufferAllocator.release(buffer);
        }
}

AndroidNativeBufferAllocator: A native buffer allocator/deallocator for android.

public class AndroidNativeBufferAllocator implements BufferAllocator {
        static {
            GarbageCollectibleBuffers.startScavenger();
        }
        
        @Override
        public ByteBuffer allocate(final int size) {
            return GarbageCollectibleBuffers.allocate(size).getReferent();
        }
        
        @Override
        public void destroyDirectBuffer(final Buffer buffer) {
             GarbageCollectibleBuffers.deallocate(buffer);
        }
}

[Incompatibility] ClassCastException when used in JME

Mar 24, 2023 5:16:29 PM com.jme3.system.JmeDesktopSystem initialize
INFO: Running on jMonkeyEngine 3.6.0-stable
 * Branch: HEAD
 * Git Hash: 53f2a49
 * Build Date: 2023-03-20
Mar 24, 2023 5:16:31 PM com.jme3.system.lwjgl.LwjglContext printContextInitInfo
INFO: LWJGL 2.9.5 context running on thread jME3 Main
 * Graphics Adapter: null
 * Driver Version: null
 * Scaling Factor: 1
Mar 24, 2023 5:16:31 PM com.jme3.renderer.opengl.GLRenderer loadCapabilitiesCommon
INFO: OpenGL Renderer Information
 * Vendor: Intel Open Source Technology Center
 * Renderer: Mesa DRI Intel(R) HD Graphics 3000 (SNB GT2)
 * OpenGL Version: 3.3 (Core Profile) Mesa 20.0.8
 * GLSL Version: 3.30
 * Profile: Core
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio Renderer Information
 * Device: OpenAL Soft
 * Vendor: OpenAL Community
 * Renderer: OpenAL Soft
 * Version: 1.1 ALSOFT 1.15.1
 * Supported channels: 64
 * ALC extensions: ALC_ENUMERATE_ALL_EXT ALC_ENUMERATION_EXT ALC_EXT_CAPTURE ALC_EXT_DEDICATED ALC_EXT_disconnect ALC_EXT_EFX ALC_EXT_thread_local_context ALC_SOFT_loopback
 * AL extensions: AL_EXT_ALAW AL_EXT_DOUBLE AL_EXT_EXPONENT_DISTANCE AL_EXT_FLOAT32 AL_EXT_IMA4 AL_EXT_LINEAR_DISTANCE AL_EXT_MCFORMATS AL_EXT_MULAW AL_EXT_MULAW_MCFORMATS AL_EXT_OFFSET AL_EXT_source_distance_model AL_LOKI_quadriphonic AL_SOFT_buffer_samples AL_SOFT_buffer_sub_data AL_SOFTX_deferred_updates AL_SOFT_direct_channels AL_SOFT_loop_points AL_SOFT_source_latency
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
WARNING: Pausing audio device not supported.
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio effect extension version: 1.0
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio max auxiliary sends: 4
Allocator=buffer.TestReleaseDirectMemory$JmeNativeAlloc
Mar 24, 2023 5:16:32 PM com.jme3.app.LegacyApplication handleError
SEVERE: Uncaught exception thrown in Thread[#33,jME3 Main,5,main]
java.lang.ClassCastException: class java.nio.DirectFloatBufferU cannot be cast to class java.nio.ByteBuffer (java.nio.DirectFloatBufferU and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
	at buffer.TestReleaseDirectMemory$JmeNativeAlloc.destroyDirectBuffer(TestReleaseDirectMemory.java:80)
	at com.jme3.util.BufferUtils.destroyDirectBuffer(BufferUtils.java:1292)
	at buffer.TestReleaseDirectMemory.simpleUpdate(TestReleaseDirectMemory.java:73)
	at com.jme3.app.SimpleApplication.update(SimpleApplication.java:261)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.runLoop(LwjglAbstractDisplay.java:160)
	at com.jme3.system.lwjgl.LwjglDisplay.runLoop(LwjglDisplay.java:224)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:242)
	at java.base/java.lang.Thread.run(Thread.java:1589)

Here is a test case:

public class TestReleaseDirectMemory extends SimpleApplication {

    public static void main(String[] args){
        System.setProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION, JmeNativeAlloc.class.getName());
        TestReleaseDirectMemory app = new TestReleaseDirectMemory();
        app.start();
    }

    @Override
    public void simpleInitApp() {
  
        System.out.println("Allocator=" + System.getProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION));
       
    }

    @Override
    public void simpleUpdate(float tpf) {
        ByteBuffer buf = BufferUtils.createByteBuffer(500000);
        BufferUtils.destroyDirectBuffer(buf);
        
        FloatBuffer buf2 = BufferUtils.createFloatBuffer(500000);
        BufferUtils.destroyDirectBuffer(buf2);
    }

    public static class JmeNativeAlloc implements BufferAllocator {

        @Override
        public void destroyDirectBuffer(Buffer toBeDestroyed) {
            NativeBufferAllocator.releaseDirectByteBuffer((ByteBuffer) toBeDestroyed);
        }

        @Override
        public ByteBuffer allocate(int size) {
            return NativeBufferAllocator.createDirectByteBuffer(size);
        }
    }
}

[Build-Bug] Invalid windows application when running the compile-x86

When running the ./helper-scripts/compile-x86.sh on the windows-latest x86-64 runner image, this error fires up:

A problem occurred evaluating project ':jme3-alloc-native'.
> Cannot run program "./helper-scripts/compile-x86.sh": CreateProcess error=193, %1 is not a valid Win32 application

Workaround:

On the jme3-alloc-native:compileX86 gradle task, add a bash command.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.