apache / mnemonic

Apache Mnemonic - A non-volatile hybrid memory storage oriented library

================================

Mnemonic Official Website

Apache Mnemonic is a non-volatile hybrid memory storage oriented library, it proposed a non-volatile/durable Java object model and durable computing service that bring several advantages to significantly improve the performance of massive real-time data processing/analytics. developers are able to use this library to design their cache-less and SerDe-less high performance applications.

Features:

• In-place data storage on local non-volatile memory
• Durable Object Model (DOM)
• Durable Native Computing Model (DNCM)
• Object graphs lazy loading & sharing
• Auto-reclaim memory resources and Mnemonic objects
• Hierarchical cache pool for massive data caching
• Extensible memory services for new device adoption and allocation optimization
• Durable data structure collection(WIP)
• Durable computing service
• Minimize memory footprint of on-heap
• Reduce GC Overheads as the following chart shown (collected from Apache Spark experiments)
• Drop-in Hadoop MapReduce support
• Drop-in Hadoop Spark support

mnemonic's Issues

Expose memory allocator with rust lang

Our current memory allocator is c-based, we could explore the possibility of rust memory allocator to improve the security and performance

Github issue history get lost

After I re-activated the project link to Github, the issue history get lost, so I think we have to remove the issue id info from commit title e.g. [MNEMONIC-], the document needs to get updated to reflect this changes. any ideas are welcome. thanks.

Refactor the project to support OpenJDK 11

I recall the similar issue has been submitted before, but we lost our issue history, so I re-submit it as a top-level ticket that should break down into several sub tasks to fulfill. it could be a milestone if OpenJDK11 is supported.

Failed to execute Antrun job for module mnemonic-nvml-vmem-service

The project building shows the following error with openJDK 11

[INFO] --- maven-antrun-plugin:1.8:run (build-native) @ mnemonic-nvml-vmem-service ---
[INFO] Executing tasks

main:
     [echo] Java Home: /usr/lib/jvm/java-11-openjdk-amd64
     [exec] -- Configuring incomplete, errors occurred!
     [exec] See also "/home/gary/ws/mnemonic/mnemonic-memory-services/mnemonic-nvml-vmem-service/src/main/native/build/CMakeFiles/CMakeOutput.log".
     [exec] See also "/home/gary/ws/mnemonic/mnemonic-memory-services/mnemonic-nvml-vmem-service/src/main/native/build/CMakeFiles/CMakeError.log".
     [exec] CMake Error at CMakeLists.txt:43 (message):
     [exec]   not found vmem library
     [exec] 
     [exec] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.071 s
[INFO] Finished at: 2020-07-18T13:39:46-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (build-native) on project mnemonic-nvml-vmem-service: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="src/main/native/build" executable="cmake">... @ 6:75 in /home/gary/ws/mnemonic/mnemonic-memory-services/mnemonic-nvml-vmem-service/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Fix all test cases after JDK 14 upgrade

Due to the recent JDK 14 upgrade, not all test cases can run smoothly. We need to review and fix all the test cases in our next release (target 0.16.0)

Replace the pmem libraries source building with distributed library packages

Pmem project releases have been managed by Linux Distribution repository e.g. Ubuntu repository, hence, it is possible to get rid of all steps about generating the pmem libs from the source code of pmem related libraries.

For instance, in Ubuntu, we can issue the following commands to install the dependent pmem libraries.

sudo apt install libndctl-dev
sudo apt install libpmem-dev
sudo apt install libvmem-dev
sudo apt install libpmemobj-dev

There is a vulnerability in Apache Hadoop 2.7.3,upgrade recommended

mnemonic/mnemonic-hadoop/mnemonic-hadoop-mapreduce/build.gradle

Line 33 in 6395b5a

def hadoop_version = '2.7.3'

CVE-2021-37404 CVE-2022-25168 CVE-2022-26612 CVE-2017-15718

Recommended upgrade version：2.10.2

Cannot initialize module SuppressionFilter Error

The maven module mnemonic-nvml-vmem-service shows the following errors.

[INFO] --- maven-checkstyle-plugin:3.0.0:check (validate) @ mnemonic-nvml-vmem-service ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.891 s
[INFO] Finished at: 2020-07-18T13:11:39-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.0.0:check (validate) on project mnemonic-nvml-vmem-service: Failed during checkstyle configuration: cannot initialize module SuppressionFilter - Unable to find: suppressions.xml -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Explore the possibility of integration with Analytics-zoo

Analytics-zoo has Spark and Flink integration, which can be potentially integrated with Mnemonic. We can evaluate this possibility, and consider it as another feature candidate for the incoming releases.

There is a vulnerability in Simple Logging Facade for Java (SLF4J) 1.7.25,upgrade recommended

mnemonic/build.gradle

Line 54 in 6395b5a

dependency 'org.slf4j:slf4j-api:1.7.25'

CVE-2018-8088

Recommended upgrade version：1.7.26

There is a vulnerability in Logback 1.1.7 ,upgrade recommended

mnemonic/pom.xml

Line 112 in 2b9db81

<logback.version>1.1.7</logback.version>

CVE-2017-5929
Recommended upgrade version：
1.1.11

Improve the deployment of Maven in Ubuntu Docker file

The steps of Maven deployment and proxy configuration are able to further simplified on Ubuntu 18.06 LTS. from my system, it shows the latest stable versions of Docker and Maven are Docker v19.03 and Maven v3.6.0, there are some associated feature can help to make those steps more elegant.

An error occurred while running notifications feature

I received several email as follows

An error occurred while running notifications feature in .asf.yaml!:
Invalid notification target '[email protected]'. Must be a valid @mnemonic.apache.org list!

That seems be triggered by github CI every time when a new PR got merged.
and the [email protected], seems not exist.

There is a vulnerability in Logback 1.1.7,upgrade recommended

mnemonic/build.gradle

Lines 56 to 57 in 6395b5a

	dependency 'ch.qos.logback:logback-classic:1.1.7'
	dependency 'ch.qos.logback:logback-core:1.1.7'

CVE-2017-5929 CVE-2021-42550

Recommended upgrade version：1.2.8

Update the Ubuntu docker file to adopt pmem packages

The current Ubuntu docker file instructs the building ndctl and pmdk project from source code, that can be replaced with apt command to install dependent pmem devel. libraries directly from Ubuntu package repository, the versions of those packages might be a little bit out of date but it is the best for compatibility and stability corresponding to specific Ubuntu distribution.

Linked to #125

Improve the deployment of Maven in CentOS Docker file

The steps of Maven deployment and proxy configuration are able to further simplified on Cetnos 8. from the Docker container, it shows the latest stable version of Maven is v3.5.4.
Regarding the Docker, I search out that it is possible to deploy v18.09.2 on CentOS 8, that should be good enough.

Review and evaluate 3rd party integration status, upgrade if neccessary

Mnemonic is currently integrated with Hadoop and Spark for drop-in support, but the integrated versions for both are not synced up-to-date. We need to review the current status for those 3rd party integration, see if we need to support them in their latest release.

Redis integration with Mnemonic

We plan to support Redis as additional backend of Mnemonic storage infrastructure. Targeting design complete for 0.16.0 release

Update the CentOS docker file to adopt pmem packages

The current CentOS docker file instructs the building ndctl and pmdk project from source code, that can be replaced with apt command to install dependent pmem devel. libraries directly from Ubuntu package repository, the versions of those packages might be a little bit out of date but it is the best for compatibility and stability corresponding to specific CentOS distribution.

Linked to #125