aarch64 image about hadoop-hdfs-fsimage-exporter

[marcelmay]

It's difficult to test, as I do not have access to aarch64 environment.

But I modified the build so that you can choose a different base image.

Here's an example using an aarm64v8 openjdk base image:

 mvn clean install -DskipITs 
       -Ddocker.base.image=arm64v8/openjdk:11-jre-slim-buster
       -Dimage.name=marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8
       -Pdocker

[tulid]

i conduct the command，but the error appear as blow：
... ...
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] 未与 -source 8 一起设置引导类路径
[WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[141,15] 找到原始类型: i o.prometheus.client.SimpleCollector
缺少泛型类io.prometheus.client.SimpleCollector的类型参数
[WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[145,15] 找到原始类型: i o.prometheus.client.SimpleCollector
缺少泛型类io.prometheus.client.SimpleCollector的类型参数
[WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[149,15] 找到原始类型: i o.prometheus.client.SimpleCollector
缺少泛型类io.prometheus.client.SimpleCollector的类型参数
[WARNING] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[158,15] 找到原始类型: i o.prometheus.client.SimpleCollector
缺少泛型类io.prometheus.client.SimpleCollector的类型参数
[INFO] 5 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[17,39] 程序包org.apache.h adoop.fs.permission不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[18,46] 程序包org.apache.h adoop.hdfs.server.namenode不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[491,53] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[499,58] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[504,56] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[302,57] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[327,62] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[349,60] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[303,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,17] 找不到符号
符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,49] 无法访问org.apach e.hadoop.hdfs.server.namenode.FsImageProto.INodeSection
找不到org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeSection的类文件
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[328,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[329,17] 找不到符号
符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[350,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[351,17] 找不到符号
符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[492,38] 程序包FsImageProt o.INodeSection不存在
[INFO] 16 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.165 s
[INFO] Finished at: 2020-11-23T13:37:36+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project fsimage-exporter: Compilat ion failure: Compilation failure:
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[17,39] 程序包org.apache.h adoop.fs.permission不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[18,46] 程序包org.apache.h adoop.hdfs.server.namenode不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[491,53] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[499,58] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[504,56] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[302,57] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[327,62] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[349,60] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[303,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,17] 找不到符号
[ERROR] 符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[304,49] 无法访问org.apach e.hadoop.hdfs.server.namenode.FsImageProto.INodeSection
[ERROR] 找不到org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeSection的类文件
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[328,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[329,17] 找不到符号
[ERROR] 符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[350,42] 程序包FsImageProt o.INodeSection不存在
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[351,17] 找不到符号
[ERROR] 符号: 类 PermissionStatus
[ERROR] /root/jinquan/hadoop-hdfs-fsimage-exporter/src/main/java/de/m3y/prometheus/exporter/fsimage/FsImageReporter.java:[492,38] 程序包FsImageProt o.INodeSection不存在
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[marcelmay]

That's strange. Can you provide the output of mvn -v ?

[tulid]

mvn -vas blow:
root@uos-test-1:~/jinquan/hadoop-hdfs-fsimage-exporter# mvn -v
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/apache-maven-3.6.3
Java version: 11.0.4, vendor: Debian, runtime: /usr/lib/jvm/java-11-openjdk-arm64
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "4.19.0-arm64-server", arch: "aarch64", family: "unix"

[marcelmay]

The versions seem ok - the JVM 11 is not at latest patch level (eg 11.0.9+) but should work.

The compiler complains it can not find the Hadoop classes. It should have complained that it can not download these, but not during compilation.

Can you check the JARs, eg if INodeSection exists:

jar vft ~/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar|grep INodeSection.class

That should print something like

17813 Mon Jul 06 18:50:24 CEST 2020 org/apache/hadoop/hdfs/server/namenode/FsImageProto$INodeSection.class

I managed to run the (experimental) docker image marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8 locally via docker and qemu arm64 emulator, the exporter using the arm64 openjdk image seems to work:

docker run --rm -t -p 9709:9709 -v $PWD/src/test/resources:/fsimage-location \
    -e "JAVA_OPTS=-server -XX:+UseG1GC -Xmx1024m" \
    marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8

[tulid]

i run the command as you said,but its result is an error as blow. can u build one image that tag is 1.2 for my emergency. thanks a lot

root@uos-test-1:~# jar vft ~/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar|grep INodeSection.class
java.nio.file.NoSuchFileException: /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.3.0/hadoop-hdfs-3.3.0.jar
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145)
at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225)
at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:727)
at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844)
at java.base/java.util.zip.ZipFile.(ZipFile.java:247)
at java.base/java.util.zip.ZipFile.(ZipFile.java:177)
at java.base/java.util.zip.ZipFile.(ZipFile.java:148)
at jdk.jartool/sun.tools.jar.Main.list(Main.java:1500)
at jdk.jartool/sun.tools.jar.Main.run(Main.java:380)
at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)

[marcelmay]

Somehow there are no dependencies, there should have been more warnings when Maven tries to download these.

Can you try the docker image marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8 of current latest head, or do require exactly 1.2 version (latest is 1.4)?

docker pull marcelmay/hadoop-hdfs-fsimage-exporter-arm64v8:1.5-SNAPSHOT

[marcelmay]

Available on https://hub.docker.com/r/marcelmay/hadoop-hdfs-fsimage-exporter/tags