xuanwo / hdfs-sys

Bindings to HDFS Native C API.

hdfs-sys  

Bindings to HDFS Native C API.

Work with these bindings directly is boring and error proven, please use hdrs instead if ever possible.

Supported Platforms

• Linux
• macOS
• Windows

Supported Versions

To target a version of libhdfs, enable a Cargo features such as one of the following:

• hdfs_2_2 : requires hdfs 2.2 or later releases.
• hdfs_2_3: requires hdfs 2.3 or later releases.
• hdfs_2_4: requires hdfs 2.4 or later releases.
• hdfs_2_5: requires hdfs 2.5 or later releases.
• hdfs_2_6: (default) requires hdfs 2.6 or later releases.
• hdfs_2_7: requires hdfs 2.7 or later releases.
• hdfs_2_8: requires hdfs 2.8 or later releases.
• hdfs_2_9: requires hdfs 2.9 or later releases.
• hdfs_2_10: requires hdfs 2.10 or later releases.
• hdfs_3_0: requires hdfs 3.0 or later releases.
• hdfs_3_1: requires hdfs 3.1 or later releases.
• hdfs_3_2: requires hdfs 3.2 or later releases.
• hdfs_3_3: requires hdfs 3.3 or later releases.

Please note:

• If you do not enable one of these features, the API provided by hdfs_2_6 will be available by default.
• Enable one feature will also enable all features before it. For example, enable hdfs_2_4 will also enable hdfs_2_3 and hdfs_2_2.
• Too old version of hdfs could contain bugs or can't compile on your platform.

Compile

hdfs-sys supports both dynamic link, static link and vendor:

• If vnedored feature has been enabled, hdfs-sys will compile and link libhdfs in static.
• Use HDFS_LIB_DIR to specify the path of libhdfs.so or libhdfs.a
• Use HDFS_STATIC=1 to choose to switch between dynamic link and static link
• If HDFS_LIB_DIR is not set, we will try to find ${HADOOP_HOME}/lib/native
• If all env are empty, we will try to compile libhdfs and link it in static

Dependencies

This crate will link to libjvm dynamically.

To make this crate works correctly, please make sure the following env set correctly:

• JAVA_HOME: hdfs-sys will search path like ${JAVA_HOME}/lib/server to link libjvm.

NOTE: hdfs-sys will ignore linking if DOCS_RS is set to build docs.

Runtime

hdfs-sys uses JNI to call functions provided by jars that provided by hadoop releases. Please make sure CLASSPATH is set correctly before calling any functions provided by hdfs-sys:

export JAVA_HOME=/path/to/java
export HADOOP_HOME=/path/to/hadoop
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server
export CLASSPATH=$(find $HADOOP_HOME -iname "*.jar" | xargs echo | tr ' ' ':')

Contributing

Check out the CONTRIBUTING.md guide for more details on getting started with contributing to this project.

Getting help

Submit issues for bug report or asking questions in discussion.

Related Projects

• hdfs-rs: libhdfs binding and wrapper APIs for Rust, not maintained since 2015.
• fs-hdfs: forks based on hdfs-rs, C files are from hadoop 2.7.3.
• dataFusion-hdfs-native: bindings of libhdfs3

Acknowledgment

This project is highly inspired by clang-sys

License

^{Licensed under Apache License, Version 2.0.}

hdfs-sys's Issues

Add links to related projects

• https://github.com/yahoNanJing/fs-hdfs
• https://github.com/hyunsik/hdfs-rs/
• https://github.com/datafusion-contrib/datafusion-hdfs-native

Any plan to support kerberos authentication?

As I know, the libhdfs itself cannot support kerberos authtication.
So is there any way to do it? Thanks.

Support runtime loading

Add test for hdfs 2.6

Tracking issue of windows support

Lacking dirent.h support

Include https://raw.githubusercontent.com/tronkko/dirent/master/include/dirent.h directly.

Can't build hdfs_3_3

I can't find a way to work around it.

running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX[64](https://github.com/Xuanwo/hdfs-sys/runs/6319524578?check_suite_focus=true#step:3:64)\\x64\\cl.exe" "-nologo" "-MT" "-Z7" "-Brepro" "-static" "-I" "C:\\hostedtoolcache\\windows\\Java_Temurin-Hotspot_jdk\\8.0.322-6\\x64/include" "-I" "C:\\hostedtoolcache\\windows\\Java_Temurin-Hotspot_jdk\\8.0.322-6\\x64/include/win32" "-I" "libhdfs" "-I" "libhdfs/hdfs_3_3" "-I" "libhdfs/hdfs_3_3/os" "-I" "libhdfs/hdfs_3_3/os/windows" "-I" "libhdfs/hdfs_3_3/include" "-I" "dirent/include" "-O2" "/W4" "/wd4100" "/wd4127" "-D_CRT_NONSTDC_NO_DEPRECATE" "-D_CRT_SECURE_NO_WARNINGS" "-DWIN32_LEAN_AND_MEAN" "-FoD:\\a\\hdfs-sys\\hdfs-sys\\target\\debug\\build\\hdfs-sys-eacfa2682459cbbb\\out\\libhdfs/hdfs_3_3/jni_helper.o" "-c" "libhdfs/hdfs_3_3/jni_helper.c"
  cargo:warning=cl : Command line warning D9002 : ignoring unknown option '-static'
  jni_helper.c
  libhdfs/hdfs_3_3/jni_helper.c(383): error C2061: syntax error: identifier 'wildcard_expandPath'
  libhdfs/hdfs_3_3/jni_helper.c(383): error C2059: syntax error: ';'
  libhdfs/hdfs_3_3/jni_helper.c(383): error C2059: syntax error: '<parameter-list>'
  libhdfs/hdfs_3_3/jni_helper.c(490): error C2061: syntax error: identifier 'getClassPath_helper'
  libhdfs/hdfs_3_3/jni_helper.c(490): error C2059: syntax error: ';'
  libhdfs/hdfs_3_3/jni_helper.c(490): error C2059: syntax error: '<parameter-list>'
  libhdfs/hdfs_3_3/jni_helper.c(570): error C20[65](https://github.com/Xuanwo/hdfs-sys/runs/6319524578?check_suite_focus=true#step:3:65): 'ssize_t': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(570): error C2146: syntax error: missing ';' before identifier 'length'
  libhdfs/hdfs_3_3/jni_helper.c(570): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(571): error C2065: 'ssize_t': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(571): error C2146: syntax error: missing ';' before identifier 'retval'
  libhdfs/hdfs_3_3/jni_helper.c(571): error C2065: 'retval': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(580): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(580): warning C4013: 'getClassPath_helper' undefined; assuming extern returning int
  libhdfs/hdfs_3_3/jni_helper.c(581): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(594): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(613): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(620): error C2065: 'retval': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(621): error C2065: 'retval': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(627): error C2065: 'retval': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(627): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(630): error C2065: 'length': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(630): error C2065: 'retval': undeclared identifier
  libhdfs/hdfs_3_3/jni_helper.c(629): warning C4477: 'fprintf' : format string '%zu' requires an argument of type 'unsigned __int64', but variadic argument 1 has type 'int'
  libhdfs/hdfs_3_3/jni_helper.c(629): note: consider using '%u' in the format string
  libhdfs/hdfs_3_3/jni_helper.c(629): note: consider using '%I32u' in the format string
  libhdfs/hdfs_3_3/jni_helper.c(629): warning C4477: 'fprintf' : format string '%zu' requires an argument of type 'unsigned __int64', but variadic argument 2 has type 'int'
  libhdfs/hdfs_3_3/jni_helper.c(629): note: consider using '%u' in the format string
  libhdfs/hdfs_3_3/jni_helper.c(629): note: consider using '%I32u' in the format string
  exit code: 2

  --- stderr


  error occurred: Command "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MT" "-Z7" "-Brepro" "-static" "-I" "C:\\hostedtoolcache\\windows\\Java_Temurin-Hotspot_jdk\\8.0.322-6\\x64/include" "-I" "C:\\hostedtoolcache\\windows\\Java_Temurin-Hotspot_jdk\\8.0.322-6\\x64/include/win32" "-I" "libhdfs" "-I" "libhdfs/hdfs_3_3" "-I" "libhdfs/hdfs_3_3/os" "-I" "libhdfs/hdfs_3_3/os/windows" "-I" "libhdfs/hdfs_3_3/include" "-I" "dirent/include" "-O2" "/W4" "/wd4100" "/wd4127" "-D_CRT_NONSTDC_NO_DEPRECATE" "-D_CRT_SECURE_NO_WARNINGS" "-DWIN32_LEAN_AND_MEAN" "-FoD:\\a\\hdfs-sys\\hdfs-sys\\target\\debug\\build\\hdfs-sys-eacfa2[68](https://github.com/Xuanwo/hdfs-sys/runs/6319524578?check_suite_focus=true#step:3:68)2459cbbb\\out\\libhdfs/hdfs_3_3/jni_helper.o" "-c" "libhdfs/hdfs_3_3/jni_helper.c" with args "cl.exe" did not execute successfully (status code exit code: 2).

Add runtime libloading support

Take https://github.com/KyleMayes/clang-sys/blob/master/src/link.rs as reference. We can load libhdfs during runtime instead.

How about using JNI directly?

Can we link against libhdfs3?

Support compile dynamic lib

Add APIs that added in HDFS 3.3.2

Add APIs that added in HDFS 3.2.3

Build static libs if we are on macos/windows?

rename can break seek(SeekFrom::End())

Seek to end reads file metadata to determine the file length. If the file has been renamed between the open and the seek, the seek will fail. If the file has been renamed and the original name has been reused, the seek may succeed but point to the wrong position.

A couple of options:

1. read file metadata (size, last modification time)
2. open the file
3. reread file metadata (size, last modification time)
4. fail the open of the values from step 1 and step 3 do not match.

(Relies on hdfsSeek returning an error for seeking beyond end of file. That seems to be the case, but is not documented.)

1. read file metadata (size)
2. open the file
3. perform a search (series of calls to hdfsSeek) to determine the actual file size, using the size value from step 1 as a hint.

Either one seems like a lot of work to get a value which HDFS probably knows internally.

Support link to dynamic lib

Export all hdfs.h from hadoop repo

https://github.com/apache/hadoop/commits/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/hdfs.h

For example, user may needs hdfs 2.6.

Add support for WebHDFS