Connecting DataFusion to HDFS through Native HDFS client (libhdfs3).
- Install libhdfs3
You can either install it via Conda
conda install -c conda-forge libhdfs3or build it from source
# A specific version that could be compiled on osx for HDFS of 2.6.x version
git clone https://github.com/ClickHouse-Extras/libhdfs3.git
cd libhdfs3
git checkout 24b058c356794ef6cc2d31323dc9adf0386652ff
# then build it
mkdir build && cd build
../bootstrap --prefix=/usr/local
make
make install# client conf to use, env LIBHDFS3_CONF or hdfs-client.xml in working directory
export LIBHDFS3_CONF=/path/to/libhdfs3-hdfs-client.xml