Programming Language
- Rust
- Python :: Implementation :: CPython
- Python :: Implementation :: PyPy
Native HDFS Python bindings based on hdfs-native Rust package.
Installation
pip install hdfs-native
Example
from hdfs_native import Client
client = Client("hdfs://localhost:9000")
status = client.get_file_info("/file.txt")
To use explicit Kerberos credentials, pass the principal and optionally a
keytab and/or credential cache to either Client or AsyncClient:
from hdfs_native import Client
client = Client(
"hdfs://namenode.example.com:9000",
kerberos_principal="client@EXAMPLE.COM",
kerberos_keytab="/run/secrets/client.keytab",
)
kerberos_principal is required when kerberos_keytab or kerberos_cache is
provided. A keytab without an explicit cache uses a native in-memory cache.
The same arguments are available on AsyncClient.
CLI
There is a built-in CLI hdfsn that implements most of the behavior of hdfs dfs but with a more bash-like syntax. The easiest way to use the CLI is with UV.
Install CLI with UV
uv tool install hdfs-ntaive
Auto-complete support
The CLI supports auto-complete for HDFS paths using argcomplete. There are two ways to enable this support
To permanently enable support for all Python modules using argcomplete:
uv tool install argcomplete
activate-global-python-argcomplete
To enable support just for hdfsn in your active shell:
uv tool install argcomplete
eval "$(register-python-argcomplete hdfsn)"
Kerberos support
Kerberos (SASL GSSAPI) is supported through a runtime dynamic link to libgssapi_krb5. This must be installed separately, but is likely already installed on your system. If not you can install it by:
Debian-based systems
apt-get install libgssapi-krb5-2
RHEL-based systems
yum install krb5-libs
MacOS
brew install krb5
Running tests
The same requirements apply as the Rust tests, requiring Java, Maven, Hadoop, and Kerberos tools to be on your path. Then you can:
python3 -m venv .venv
source .venv/bin/activate
pip3 install maturin
maturin develop -E devel
pytest
