The world's fastest WARC parsing library written in Rust with bindings for Python.
Project Links
Meta
Author: Janek Bevendorff
Requires Python: >=3.10
Classifiers
FastWARC (Python)
FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python. FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip, Zstd, and LZ4.
This package contains the Python bindings for FastWARC.
FastWARC belongs to the ChatNoir Resiliparse toolkit for fast and robust web data processing.
Installing FastWARC
Pre-built FastWARC binaries for Python can be installed from PyPi:
pip install fastwarc
Building Python Bindings From Source
To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub repository, follow these steps:
# Clone repository
git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
cd chatnoir-resiliparse
# Install Rust toolchain (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Create a fresh venv first (recommended)
python3 -m venv venv && source venv/bin/activate
# Option 1: Build and install in editable mode (best for development)
python3 -m pip install -e ./fastwarc-py
# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
python3 -m pip wheel -w build ./fastwarc-py
ls ./build/*.whl | xargs python3 -m pip install
Usage Instructions
For detailed usage instructions, please consult the FastWARC User Manual.
Cite Us
If you use FastWARC, please consider citing our OSSYM 2021 abstract paper:
@InProceedings{bevendorff:2021,
author = {Janek Bevendorff and Martin Potthast and Benno Stein},
booktitle = {3rd International Symposium on Open Search Technology (OSSYM 2021)},
editor = {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
month = oct,
publisher = {International Open Search Symposium},
site = {CERN, Geneva, Switzerland},
title = {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
year = 2021
}
You can also cite our ECIR 2018 demo paper:
@InProceedings{bevendorff:2018,
address = {Berlin Heidelberg New York},
author = {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
booktitle = {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
editor = {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
month = mar,
publisher = {Springer},
series = {Lecture Notes in Computer Science},
site = {Grenoble, France},
title = {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
year = 2018
}
1.0.9
Jul 20, 2026
1.0.8
Jun 15, 2026
1.0.7
Jun 11, 2026
1.0.6
Jun 10, 2026
1.0.5
Jun 09, 2026
1.0.4
Jun 08, 2026
1.0.3
May 29, 2026
1.0.2
May 28, 2026
1.0.1
May 26, 2026
1.0.0
May 24, 2026
0.16.0
Apr 02, 2026
0.15.2
Mar 27, 2025
0.15.1
Dec 05, 2024
0.15.0
Dec 04, 2024
0.14.9
Aug 13, 2024
0.14.8
Aug 09, 2024
0.14.7
Apr 29, 2024
0.14.6
Apr 04, 2024
0.14.5
Aug 11, 2023
0.14.3
Mar 22, 2023
0.14.2
Mar 07, 2023
0.14.1
Mar 06, 2023
0.14.0
Mar 06, 2023
0.13.7
Nov 01, 2022
0.13.6
Nov 01, 2022
0.13.5
Sep 15, 2022
0.13.4
Sep 15, 2022
0.13.3
Sep 14, 2022
0.13.2
Sep 14, 2022
0.13.1
Sep 14, 2022
0.13.0
Sep 14, 2022
0.12.2
Jun 10, 2022
0.12.1
May 04, 2022
0.12.0
Mar 18, 2022
0.11.2
Mar 16, 2022
0.11.1
Mar 16, 2022
0.11.0
Mar 15, 2022
0.10.6
Mar 11, 2022
0.10.5
Feb 28, 2022
0.10.4
Jan 27, 2022
0.10.3
Jan 27, 2022
0.10.2
Dec 17, 2021
0.10.1
Dec 16, 2021
0.10.0
Dec 09, 2021
0.9.0
Dec 06, 2021
0.8.1
Dec 03, 2021
0.8.0
Dec 02, 2021
0.7.2
Nov 30, 2021
0.7.1
Nov 30, 2021
0.7.0
Nov 23, 2021
0.6.6
Nov 23, 2021
0.6.5
Nov 09, 2021
0.6.4
Nov 08, 2021
0.6.3
Nov 03, 2021
0.6.2
Oct 11, 2021
0.6.1
Oct 04, 2021
0.6.0
Oct 04, 2021
0.5.0
Sep 23, 2021
0.4.2
Sep 13, 2021
0.4.1
Sep 12, 2021
0.4.0
Sep 12, 2021
0.3.15
Aug 23, 2021
0.3.14
Jul 17, 2021
0.3.13
Jul 17, 2021
0.3.12
Jul 17, 2021
0.3.11
Jul 16, 2021
0.3.10
Jul 12, 2021
0.3.9
Jul 12, 2021
0.3.8
Jul 09, 2021
0.3.7
Jul 08, 2021
0.3.6
Jul 08, 2021
0.3.5
Jul 08, 2021
0.3.4
Jul 08, 2021
0.3.3
Jul 07, 2021
0.3.2
Jul 07, 2021
0.3.1
Jul 07, 2021
0.3.0
Jul 07, 2021
0.2.10
Jul 06, 2021
0.2.9
Jul 06, 2021
0.2.8
Jul 05, 2021
0.2.7
Jul 05, 2021
0.2.6
Jul 05, 2021
0.2.5
Jun 25, 2021
0.2.4
Jun 25, 2021
0.2.3
Jun 25, 2021
0.2.2
Jun 24, 2021
0.2.1
Jun 24, 2021
0.2
Jun 24, 2021
Wheel compatibility matrix
Files in release
fastwarc-1.0.9-cp310-cp310-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp310-cp310-macosx_12_0_x86_64.whl (2.1MiB)
fastwarc-1.0.9-cp310-cp310-manylinux_2_28_aarch64.whl (3.8MiB)
fastwarc-1.0.9-cp310-cp310-manylinux_2_28_x86_64.whl (3.9MiB)
fastwarc-1.0.9-cp310-cp310-win_amd64.whl (1.9MiB)
fastwarc-1.0.9-cp311-cp311-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp311-cp311-macosx_12_0_x86_64.whl (2.1MiB)
fastwarc-1.0.9-cp311-cp311-manylinux_2_28_aarch64.whl (3.9MiB)
fastwarc-1.0.9-cp311-cp311-manylinux_2_28_x86_64.whl (3.9MiB)
fastwarc-1.0.9-cp311-cp311-win_amd64.whl (1.9MiB)
fastwarc-1.0.9-cp312-cp312-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp312-cp312-macosx_12_0_x86_64.whl (2.0MiB)
fastwarc-1.0.9-cp312-cp312-manylinux_2_28_aarch64.whl (3.8MiB)
fastwarc-1.0.9-cp312-cp312-manylinux_2_28_x86_64.whl (3.9MiB)
fastwarc-1.0.9-cp312-cp312-win_amd64.whl (1.9MiB)
fastwarc-1.0.9-cp313-cp313-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp313-cp313-macosx_12_0_x86_64.whl (2.0MiB)
fastwarc-1.0.9-cp313-cp313-manylinux_2_28_aarch64.whl (3.8MiB)
fastwarc-1.0.9-cp313-cp313-manylinux_2_28_x86_64.whl (3.8MiB)
fastwarc-1.0.9-cp313-cp313-win_amd64.whl (1.9MiB)
fastwarc-1.0.9-cp314-cp314-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp314-cp314-macosx_12_0_x86_64.whl (2.1MiB)
fastwarc-1.0.9-cp314-cp314-manylinux_2_28_aarch64.whl (3.8MiB)
fastwarc-1.0.9-cp314-cp314-manylinux_2_28_x86_64.whl (3.8MiB)
fastwarc-1.0.9-cp314-cp314-win_amd64.whl (2.0MiB)
fastwarc-1.0.9-cp314-cp314t-macosx_12_0_arm64.whl (1.9MiB)
fastwarc-1.0.9-cp314-cp314t-macosx_12_0_x86_64.whl (2.1MiB)
fastwarc-1.0.9-cp314-cp314t-manylinux_2_28_aarch64.whl (3.9MiB)
fastwarc-1.0.9-cp314-cp314t-manylinux_2_28_x86_64.whl (3.9MiB)
fastwarc-1.0.9-cp314-cp314t-win_amd64.whl (2.1MiB)
fastwarc-1.0.9.tar.gz (70.5KiB)
