The simplest, highest-throughput interface to Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compliant APIs like Cloudflare R2.
Project Links
Meta
Author: Kyle Barron <kyle@developmentseed.org>
Requires Python: >=3.9
Classifiers
Development Status
- 4 - Beta
Framework
- AsyncIO
- FastAPI
- aiohttp
Intended Audience
- Developers
- Science/Research
License
- OSI Approved :: MIT License
Operating System
- MacOS
- Microsoft :: Windows
- Unix
Programming Language
- Rust
- Python :: Implementation :: CPython
- Python :: Implementation :: PyPy
Topic
- Internet
Typing
- Typed
obstore
The simplest, highest-throughput [^1] Python interface to S3, GCS, Azure Storage, & other S3-compliant APIs, powered by Rust.
- Sync and async API with full type hinting.
- Streaming downloads with configurable chunking.
- Streaming uploads from files or async or sync iterators.
- Streaming list, with no need to paginate.
- Automatic multipart uploads for large file objects.
- Automatic credential refresh before expiration.
- File-like object API and fsspec integration.
- Easy to install with no required Python dependencies.
- Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python
dicts. - Zero-copy data exchange between Rust and Python via the buffer protocol.
For Rust developers looking to add object_store support to their own Python packages, refer to pyo3-object_store.
[^1]: Benchmarking is ongoing, but preliminary results indicate roughly 9x higher throughput than fsspec and 2.8x higher throughput than aioboto3 for many concurrent, small, get requests from an async context.
Installation
To install obstore using pip:
pip install obstore
Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:
conda install -c conda-forge obstore
Documentation
Full documentation is available on the website.
Head to Getting Started to dig in.
0.6.0
Mar 24, 2025
0.5.1
Mar 17, 2025
0.5.0
Mar 17, 2025
0.5.0rc1
Mar 17, 2025
0.5.0b1
Mar 04, 2025
0.4.0
Feb 10, 2025
0.4.0b4
Feb 05, 2025
0.4.0b3
Feb 05, 2025
0.4.0b2
Feb 05, 2025
0.4.0b1
Jan 31, 2025
0.3.0
Jan 16, 2025
0.3.0b11
Jan 16, 2025
0.3.0b10
Jan 08, 2025
0.3.0b9
Dec 13, 2024
0.3.0b8
Nov 22, 2024
0.3.0b7
Nov 21, 2024
0.3.0b6
Nov 13, 2024
0.3.0b5
Nov 12, 2024
0.3.0b4
Nov 11, 2024
0.3.0b3
Nov 11, 2024
0.3.0b2
Nov 01, 2024
0.3.0b1
Nov 01, 2024
0.2.0
Oct 25, 2024
0.2.0b2
Oct 25, 2024
0.1.0b1
Oct 24, 2024
Wheel compatibility matrix
Files in release
obstore-0.6.0-cp310-cp310-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-cp310-cp310-macosx_11_0_arm64.whl (3.2MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-cp310-cp310-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-cp310-cp310-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-cp310-cp310-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-cp310-cp310-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-cp310-cp310-musllinux_1_2_x86_64.whl (3.7MiB)
obstore-0.6.0-cp310-cp310-win_amd64.whl (3.7MiB)
obstore-0.6.0-cp311-cp311-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-cp311-cp311-macosx_11_0_arm64.whl (3.2MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-cp311-cp311-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-cp311-cp311-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-cp311-cp311-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-cp311-cp311-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-cp311-cp311-musllinux_1_2_x86_64.whl (3.7MiB)
obstore-0.6.0-cp311-cp311-win_amd64.whl (3.7MiB)
obstore-0.6.0-cp312-cp312-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-cp312-cp312-macosx_11_0_arm64.whl (3.1MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-cp312-cp312-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-cp312-cp312-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-cp312-cp312-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-cp312-cp312-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-cp312-cp312-musllinux_1_2_x86_64.whl (3.7MiB)
obstore-0.6.0-cp312-cp312-win_amd64.whl (3.7MiB)
obstore-0.6.0-cp313-cp313-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-cp313-cp313-macosx_11_0_arm64.whl (3.1MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-cp313-cp313-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-cp313-cp313-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-cp313-cp313-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-cp313-cp313-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-cp313-cp313-musllinux_1_2_x86_64.whl (3.7MiB)
obstore-0.6.0-cp313-cp313-win_amd64.whl (3.7MiB)
obstore-0.6.0-cp39-cp39-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-cp39-cp39-macosx_11_0_arm64.whl (3.2MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-cp39-cp39-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-cp39-cp39-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-cp39-cp39-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-cp39-cp39-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-cp39-cp39-musllinux_1_2_x86_64.whl (3.7MiB)
obstore-0.6.0-cp39-cp39-win_amd64.whl (3.7MiB)
obstore-0.6.0-pp310-pypy310_pp73-macosx_10_12_x86_64.whl (3.4MiB)
obstore-0.6.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl (3.2MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (3.4MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5MiB)
obstore-0.6.0-pp310-pypy310_pp73-manylinux_2_24_aarch64.whl (3.3MiB)
obstore-0.6.0-pp310-pypy310_pp73-musllinux_1_2_aarch64.whl (3.5MiB)
obstore-0.6.0-pp310-pypy310_pp73-musllinux_1_2_armv7l.whl (3.5MiB)
obstore-0.6.0-pp310-pypy310_pp73-musllinux_1_2_i686.whl (3.5MiB)
obstore-0.6.0-pp310-pypy310_pp73-musllinux_1_2_x86_64.whl (3.7MiB)
Extras:
None
Dependencies:
typing-extensions