Development Status
- 5 - Production/Stable
Intended Audience
- Developers
License
- OSI Approved :: Apache Software License
Operating System
- MacOS :: MacOS X
- Microsoft :: Windows
- POSIX :: Linux
Programming Language
- Python :: 3.10
- Python :: 3.11
- Python :: 3.12
- Python :: 3.13
- Python :: 3.14
Topic
- Software Development :: Libraries
Typing
- Typed
data load tool (dlt) — the open-source Python library that automates all your tedious data loading tasks
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or a GPT-4 assisted development playground—dlt can be dropped in anywhere.
🚀 Join our thriving community of likeminded developers and build the future together!
Installation
dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.
pip install dlt
Quick Start
Load chess game data from chess.com API and save it in DuckDB:
import dlt
from dlt.sources.helpers import requests
# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='duckdb',
dataset_name='player_data'
)
# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
response = requests.get(f'https://api.chess.com/pub/player/{player}')
response.raise_for_status()
data.append(response.json())
# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
Check out a super simple demo in Colab or a more advanced Hugging Face demo with Marimo notebooks.
Features
dlt is an open-source Python library that loads data from various, often messy data sources into well-structured datasets. It provides lightweight Python interfaces to extract, load, inspect, and transform data. dlt and dlt docs are built from the ground up to be used with LLMs: the LLM-native workflow will take your pipeline code to data in a notebook for over 5000 sources.
dlt is designed to be easy to use, flexible, and scalable:
- dlt extracts data from REST APIs, SQL databases, cloud storage, Python data structures, and many more.
- dlt infers schemas and data types, normalizes the data, and handles nested data structures.
- dlt supports a variety of popular destinations and has an interface to add custom destinations to create reverse ETL pipelines.
- dlt automates pipeline maintenance with incremental loading, schema evolution, and schema and data contracts.
- dlt supports Python and SQL data access, transformations, pipeline inspection, and visualizing data in Marimo Notebooks.
- dlt can be deployed anywhere Python runs, be it on Airflow, serverless functions, or any other cloud deployment of your choice.
Documentation
For detailed usage and configuration, please refer to the official documentation.
Examples
You can find examples for various use cases in the examples folder, or in the code examples section of our docs page.
Adding as dependency
dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.
majormeans breaking changes and removed deprecationsminornew features, sometimes automatic migrationspatchbug fixes
We suggest that you allow only patch level updates automatically using the Compatible Release Specifier. For example dlt~=1.23.0 allows only versions >=1.23.0 and less than <1.24.0
Please also see our release notes for notable changes between versions.
Get Involved
The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:
- Connect with the Community: Join other dlt users and contributors on our Slack
- Report issues and suggest features: Please use the GitHub Issues to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- Track progress of our work and our plans: Please check out our public Github project
- Improve documentation: Help us enhance the dlt documentation.
Contribute code
Please read CONTRIBUTING before you make a PR.
- 📣 New destinations are unlikely to be merged due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
- Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
- Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.
Sponsors
Blacksmith is a drop-in replacement for GitHub-hosted runners that speed up our CI/CD pipelines by 2x and up to 75% cheaper. We're grateful to Blacksmith for sponsoring us with free CI/CD minutes--which helps us keep builds fast and our costs lower.
License
dlt is released under the Apache 2.0 License.
Wheel compatibility matrix
Files in release
click
(>=7.1)
fsspec
(>=2022.4.0)
gitpython
(>=3.1.42)
giturlparse
(>=0.10.0)
humanize
(>=4.4.0)
jsonpath-ng
(>=1.8.0)
orjson
and
(!=3.10.1,!=3.9.11,!=3.9.12,!=3.9.13,!=3.9.14,<4,>=3.6.7)
orjson
and
(>=3.10.1)
orjson
(>=3.11.0)
packaging
(>=21.1)
pathvalidate
(>=2.5.2)
pendulum
(>=2.1.2)
pendulum
(>=3.0.0)
pluggy
(>=1.3.0)
pytz
(>=2022.6)
pywin32
(>=306)
pyyaml
(>=5.4.1)
requests
(>=2.26.0)
requirements-parser
(>=0.5.0)
rich-argparse
(>=1.6.0)
semver
(>=3.0.0)
setuptools
(>=65.6.0)
simplejson
(>=3.17.5)
sqlglot
(>=26.32.0)
tenacity
(>=8.0.2)
tomlkit
(>=0.11.3)
typing-extensions
(>=4.8.0)
tzdata
(>=2022.1)
win-precise-time
and
(>=1.4.2)