sqlframe · Python Simple Repository Browser

Project Links

Meta

Author: Ryan Eakman

Requires Python: >=3.10

Classifiers

Development Status

5 - Production/Stable

Intended Audience

Developers
Science/Research

License

OSI Approved :: MIT License

Operating System

OS Independent

Programming Language

SQL
Python :: 3 :: Only

Your PySpark code. Your database.

Run the PySpark DataFrame API directly on your database.
No Spark cluster required.

Quickstart · Documentation · Report an issue

Why SQLFrame?

Familiar API. Write transformations with PySpark DataFrame operations, joins, and window functions.
Direct execution. Run your transformations on your database engine.
Visible SQL. Inspect, debug, and share the queries behind your DataFrame code.

Try it locally

Install SQLFrame with DuckDB. You need Python 3.10 or later; no cloud account or Spark installation is required.

pip install "sqlframe[duckdb]"

from sqlframe.duckdb import DuckDBSession

session = DuckDBSession()
df = session.createDataFrame(
    [("US", 120), ("UK", 80), ("US", 30)],
    ["country", "revenue"],
)
df.groupBy("country").sum("revenue").orderBy("country").show()

+---------+--------------+
| country | sum(revenue) |
+---------+--------------+
|    UK   |      80      |
|    US   |     150      |
+---------+--------------+

To inspect a transformation's SQL, call df.sql(optimize=True). Follow the quickstart for a complete walkthrough.

Choose your engine

Engine	Install
DuckDB	`pip install "sqlframe[duckdb]"`
BigQuery	`pip install "sqlframe[bigquery]"`
Snowflake	`pip install "sqlframe[snowflake]"`
Postgres	`pip install "sqlframe[postgres]"`
ClickHouse (preview)	`pip install "sqlframe[clickhouse]"`
Databricks	`pip install "sqlframe[databricks]"`
Spark	`pip install "sqlframe[spark]"`
Standalone	`pip install sqlframe`
Trino (preview)	`pip install "sqlframe[trino]"`

Standalone sessions generate SQL without a database connection. For local Parquet support, install sqlframe[standalone].

ClickHouse and Trino are preview integrations; see their guides for supported workflows and limits.

Community integrations: GizmoSQL · Redshift (pip install "sqlframe[redshift]").

SQLFrame is also available on conda-forge: conda install -c conda-forge sqlframe.

Already using PySpark?

Activate SQLFrame before importing PySpark to run supported transformations on another engine.

from sqlframe import activate

activate("duckdb")

from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()

You can also import SQLFrame directly, as shown in the quickstart. See the activation guide and your engine's API compatibility reference for details.

Keep exploring

Examples: longer pipelines with aggregations and window functions.
Generated SQL: formatting, optimization, dialect overrides, and OpenAI enrichment.
Input and output dialects: configure SQLFrame for your engine's conventions.
Standalone SQL generation: generate queries without connecting to an engine.

SQLFrame is open source under the MIT license.