Search Shortcut cmd + k | ctrl + k

DuckLake is an integrated data lake and catalog format

DuckLake delivers advanced data lake features without traditional lakehouse complexity by using Parquet files and your SQL database. It's an open, standalone format from the DuckDB team.

Get started Documentation

DuckLake's architecture

With DuckLake, all you need to run your own data warehouse is a catalog database and storage for Parquet files.

Client

DuckLake client

Clients

Users can run multiple DuckLake clients and connect concurrently to the catalog database – PostgreSQL, MySQL or SQLite – to work over the same DuckLake dataset.

Client

DuckLake also works with DuckDB as the catalog database. In this case, you are limited to a single client but still get to enjoy the data lake features such as time travel.

Catalog database

DuckDB

Catalog database

DuckLake can use any SQL system as its catalog database, provided that it supports ACID transactions and primary key constraints.

Storage

Parquet

Storage

DuckLake can store your data on any object storage such as AWS S3.

DuckLake’s key features

Data lake operations

DuckLake supports snapshots, time travel queries, schema evolution and partitioning.

Lightweight snapshots

You can have as many snapshots as you want without frequent compacting steps!

ACID transactions

DuckLake allows concurrent access with ACID transactional guarantees over multi-table operations.

Performance-oriented

DuckLake uses statistics for filter pushdown, enabling fast queries even on large datasets.

In conversation: DuckDB founders on DuckLake

Listen to Hannes Mühleisen and Mark Raasveldt walk through the history of data lakes and introduce DuckLake, a new lakehouse format.

Create your first DuckLake with DuckDB

DuckDB provides first-class support for DuckLake through its highly portable extension, running wherever DuckDB does.

INSTALL ducklake;

ATTACH 'ducklake:metadata.ducklake' AS my_ducklake;
USE my_ducklake;

INSTALL ducklake;

ATTACH 'ducklake:metadata.ducklake' AS my_ducklake;
USE my_ducklake;

INSTALL ducklake;
INSTALL postgres;

-- Make sure that the database `ducklake_catalog` exists in PostgreSQL.
ATTACH 'ducklake:postgres:dbname=ducklake_catalog host=your_postgres_host' AS my_ducklake
     (DATA_PATH 'data_files/');
USE my_ducklake;

INSTALL ducklake;
INSTALL sqlite;

ATTACH 'ducklake:sqlite:metadata.sqlite' AS my_ducklake
    (DATA_PATH 'data_files/');
USE my_ducklake;

INSTALL ducklake;
INSTALL mysql;

-- Make sure that the database `ducklake_catalog` exists in MySQL
ATTACH 'ducklake:mysql:db=ducklake_catalog host=your_mysql_host' AS my_ducklake
    (DATA_PATH 'data_files/');
USE my_ducklake;

Frequently asked questions

Answers to common questions to help you understand and make the most of DuckLake.

Why should I use DuckLake?

DuckLake provides a lightweight one-stop solution if you need a data lake and catalog.

You can use DuckLake for a “multiplayer DuckDB” setup with multiple DuckDB instances reading and writing the same dataset – a concurrency model not supported by vanilla DuckDB.

If you only use DuckDB for both your DuckLake entry point and your catalog database, you can still benefit from DuckLake: you can run time travel queries, exploit data partitioning, and can store your data in multiple files instead of using a single (potentially very large) database file.

What is “DuckLake”?

“DuckLake” can refer to a number of things:

The DuckLake format that uses a catalog database and a Parquet storage to store data.
A DuckLake instance storing a dataset with the DuckLake Lakehouse format.
The ducklake DuckDB extension, which supports reading/writing datasets using the DuckLake format.

What is the license of DuckLake?

The DuckLake specification and the ducklake DuckDB extension are released under the MIT license.

More FAQs