What's New in DuckLake 1.1

2026-10-21 9 min read

Pedro Holanda

DuckLake 1.0 has been out since April 2026, and today it is DuckDB's most downloaded lakehouse solution, with more than 1.5 million downloads last week (October 12 to 18, 2026). Since then, we have merged more than 200 bug fixes, implemented some smaller features, and generally improved performance for catalog operations.

With DuckDB 2.0, we are now releasing a new DuckLake extension that follows the new DuckLake 1.1 specification. In this post, we cover the most important changes in this release compared to the DuckLake extension that ships with DuckDB 1.5.6. We separate changes that require using the DuckLake 1.1 specification from those available to everyone using the new ducklake extension in the DuckDB 2.0, including users who keep their catalogs on DuckLake 1.0.

Compatibility and Migration Plan

The new ducklake extension is backward compatible. It keeps reading and writing DuckLake 1.0 catalogs, so you get most of the improvements without migrating. Only the features that depend on the new specification require a migration. Migrating is opt-in, so existing catalogs stay on 1.0 until you explicitly ask to move to the new version when attaching. Because the migration changes your catalog, make sure to first create a backup and test whether the resulting catalog still works with your workflows. Once you have validated the new catalog, perform the migration by attaching to the catalog with AUTOMATIC_MIGRATION.

ATTACH 'ducklake:metadata.ducklake' AS lake
    (AUTOMATIC_MIGRATION true);

Changes That Require the 1.1 Specification

Nested Schemas

Nested schemas are common in data lakes, as they provide a natural organization path for our tables (e.g., grouping sales data by region), and DuckDB 2.0 now supports them natively. We can use nested schemas in DuckLake to express that hierarchy directly, with paths such as lake.sales.europe.orders. We create the parent schema first, then its child.

CREATE SCHEMA lake.sales;
CREATE SCHEMA lake.sales.europe;
CREATE TABLE lake.sales.europe.orders (id INTEGER);

EXPLAIN also shows the catalog and full schema path for DuckLake scans. When several schemas contain an orders table, we can see which one a query reads without having to infer it from the rest of the plan.

VARIANT Inlining for PostgreSQL and SQLite Catalogs

Data inlining lets small batches stay in the metadata catalog instead of creating separate Parquet files. Inlining VARIANT values used to be limited to DuckDB catalogs, since DuckDB has a native representation of the type. Other supported catalogs, like PostgreSQL and SQLite, do not.

We now support VARIANT inlining in these catalogs by storing the unshredded Parquet Variant encoding (metadata plus value) in the inlining table. PostgreSQL stores it as BYTEA, and SQLite as a BLOB.

ATTACH 'ducklake:sqlite:metadata.sqlite' AS lake;
CREATE TABLE lake.events (id INTEGER, payload VARIANT);
INSERT INTO lake.events VALUES
    (1, 42::VARIANT),
    (2, {'user': 'alice', 'clicks': 3}::VARIANT);

The "NULL" Type

Incoming data can contain columns whose values are all null, before we know their eventual type. DuckLake 1.0 had no way to store such a column, so a CREATE TABLE ... AS with an untyped NULL column silently stored it as INTEGER. Changing it to VARCHAR later then failed, because only widening type promotions are allowed, so we can avoid rewriting existing Parquet files.

DuckLake 1.1 brings support for such columns with the "NULL" type, which we can later promote to the type we actually need.

CREATE TABLE lake.signups AS SELECT 1 AS id, NULL AS referrer;
ALTER TABLE lake.signups ALTER referrer TYPE VARCHAR;

Epoch Partition Transforms

The year, month, day and hour partition transforms of DuckLake store calendar parts, so month(ts) is a number from 1 to 12. Iceberg instead counts the units since the Unix epoch. The new epoch_year, epoch_month, epoch_day and epoch_hour transforms follow the Iceberg convention, so these functions match the partition values that Iceberg writes for the same data.

CREATE TABLE lake.clicks (user_id INTEGER, ts TIMESTAMP)
PARTITIONED BY (epoch_month(ts));

Multi-snapshot Puffin Delete Files

A single Puffin delete file can now hold the deletion vectors of several snapshots of the same data file. When deletion vectors are enabled, this keeps the deletion history together while preserving time travel.

CALL lake.set_option('write_deletion_vectors', true);
DELETE FROM lake.orders WHERE id < 1000;
DELETE FROM lake.orders WHERE id BETWEEN 5000 AND 5999;

Exact String Statistics

Parquet writers truncate the minimum and maximum statistics of long strings. DuckDB, for example, keeps only the first 256 bytes. A truncated value is still a valid bound for skipping files, but it is not the actual minimum or maximum. DuckLake 1.0 could not tell these cases apart, so it always scanned the data to answer min or max on a string column, even though it answers them for numeric columns directly from the catalog. DuckLake 1.1 records whether each string bound is exact. Queries like the one below are now answered from the catalog.

CREATE TABLE lake.users AS
    SELECT i AS id, 'user_' || i AS name FROM range(1000) t(i);
SELECT min(name), max(name) FROM lake.users;

Row Group Counts

A row group is the unit of parallelism of a Parquet scan, so the number of row groups in a file tells a reader how much work it can split across threads or machines. So far, this number was only known after reading the footer of each file, which on object storage means one extra request per file.

DuckLake now stores the number of row groups of every data and delete file in the catalog. Readers can use it to plan their work, such as how to split a scan, without first fetching the footer of each file. Files written before a migration keep an unknown count, since filling it in would require reading the footers of all existing files.

SELECT path, record_count, row_group_count
FROM __ducklake_metadata_lake.ducklake_data_file;

Renamed Columns in Inlined Tables

Inlined data is stored in catalog tables that keep three internal columns next to the columns of the user table. In DuckLake 1.0, these were named row_id, begin_snapshot and end_snapshot, so a table with a column of the same name could not be created while data inlining was enabled. DuckLake 1.1 prefixes them with _ducklake_, so these names now work and their rows are inlined like any other. In turn, column names that start with _ducklake_ are now reserved. The migration renames the internal columns of existing inlined tables.

CREATE TABLE lake.measurements (
    row_id INTEGER,
    begin_snapshot INTEGER,
    end_snapshot INTEGER
);
INSERT INTO lake.measurements VALUES (1, 2, 3);

Changes Available without Migrating

Sorted and Partitioned Tables

We can now declare sorting and partitioning choices directly in CREATE TABLE, as well as CREATE TABLE ... AS, instead of doing so at a later stage with an alter operation. That way, the layout of the table can be declared directly when it is created.

CREATE TABLE lake.orders (
    id INTEGER,
    region VARCHAR,
    ordered_at TIMESTAMP
)
PARTITIONED BY (region)
SORTED BY (ordered_at);

Table Options in DDL

Tables often need different write settings. An event table with large payloads may need different compression or file sizes from a small lookup table. We can now put these options in WITH (...) when creating a table, change them with ALTER TABLE ... SET, and remove overrides with ALTER TABLE ... RESET.

CREATE TABLE lake.events (id INTEGER, payload VARCHAR)
WITH (parquet_compression = 'zstd');
ALTER TABLE lake.events
    SET (skip_stats_columns = ['payload']);
ALTER TABLE lake.events RESET (parquet_compression);

Smarter File Pruning

Skipping a file is often more useful than reading it faster. We now use file statistics for more filter shapes, including OR conditions that span columns. In the query below, a file can be skipped when its statistics show that it contains neither the requested order ID nor orders from the requested region. Pruning also covers IS NULL conditions and fields inside STRUCT columns. We can also push down filters from the build side of a hash join.

SELECT count(*)
FROM lake.orders
WHERE id = 42 OR region = 'eu';

More Control over Compaction

We can now give ducklake_merge_adjacent_files a window with newer_than and older_than, selecting files by when they entered the lake. The example processes files from the past week while leaving the most recent day for a later run.

CALL ducklake_merge_adjacent_files('lake', 'orders',
    newer_than => now()::TIMESTAMP - INTERVAL 7 DAYS,
    older_than => now()::TIMESTAMP - INTERVAL 1 DAY);

For tables with many deleted rows, ducklake_rewrite_data_files now accepts max_compacted_files to cap how many files a single call rewrites.

Broader Parquet Imports

With ducklake_add_data_files, we can register Parquet files that were written by other tools without copying them. DuckLake 1.0 refused files with several column types that DuckDB reads, so those files had to be rewritten first. For example, a file with a FLOAT16 column written by PyArrow failed with an error that a FLOAT was expected but a BLOB was found.

We have broadened the Parquet types accepted by ducklake_add_data_files, including VARIANT, TIME WITH TIME ZONE, TIME_NS, and TIMESTAMPTZ_NS. Parquet FLOAT16 values can now be imported into FLOAT columns.

CREATE TABLE lake.readings (temperature FLOAT);
CALL ducklake_add_data_files('lake', 'readings', 'readings.parquet');

Performance Improvements

Many catalog operations are faster in DuckLake 1.1. To measure four of them, we compared DuckLake 1.1 with the DuckLake 1.0 release on DuckDB 1.5.6, using both a DuckDB and a PostgreSQL 17 catalog. We ran each benchmark three times, each in a fresh process with 8 threads, and report the median. All runs used local files on an Apple M5 Pro machine with 48 GB of memory, with PostgreSQL running on the same machine. The table below reports the results.

Conclusion

DuckLake 1.1 brings nested schemas, new types and partition transforms, and a catalog that records more about its files, together with faster catalog operations and more control over table layout and compaction. Most of these improvements work with existing 1.0 catalogs as soon as you upgrade the extension. For the features that need the new specification, back up your catalog, test the migration, and attach it with AUTOMATIC_MIGRATION. If you run into any issues, please let us know on GitHub.