Feat : added truncate table support #19633

Nachiket-Roy · 2026-01-04T06:49:24Z

Which issue does this PR close?

Related to [EPIC] Complete DML Support (MERGE, INSERT OVERWRITE, TRUNCATE) #19617

Rationale for this change

DataFusion recently added TableProvider hooks for row-level DML operations such as DELETE and UPDATE, but TRUNCATE TABLE was still unsupported.

What changes are included in this PR?

This PR adds planning and integration support for TRUNCATE TABLE in DataFusion, completing another part of the DML surface alongside existing DELETE and UPDATE support.

Specifically, it includes:

SQL parsing support for TRUNCATE TABLE
Logical plan support via a new WriteOp::Truncate DML operation
Physical planner routing for TRUNCATE statements
A new TableProvider::truncate() hook for storage-native implementations
Protobuf / DML node support for serializing and deserializing TRUNCATE operations
SQL logic tests validating logical and physical planning behavior

The implementation follows the same structure and conventions as the existing DELETE and UPDATE DML support.
Execution semantics are delegated to individual TableProvider implementations via the new hook.

Are these changes tested?

Yes. The PR includes:
SQL logic tests that verify:

Parsing of TRUNCATE TABLE
Correct logical plan generation
Correct physical planner routing
Clear and consistent errors for providers that do not yet support TRUNCATE

These tests mirror the existing testing strategy used for unsupported DELETE and UPDATE operations.

Are there any user-facing changes?

Yes. Users can now execute TRUNCATE TABLE statements in DataFusion for tables whose TableProvider supports the new truncate() hook. Tables that do not support TRUNCATE will return a clear NotImplemented error.

Nachiket-Roy · 2026-01-04T08:13:22Z

@ethan-tyler, please review this PR. Thank you!

ethan-tyler

Thanks for adding TRUNCATE @Nachiket-Roy! This threads through WriteOp, TableProvider, physical planner, and protobuf correctly. Looking nicely done.

Couple things I noticed:

Schema: DmlStatement is constructed with an empty schema instead of using DmlStatement::new(), which sets {count: UInt64}. Breaks the DML contract and causes a proto roundtrip mismatch.

Silent option ignoring: Options like CASCADE, RESTART IDENTITY, PARTITION are dropped silently via ... TRUNCATE TABLE t CASCADE would succeed without actually cascading. Worth rejecting explicitly.

Once those are in, adding TRUNCATE to roundtrip_logical_plan_dml and some negative tests for unsupported options would round it out.

ethan-tyler · 2026-01-05T05:27:15Z

datafusion/sql/src/statement.rs

+                Ok(LogicalPlan::Dml(DmlStatement {
+                    table_name: table.clone(),
+                    target: source,
+                    op: WriteOp::Truncate,
+                    input: Arc::new(LogicalPlan::EmptyRelation(EmptyRelation {
+                        produce_one_row: false,
+                        schema: DFSchemaRef::new(DFSchema::empty()),
+                    })),
+                    output_schema: DFSchemaRef::new(DFSchema::empty()),
+                }))
+            }


This should use DmlStatement::new() rather than constructing the struct directly. The constructor sets output_schema: make_count_schema() which gives you the {count: UInt64} that DML ops return.
Also noticed this causes a proto roundtrip mismatch: encode uses the struct directly but decode uses the constructor (at mod.rs:978), so you'd get different plans before/after serialization.

Something like:

Suggested change

Ok(LogicalPlan::Dml(DmlStatement {

table_name: table.clone(),

target: source,

op: WriteOp::Truncate,

input: Arc::new(LogicalPlan::EmptyRelation(EmptyRelation {

produce_one_row: false,

schema: DFSchemaRef::new(DFSchema::empty()),

})),

output_schema: DFSchemaRef::new(DFSchema::empty()),

}))

}

Ok(LogicalPlan::Dml(DmlStatement::new(

table.clone(),

source,

WriteOp::Truncate,

Arc::new(LogicalPlan::EmptyRelation(EmptyRelation {

produce_one_row: false,

schema: DFSchemaRef::new(DFSchema::empty()),

})),

)))

ethan-tyler · 2026-01-05T05:38:00Z

datafusion/sql/src/statement.rs

+            Statement::Truncate { table_names, .. } => {
+                if table_names.len() != 1 {
+                    return not_impl_err!(
+                        "TRUNCATE with multiple tables is not supported"
+                    );
+                }


The .. here silently drops partitions, identity, cascade, on_cluster, and target.only. If someone writes TRUNCATE TABLE t CASCADE expecting cascade behavior, they'd get a normal truncate with no indication the option was ignored.
Worth rejecting these explicitly? Something like:

Suggested change

Statement::Truncate { table_names, .. } => {

if table_names.len() != 1 {

return not_impl_err!(

"TRUNCATE with multiple tables is not supported"

);

}

Statement::Truncate { table_names, partitions, identity, cascade, on_cluster, .. } => {

if table_names.len() != 1 {

return not_impl_err!("TRUNCATE with multiple tables is not supported");

}

let target = &table_names[0];

if target.only {

return not_impl_err!("TRUNCATE with ONLY is not supported");

}

if partitions.is_some() {

return not_impl_err!("TRUNCATE with PARTITION is not supported");

}

if identity.is_some() {

return not_impl_err!("TRUNCATE with RESTART/CONTINUE IDENTITY is not supported");

}

if cascade.is_some() {

return not_impl_err!("TRUNCATE with CASCADE/RESTRICT is not supported");

}

if on_cluster.is_some() {

return not_impl_err!("TRUNCATE with ON CLUSTER is not supported");

}

// ... rest

}

If you add the option rejection, some negative tests would round it out:

statement error TRUNCATE with CASCADE/RESTRICT is not supported TRUNCATE TABLE t1 CASCADE; statement error TRUNCATE with multiple tables is not supported TRUNCATE TABLE t1, t2;

ethan-tyler · 2026-01-05T05:47:36Z

datafusion/catalog/src/table.rs

+    /// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64),
+    /// representing the number of rows removed.
+    async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {
+        not_impl_err!("TRUNCATE not supported for {}", self.table_type())


nit: DELETE/UPDATE messages say "... for {} table" but this one drops "table"

Nachiket-Roy · 2026-01-05T08:30:55Z

@ethan-tyler Thanks for the thorough review. I've addressed all your comments, Please take another look.

martin-g · 2026-01-05T16:43:02Z

datafusion/sql/src/statement.rs

+                    WriteOp::Truncate,
+                    Arc::new(LogicalPlan::EmptyRelation(EmptyRelation {
+                        produce_one_row: false,
+                        schema: DFSchemaRef::new(DFSchema::empty()),


datafusion/datafusion/expr/src/logical_plan/dml.rs

Line 147 in 47df535

/// The relation that determines the tuples to add/remove/modify the schema must match with table_schema

says the schema must match with table_schema. Is this important here ?

Unlike DELETE or UPDATE, TRUNCATE is a table-level operation with no filters, assignments. The input relation is only a placeholder to satisfy the DML plan shape and is never inspected or executed for row data. Because no rows or columns are read, the usual “input schema must match table schema” invariant does not apply here.

I verified this and you're right that TRUNCATE doesn't use the input.

Physical planner shows:

DELETE/UPDATE: extract filters/assignments from input

TRUNCATE: ignores input entirely, just calls provider.truncate()

So functionally the empty schema is fine. The doc comment is aspirational. Could either keep as-is and update the DmlStatement docs to note TRUNCATE is special, or use the table schema for consistency (extra code, no functional benefit). Former seems fine.

ethan-tyler

Thanks for the follow-up @Nachiket-Roy DmlStatement::new() fix and option rejection look good.

Few things on closer look:

Test contract: CaptureTruncateProvider returns EmptyExec (zero rows), it checks the hook was called but doesn't validate the "returns one row with count" contract.
Proto roundtrip: Adding TRUNCATE to roundtrip_logical_plan_dml would catch regressions on the proto mapping.
Negative tests: CASCADE and multi-table are covered, PARTITION, ONLY, and IDENTITY are also rejected but not tested.

ethan-tyler · 2026-01-05T21:27:17Z

datafusion/core/tests/custom_sources_cases/dml_planning.rs

+    async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {
+        *self.truncate_called.lock().unwrap() = true;
+
+        Ok(Arc::new(EmptyExec::new(Arc::new(Schema::new(vec![
+            Field::new("count", DataType::UInt64, false),
+        ])))))
+    }
+}


TableProvider::truncate docs say it returns an ExecutionPlan producing {count: UInt64}. This test returns EmptyExec which produces zero rows, it validates the hook was called, but not the contract.

Could tighten this to return an actual row and assert on it:

Suggested change

async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {

*self.truncate_called.lock().unwrap() = true;

Ok(Arc::new(EmptyExec::new(Arc::new(Schema::new(vec![

Field::new("count", DataType::UInt64, false),

])))))

}

}

// Add these imports:

use arrow::array::UInt64Array;

use arrow::record_batch::RecordBatch;

use datafusion_physical_plan::test::TestMemoryExec;

// Then update the truncate impl:

async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {

*self.truncate_called.lock().unwrap() = true;

let schema = Arc::new(Schema::new(vec![

Field::new("count", DataType::UInt64, false),

]));

let batch = RecordBatch::try_new(

Arc::clone(&schema),

vec![Arc::new(UInt64Array::from(vec![0u64]))],

)?;

Ok(Arc::new(TestMemoryExec::try_new(&[vec![batch]], schema, None)?))

}

ethan-tyler · 2026-01-05T21:33:55Z

datafusion/proto/src/logical_plan/to_proto.rs

            WriteOp::Delete => protobuf::dml_node::Type::Delete,
            WriteOp::Update => protobuf::dml_node::Type::Update,
            WriteOp::Ctas => protobuf::dml_node::Type::Ctas,
+            WriteOp::Truncate => protobuf::dml_node::Type::Truncate,


You added the proto enum and conversions and worth adding a roundtrip test to lock it in.

In datafusion/proto/tests/cases/roundtrip_logical_plan.rs, the roundtrip_logical_plan_dml test covers INSERT, DELETE, UPDATE, CTAS.

Adding TRUNCATE there would catch any future regressions:

// In the queries array: "TRUNCATE TABLE test_table",

ethan-tyler · 2026-01-05T21:35:55Z

datafusion/sqllogictest/test_files/truncate.slt

+
+# Test TRUNCATE with multiple tables
+statement error TRUNCATE with multiple tables is not supported
+TRUNCATE TABLE t1, t2;


You have negative tests for CASCADE and multi-table. The planner also rejects PARTITION, ONLY, IDENTITY, and ON CLUSTER.

Might be worth covering those too:

statement error TRUNCATE with PARTITION is not supported TRUNCATE TABLE t1 PARTITION (p1); statement error TRUNCATE with ONLY is not supported TRUNCATE ONLY t1; statement error TRUNCATE with RESTART/CONTINUE IDENTITY is not supported TRUNCATE TABLE t1 RESTART IDENTITY;

Nachiket-Roy · 2026-01-06T06:04:24Z

@ethan-tyler Thanks again for the review! I’ve tightened the CaptureTruncateProvider test to return a single {count: UInt64} row, added negative tests covering PARTITION, ONLY, and IDENTITY (in addition to CASCADE and multi-table), and added TRUNCATE to roundtrip_logical_plan_dml to lock in the proto roundtrip behavior.

Nachiket-Roy · 2026-01-06T07:36:02Z

@ethan-tyler I tried tightening the test using TestMemoryExec to materialize a {count: UInt64} row as suggested.
However this causes the test to fail:

TestMemoryExec::with_new_children not implemented

This happens because physical optimizer rules run and require ExecutionPlan::with_new_children, which TestMemoryExec does not implement. Could you please look into this?

ethan-tyler · 2026-01-06T15:13:12Z

@ethan-tyler I tried tightening the test using TestMemoryExec to materialize a {count: UInt64} row as suggested. However this causes the test to fail:
TestMemoryExec::with_new_children not implemented
This happens because physical optimizer rules run and require ExecutionPlan::with_new_children, which TestMemoryExec does not implement. Could you please look into this?

@Nachiket-Roy My bad - should have validated that suggestion first. TestMemoryExec doesn't implement with_new_children, so it panics during filter pushdown.

Better to stick with EmptyExec:

async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {
    *self.truncate_called.lock().unwrap() = true;

    Ok(Arc::new(EmptyExec::new(Arc::new(Schema::new(vec![
        Field::new("count", DataType::UInt64, false),
    ])))))
}

This matches what CaptureDeleteProvider and CaptureUpdateProvider already do. These tests just verify the hook gets called and validating provider output is the implementor's responsibility.

The "proper" way to return an actual count is DmlResultExec, but it's private to the memory table module. Making it public would be a separate discussion. Sorry for the detour, the original implementation was fine.

ethan-tyler

LGTM - approving, thanks for the contribution!

added truncate table support

c73c0bb

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate proto Related to proto crate labels Jan 4, 2026

Merge branch 'main' into feat/dml-truncate

663a69b

Nachiket-Roy marked this pull request as draft January 4, 2026 06:52

Nachiket-Roy added 3 commits January 4, 2026 07:08

formatting fixed

9f44f39

test and clippy issue fixed

aad0378

format resolved

4a1fb16

Nachiket-Roy marked this pull request as ready for review January 4, 2026 08:01

Nachiket-Roy changed the title ~~added truncate table support~~ Feat : added truncate table support Jan 4, 2026

comment improved

a6196a3

ethan-tyler reviewed Jan 5, 2026

View reviewed changes

Nachiket-Roy added 2 commits January 5, 2026 11:25

chore : reject unsupported options

12e5920

Merge branch 'main' into feat/dml-truncate

7b4d016

Merge branch 'main' into feat/dml-truncate

b331a34

martin-g reviewed Jan 5, 2026

View reviewed changes

ethan-tyler reviewed Jan 5, 2026

View reviewed changes

Nachiket-Roy added 2 commits January 6, 2026 08:59

added negative tests

e59cb23

Merge branch 'main' into feat/dml-truncate

4cfe422

Nachiket-Roy added 4 commits January 6, 2026 09:37

disable the optimizer

32075e0

test fixed

760ed22

Merge branch 'main' into feat/dml-truncate

5c0aeeb

type matching fixed

7a9b4b1

Nachiket-Roy added 3 commits January 6, 2026 16:14

revert to EmptyExec

e7dc856

Merge branch 'main' into feat/dml-truncate

2cf2913

Merge branch 'main' into feat/dml-truncate

fb14364

ethan-tyler approved these changes Jan 6, 2026

View reviewed changes

-            Statement::Truncate { table_names, .. } => {
-                if table_names.len() != 1 {
-                    return not_impl_err!(
-                        "TRUNCATE with multiple tables is not supported"
-                    );
-                }
+               Statement::Truncate { table_names, partitions, identity, cascade, on_cluster, .. } => {
+    if table_names.len() != 1 {
+        return not_impl_err!("TRUNCATE with multiple tables is not supported");
+    }
+    let target = &table_names[0];
+    if target.only {
+        return not_impl_err!("TRUNCATE with ONLY is not supported");
+    }
+    if partitions.is_some() {
+        return not_impl_err!("TRUNCATE with PARTITION is not supported");
+    }
+    if identity.is_some() {
+        return not_impl_err!("TRUNCATE with RESTART/CONTINUE IDENTITY is not supported");
+    }
+    if cascade.is_some() {
+        return not_impl_err!("TRUNCATE with CASCADE/RESTRICT is not supported");
+    }
+    if on_cluster.is_some() {
+        return not_impl_err!("TRUNCATE with ON CLUSTER is not supported");
+    }
+    // ... rest
+}

-    async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {
-        *self.truncate_called.lock().unwrap() = true;
-        Ok(Arc::new(EmptyExec::new(Arc::new(Schema::new(vec![
-            Field::new("count", DataType::UInt64, false),
-        ])))))
-    }
-}
+// Add these imports:
+use arrow::array::UInt64Array;
+use arrow::record_batch::RecordBatch;
+use datafusion_physical_plan::test::TestMemoryExec;
+// Then update the truncate impl:
+async fn truncate(&self, _state: &dyn Session) -> Result<Arc<dyn ExecutionPlan>> {
+    *self.truncate_called.lock().unwrap() = true;
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("count", DataType::UInt64, false),
+    ]));
+    let batch = RecordBatch::try_new(
+        Arc::clone(&schema),
+        vec![Arc::new(UInt64Array::from(vec![0u64]))],
+    )?;
+    Ok(Arc::new(TestMemoryExec::try_new(&[vec![batch]], schema, None)?))
+}

Feat : added truncate table support #19633

Are you sure you want to change the base?

Feat : added truncate table support #19633

Conversation

Nachiket-Roy commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Nachiket-Roy commented Jan 4, 2026

Uh oh!

ethan-tyler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nachiket-Roy commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethan-tyler left a comment

Choose a reason for hiding this comment

Uh oh!

ethan-tyler Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nachiket-Roy commented Jan 6, 2026

Uh oh!

Nachiket-Roy commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethan-tyler commented Jan 6, 2026

Uh oh!

ethan-tyler left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Nachiket-Roy commented Jan 4, 2026 •

edited

Loading

ethan-tyler Jan 5, 2026 •

edited

Loading

Nachiket-Roy commented Jan 6, 2026 •

edited

Loading