Summary
Remove pyiceberg_core.datafusion.IcebergDataFusionTable from the pyiceberg-core Python bindings.
This proposal only removes the Python export. It does not remove any DataFusion table provider from iceberg-rust.
Rationale
This experimental feature was intended to integrate PyIceberg with DataFusion Python. However, the resulting query does not use PyIceberg's scan implementation. PyIceberg passes the table identifier, metadata location, and file I/O properties to pyiceberg-core, which loads a separate Rust StaticTable. iceberg-rust then handles metadata and manifest parsing, pruning, scan planning, delete handling, and data reads.
This creates two independent Iceberg implementations behind a PyIceberg Table: Python for Table.scan() and Rust when the table is registered with DataFusion. Their features and behavior can differ, and PyIceberg fixes do not apply to the Rust path. The API therefore appears to query through PyIceberg while actually relying on iceberg-rust's feature set and semantics.
The binding also tightly couples pyiceberg-core to DataFusion's Rust FFI and its major-version compatibility requirements.
Proposal
Remove the following from the pyiceberg-core Python bindings:
- The
pyiceberg_core.datafusion module
- The
IcebergDataFusionTable Python class
- Related module registration, dependencies, and tests
The corresponding PyIceberg change should remove Table.__datafusion_table_provider__ and update its documentation and optional dependencies.
Non-goals
This proposal does not remove or deprecate the Rust iceberg-datafusion crate, IcebergTableProvider, or IcebergStaticTableProvider. The Rust integration remains available to applications that choose iceberg-rust as their Iceberg implementation.
Because the Python API has been released, the removal should be coordinated with PyIceberg and included in an appropriate breaking release.
Summary
Remove
pyiceberg_core.datafusion.IcebergDataFusionTablefrom thepyiceberg-corePython bindings.This proposal only removes the Python export. It does not remove any DataFusion table provider from iceberg-rust.
Rationale
This experimental feature was intended to integrate PyIceberg with DataFusion Python. However, the resulting query does not use PyIceberg's scan implementation. PyIceberg passes the table identifier, metadata location, and file I/O properties to
pyiceberg-core, which loads a separate RustStaticTable. iceberg-rust then handles metadata and manifest parsing, pruning, scan planning, delete handling, and data reads.This creates two independent Iceberg implementations behind a PyIceberg
Table: Python forTable.scan()and Rust when the table is registered with DataFusion. Their features and behavior can differ, and PyIceberg fixes do not apply to the Rust path. The API therefore appears to query through PyIceberg while actually relying on iceberg-rust's feature set and semantics.The binding also tightly couples
pyiceberg-coreto DataFusion's Rust FFI and its major-version compatibility requirements.Proposal
Remove the following from the
pyiceberg-corePython bindings:pyiceberg_core.datafusionmoduleIcebergDataFusionTablePython classThe corresponding PyIceberg change should remove
Table.__datafusion_table_provider__and update its documentation and optional dependencies.Non-goals
This proposal does not remove or deprecate the Rust
iceberg-datafusioncrate,IcebergTableProvider, orIcebergStaticTableProvider. The Rust integration remains available to applications that choose iceberg-rust as their Iceberg implementation.Because the Python API has been released, the removal should be coordinated with PyIceberg and included in an appropriate breaking release.