Skip to content

Add RewriteFiles support to iceberg-rust #1607

@CTTY

Description

@CTTY

Is your feature request related to a problem or challenge?

Currently, iceberg-rust does not support committing operations that both add and delete files. This feature, known as RewriteFiles in iceberg-java, is crucial for our use case, particularly for compaction.

Describe the solution you'd like

Based on my initial investigation in #1606, I believe the following three components are necessary to implement RewriteFiles in iceberg-rust:

  • Process delete files: We need to add logic to handle delete files when generating snapshots.
  • Implement RewriteFilesAction: This will require creating a new transaction action, RewriteFilesAction, and related classes like RewriteFilesOperation.
  • Support validation: We need to add validation for rewrite operations to prevent conflicts. Currently we don't have any validation mechnism in the codebase, so a new trait SnapshotValidator would be needed.

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions