Skip to content

[Improvement] Managed fileset silently adopts existing directories, causing unintended data deletion on drop #12550

Open

Description

@geyanggang

What would you like to be improved?

Currently, when creating a managed fileset that points to an already-existing directory, Gravitino
silently "adopts" that directory without any warning or rejection. Later, when the fileset (or its
parent schema/catalog) is dropped, the physical directory is recursively deleted because it is
managed(default).

This creates a serious data-loss issue in the following scenario:

  1. User creates a managed fileset pointing to an existing path (e.g., /data/production/reports)
    — Gravitino logs "Fileset {} manages the existing location {}" but proceeds without error.
  2. User later realizes the fileset was misconfigured and drops it, or drops the catalog with
    force=true.
  3. The entire /data/production/reports directory is recursively deleted.

Relevant code:

  • FilesetCatalogOperations.createMultipleLocationFileset(): when the path already exists, it only
    logs an INFO message and continues (no rejection, no type enforcement).
  • FilesetCatalogOperations.dropFileset(): unconditionally calls fs.delete(location, true) for
    MANAGED filesets when disableFSOps is false.

There is no fileset-level property to prevent physical deletion on drop. The only available controls
are:

  • Choosing EXTERNAL type at creation time (but users may not be aware of the consequence of
    MANAGED being the default).
  • Setting disable-filesystem-ops=true at catalog level (which disables ALL filesystem operations,
    not just delete-on-drop).

How should we improve?

Proposed improvements (in priority order):

  1. Reject or warn when creating a MANAGED fileset on an existing path

    • Option A (safer): If the storage path already exists, reject creating it as MANAGED and require
      the user to explicitly use EXTERNAL type, or add a property like
      "allow-existing-location-as-managed": "true" to override.
    • Option B (less disruptive): Log a WARN and add documentation, but still allow it.
  2. Add a fileset-level delete-data-on-drop property

    • Default to true for MANAGED filesets (preserving current behavior).
    • Allow users to set it to false on a per-fileset basis to protect data even for MANAGED type.
    • This provides a safety net for cases where users want Gravitino to manage the lifecycle
      metadata but not the physical deletion.
  3. Documentation improvement

    • Clearly document in the fileset creation API/docs that MANAGED type will delete physical data
      on drop, and that pointing a managed fileset at an existing directory means that directory
      will be deleted when the fileset is dropped.

Activity

  1. Ystk-hsn commented on Sep 25, 2026

    @Ystk-hsn

    I'd like to work on this. I've read through FilesetCatalogOperations and put up a PR covering the three improvements, with Option A for the first one:

    • Creating a MANAGED fileset on a location that already exists and is not empty is rejected, unless allow-existing-location-as-managed=true is set. Empty existing directories are still adopted as before, so pre-created directories keep working.
    • New mutable fileset property delete-data-on-drop (default true), checked in both dropFileset and the cascade path of dropSchema.
    • Docs for both properties and the managed/external distinction.

    If Option B is preferred, the first part can be reduced to a WARN.

    PR: #13506

  2. added 2 commits that reference this issue on Oct 23, 2026
  3. added a commit that references this issue on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    2.0.0Release v2.0.0improvementImprovements on everything

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions