You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Improvement] Managed fileset silently adopts existing directories, causing unintended data deletion on drop #12550
Currently, when creating a managed fileset that points to an already-existing directory, Gravitino
silently "adopts" that directory without any warning or rejection. Later, when the fileset (or its
parent schema/catalog) is dropped, the physical directory is recursively deleted because it is
managed(default).
This creates a serious data-loss issue in the following scenario:
User creates a managed fileset pointing to an existing path (e.g., /data/production/reports)
— Gravitino logs "Fileset {} manages the existing location {}" but proceeds without error.
User later realizes the fileset was misconfigured and drops it, or drops the catalog with force=true.
The entire /data/production/reports directory is recursively deleted.
Relevant code:
FilesetCatalogOperations.createMultipleLocationFileset(): when the path already exists, it only
logs an INFO message and continues (no rejection, no type enforcement).
FilesetCatalogOperations.dropFileset(): unconditionally calls fs.delete(location, true) for
MANAGED filesets when disableFSOps is false.
There is no fileset-level property to prevent physical deletion on drop. The only available controls
are:
Choosing EXTERNAL type at creation time (but users may not be aware of the consequence of
MANAGED being the default).
Setting disable-filesystem-ops=true at catalog level (which disables ALL filesystem operations,
not just delete-on-drop).
How should we improve?
Proposed improvements (in priority order):
Reject or warn when creating a MANAGED fileset on an existing path
Option A (safer): If the storage path already exists, reject creating it as MANAGED and require
the user to explicitly use EXTERNAL type, or add a property like "allow-existing-location-as-managed": "true" to override.
Option B (less disruptive): Log a WARN and add documentation, but still allow it.
Add a fileset-level delete-data-on-drop property
Default to true for MANAGED filesets (preserving current behavior).
Allow users to set it to false on a per-fileset basis to protect data even for MANAGED type.
This provides a safety net for cases where users want Gravitino to manage the lifecycle
metadata but not the physical deletion.
Documentation improvement
Clearly document in the fileset creation API/docs that MANAGED type will delete physical data
on drop, and that pointing a managed fileset at an existing directory means that directory
will be deleted when the fileset is dropped.
I'd like to work on this. I've read through FilesetCatalogOperations and put up a PR covering the three improvements, with Option A for the first one:
Creating a MANAGED fileset on a location that already exists and is not empty is rejected, unless allow-existing-location-as-managed=true is set. Empty existing directories are still adopted as before, so pre-created directories keep working.
New mutable fileset property delete-data-on-drop (default true), checked in both dropFileset and the cascade path of dropSchema.
Docs for both properties and the managed/external distinction.
If Option B is preferred, the first part can be reduced to a WARN.
Activity
Ystk-hsn commentedon Sep 25, 2026
I'd like to work on this. I've read through
FilesetCatalogOperationsand put up a PR covering the three improvements, with Option A for the first one:MANAGEDfileset on a location that already exists and is not empty is rejected, unlessallow-existing-location-as-managed=trueis set. Empty existing directories are still adopted as before, so pre-created directories keep working.delete-data-on-drop(defaulttrue), checked in bothdropFilesetand the cascade path ofdropSchema.If Option B is preferred, the first part can be reduced to a WARN.
PR: #13506
[apache#12550] fix(fileset): Preflight locations and remove drop rete…
[apache#12550] fix(fileset): Address creation guard review feedback
[#14123] feat(fileset): Support managed and external type conversion (#…