Group slots: group-safe WAL retention for Spock multi-master clusters.

docs/managing/group_slots.md

@@ -0,0 +1,186 @@
+# Group Replication Slots
+
+Group replication slots are internally managed replication slots that track the
+oldest safe WAL position for an entire Spock replication group. Modeled after
+BDR/PGD-style group slots, they let a node retain the WAL that any active or
+relevant downstream member of the group might still need, and release it only
+once every required member has confirmed durable progress for the current
+membership generation.
+
+Each Spock database maintains exactly **one** local, **inactive** logical
+replication slot for its group. The slot is never streamed; it exists purely to
+pin (retain) WAL at the group-safe horizon.
+
+> **Operational rule:** group slots are managed by Spock. **Never** drop a group
+> slot manually with `pg_drop_replication_slot()`. Doing so can discard WAL that
+> other group members still require. Use [`spock.repair_group_slot()`](#spockrepair_group_slot)
+> for recovery instead.
+
+## Enabling the feature
+
+Group slots are **disabled by default** and can be enabled without changing any
+existing replication behaviour. Configure them with the following GUCs (all are
+`PGC_SIGHUP`, so a configuration reload is enough — no restart required):
+
+| GUC | Default | Description |
+|-----|---------|-------------|
+| `spock.group_slots_enabled` | `off` | Master switch. When `on`, each Spock database maintains a group slot. When `off`, no group slot is created and normal replication behaviour is unchanged. |
+| `spock.group_slots_worker_interval` | `5s` | How often the per-database group-slot worker recomputes the group-safe horizon and, when safe, advances the group slot. |
+| `spock.group_slots_progress_staleness_timeout` | `60s` | If a required member has not reported fresh progress within this interval, advancement is refused and the reason recorded. |
+| `spock.group_slots_safety_mode` | `strict` | `strict` refuses any unsafe advancement or repair. `repair` additionally allows guarded recreation/relink of a damaged slot. `off` keeps metadata up to date but never advances the slot. |
+
+Example:
+
+```sql
+ALTER SYSTEM SET spock.group_slots_enabled = 'on';
+SELECT pg_reload_conf();
+```
+
+## How it works
+
+* When a node is created (or on the worker's first tick after enabling the
+  feature), Spock seeds durable metadata and the background worker creates the
+  inactive group slot. The name is deterministic: the reserved `spkgrp_`
+  prefix plus hashed database and node names, the same shortening used for
+  subscription slots. Call [`spock.local_group_slot_name()`](#spocklocal_group_slot_name)
+  rather than guessing the literal. The `spkgrp_` prefix deliberately does
+  **not** match the `spk_*` pattern used to clean up per-subscription slots, so
+  ordinary subscription/node cleanup never removes a group slot.
+* A per-database background worker periodically computes the **safe LSN** as the
+  minimum `confirmed_flush_lsn` over local logical slots whose plugin is
+  `spock_output` or `spock` and whose name matches `spk_%` (subscription
+  slots). The group slot itself is excluded. If none exist, the horizon is
+  `pg_current_wal_lsn()`. Advancement happens only when it is safe: a leftover
+  or slow subscription slot pins the horizon until it is removed or catches up.
+  `spock.progress` is used only to decide `stale_progress`, not to compute the
+  LSN.
+* All state (safe LSN, freeze LSN, membership generation, node state, blocked
+  reasons, per-member progress) is stored in durable catalog tables
+  (`spock.group_slot_state`, `spock.group_slot_membership`,
+  `spock.group_slot_member_progress`), so decisions survive restarts without
+  relying on shared memory.
+
+### When advancement is refused
+
+The worker never advances the group slot when any of the following holds. The
+reason is recorded in `spock.group_slot_state.blocked_reason` and surfaced by
+[`spock.group_slot_status()`](#spockgroup_slot_status):
+
+| `blocked_reason` | Meaning |
+|------------------|---------|
+| `join_in_progress` | A node is joining; the horizon is held so the joining node has a stable base point. |
+| `part_in_progress` | A node is parting; the slot is frozen at the part boundary. |
+| `unknown_node_state` | A member is in an unknown lifecycle state. |
+| `stale_progress` | A required member has not reported fresh progress within the staleness timeout. |
+| `membership_generation_mismatch` | The cluster has not converged on a single membership generation. |
+| `missing_slot_state` | The physical slot is gone, or the `group_slot_state` row is missing. Repair is required. The enabled worker does **not** recreate a previously existing slot in `strict` mode. |
+| `safety_mode_off` | `spock.group_slots_safety_mode = off`; metadata is maintained but the slot is not advanced. |
+
+## Zero-downtime node addition and removal
+
+The lifecycle functions below exist so that adding or removing a node cannot
+discard WAL the operation still needs. They are **not wired into any workflow
+on this branch** — a caller (an orchestration script, or an operator) invokes
+them explicitly:
+
+* **Adding a node** — call `spock.group_slot_begin_join()` on the source once
+  the new node is registered. Group-slot advancement pauses, so the group-safe
+  horizon is retained as a stable base point for the joining node. Once the new
+  node is a full, bidirectionally replicating member, call
+  `spock.group_slot_complete_join()` to resume advancement.
+* **Removing a node** — call `spock.group_slot_begin_part()` before the departing node's slots are torn
+  down, freezing the group slot at the pre-removal boundary so retained WAL
+  covers the part. After the node is gone, call
+  `spock.group_slot_complete_part()` to advance to the next generation, clear
+  the freeze, and resume advancement. Because each node maintains its own group
+  slot, run the same call on every other remaining node:
+
+  ```sql
+  SELECT spock.group_slot_complete_part('<removed_node_name>');
+  ```
+
+## Functions
+
+### spock.local_group_slot_name
+
+```sql
+spock.local_group_slot_name() -> name
+```
+
+Returns the deterministic group slot name for the current database and local
+node (`spkgrp_` plus hashed database and node names), or `NULL` when the
+current database is not a Spock node. Always read the name from this function;
+do not construct it by concatenating the raw database and node names.
+
+### spock.group_slot_status
+
+```sql
+spock.group_slot_status()
+```
+
+Returns a single row describing the local group slot: `slot_name`,
+`membership_generation`, `node_state`, `safe_lsn`, `freeze_lsn`,
+`last_advanced_lsn`, `blocked_reason`, `repair_required`, `slot_present`,
+`restart_lsn`, `confirmed_flush_lsn`, `updated_at`, `required_members`, and
+`stale_members`. Use it to inspect the current horizon and any blocked reason.
+
+### spock.advance_group_slot
+
+```sql
+spock.advance_group_slot(target_lsn pg_lsn DEFAULT NULL, force boolean DEFAULT false) -> pg_lsn
+```
+
+Performs a controlled manual advancement to `target_lsn` (or the computed
+group-safe horizon when `NULL`). Even with `force`, hard blockers
+(`join_in_progress`, `part_in_progress`, `unknown_node_state`,
+`membership_generation_mismatch`, `missing_slot_state`) are never bypassed.
+Soft blockers (`stale_progress`, `safety_mode_off`) and advancing past the
+group-safe horizon require both `force = true` **and** a non-strict
+`spock.group_slots_safety_mode`. Returns the LSN the slot was advanced to.
+
+### spock.repair_group_slot
+
+```sql
+spock.repair_group_slot(mode text DEFAULT 'recreate') -> text
+```
+
+Safely repairs a damaged group slot.
+
+* `relink` — re-synchronizes metadata to the deterministic slot name and adopts
+  an existing slot. Always safe.
+* `recreate` — recreates a **missing** slot. Because recreation starts
+  protection at the current WAL position and therefore drops protection for
+  older WAL, it is refused in `strict` safety mode. A non-strict mode (`repair`,
+  or `off`) allows it; use `repair` in production, then switch back to
+  `strict`. A warning is logged when it runs. The group-slot worker does not
+  recreate a previously existing slot while `strict` is set.
+
+Returns `relinked`, `recreated`, `already_present`, or `missing_slot`.
+
+### spock.group_slot_complete_part
+
+```sql
+spock.group_slot_complete_part(parting_node_name name) -> bigint
+```
+
+Finalizes the removal of a node: advances the membership generation,
+clears the part freeze, and resumes advancement. Because each node keeps
+its own group slot, run it once on every remaining node, passing the
+removed node's name. Returns the new membership generation.
+
+It is also safe to call after the departing node has already been dropped
+from `spock.node`: the name is then resolved through the membership rows,
+which is the ordinary case when a node is lost rather than parted cleanly.
+
+Maintenance functions (`advance_group_slot`, `repair_group_slot`, the join/part
+lifecycle functions) are `REVOKE`d from `PUBLIC`. Superusers can call them. To
+let a monitoring or operations role call them, `GRANT EXECUTE` on the specific
+function. `spock.group_slot_status()` and `spock.local_group_slot_name()` remain
+executable by PUBLIC.
+
+## Backward compatibility
+
+When `spock.group_slots_enabled` is `off` (the default), no group slot is
+created and replication behaves exactly as before. Existing deployments upgrade
+transparently; the group-slot catalog objects are added by the extension
+upgrade and remain dormant until the feature is enabled.

docs/managing/index.md

@@ -9,3 +9,4 @@ The Spock extension is a powerful addition to any PostgreSQL installation; manag
 * use [Snowflake sequences](snowflake.md) for sequence management in a distributed cluster.
 * use [the Lolor extension](lolor.md) to replicate large objects.
 * enable [automatic DDL replication](spock_autoddl.md).
+* use [group replication slots](group_slots.md) to retain WAL at the oldest safe position for the whole replication group.

docs/spock_release_notes.md

@@ -1,5 +1,30 @@
 # Spock Release Notes
 
+## Spock 6.1.0
+
+### Highlights
+
+* **Group replication slots** — opt-in, BDR/PGD-style group slots that retain
+  WAL at the oldest safe position for the whole replication group. Each Spock
+  database maintains one internally managed, inactive logical slot whose
+  horizon is advanced by a background worker only when every required member
+  has fresh, durable progress for the current membership generation. Advancement
+  is refused during join, part, unknown node state, stale progress, missing
+  slot state, or membership-generation mismatch. New GUCs
+  (`spock.group_slots_enabled`, `spock.group_slots_worker_interval`,
+  `spock.group_slots_progress_staleness_timeout`,
+  `spock.group_slots_safety_mode`), catalog tables, and SQL functions
+  (`spock.local_group_slot_name()`, `spock.group_slot_status()`,
+  `spock.advance_group_slot()`, `spock.repair_group_slot()`) are added. The
+  feature is disabled by default and does not change replication behaviour when
+  off. See [Group replication slots](managing/group_slots.md).
+
+### Upgrading
+
+Run `ALTER EXTENSION spock UPDATE TO '6.1.0';` (or `ALTER EXTENSION spock
+UPDATE;`). The upgrade only adds new, dormant objects; no existing behaviour
+changes until `spock.group_slots_enabled` is turned on.
+
 ## Spock 6.0.0
 
 The on-disk catalog format and the GUC surface both change in this release;

include/spock.h

@@ -24,8 +24,8 @@
 #include "spock_fe.h"
 #include "spock_node.h"
 
-#define SPOCK_VERSION "6.0.0"
-#define SPOCK_VERSION_NUM 60000
+#define SPOCK_VERSION "6.1.0"
+#define SPOCK_VERSION_NUM 60100
 
 #define EXTENSION_NAME "spock"
 

include/spock_group_slot.h

@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ *
+ * spock_group_slot.h
+ *		Group replication slot subsystem.
+ *
+ * One inactive logical slot per database pins WAL at the oldest position
+ * still needed by the group. Decision logic lives in the spock.group_slot_*
+ * SQL functions; this module owns naming, the worker, and lifecycle hooks.
+ *
+ * Copyright (c) 2022-2026, pgEdge, Inc.
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, The Regents of the University of California
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SPOCK_GROUP_SLOT_H
+#define SPOCK_GROUP_SLOT_H
+
+#include "postgres.h"
+#include "fmgr.h"
+
+/* Prefix must not match the "spk_.*" per-subscription cleanup pattern. */
+#define SPOCK_GROUP_SLOT_PREFIX "spkgrp_"
+
+typedef enum SpockGroupSlotSafetyMode
+{
+	SPOCK_GROUP_SLOT_SAFETY_STRICT = 0, /* refuse any unsafe action */
+	SPOCK_GROUP_SLOT_SAFETY_REPAIR,		/* allow guarded repair actions */
+	SPOCK_GROUP_SLOT_SAFETY_OFF			/* advisory only (no advancement) */
+} SpockGroupSlotSafetyMode;
+
+/* GUCs (defined in spock.c) */
+extern bool spock_group_slots_enabled;
+extern int	spock_group_slots_worker_interval;
+extern int	spock_group_slots_staleness_timeout;
+extern int	spock_group_slots_safety_mode;
+
+/* Build the local node's slot name; false when there is no local node. */
+extern bool spock_build_local_group_slot_name(Name out_name);
+
+/* Seed metadata for a new local node; call inside an open transaction. */
+extern void spock_group_slot_init_local(Oid node_id);
+
+/* Drop the local group slot and metadata (from spock_drop_node() only). */
+extern void spock_group_slot_drop_local(void);
+
+/* Worker entry point (registered per database by the manager). */
+PGDLLEXPORT void spock_group_slot_main(Datum main_arg);
+
+#endif							/* SPOCK_GROUP_SLOT_H */

include/spock_worker.h

@@ -24,8 +24,9 @@ typedef enum
 	SPOCK_WORKER_NONE,			/* Unused slot. */
 	SPOCK_WORKER_MANAGER,		/* Manager. */
 	SPOCK_WORKER_APPLY,			/* Apply. */
-	SPOCK_WORKER_SYNC			/* Special type of Apply that synchronizes one
+	SPOCK_WORKER_SYNC,			/* Special type of Apply that synchronizes one
 								 * table. */
+	SPOCK_WORKER_GROUP_SLOT		/* Group-slot maintainer (one per database). */
 } SpockWorkerType;
 
 typedef enum
@@ -170,6 +171,7 @@ extern int	spock_worker_register(SpockWorker *worker);
 extern void spock_worker_attach(int slot, SpockWorkerType type);
 
 extern SpockWorker *spock_manager_find(Oid dboid);
+extern SpockWorker *spock_group_slot_find(Oid dboid);
 extern SpockWorker *spock_apply_find(Oid dboid, Oid subscriberid);
 extern List *spock_apply_find_all(Oid dboid);
 

mkdocs.yml

@@ -67,6 +67,7 @@ nav:
     - Using Snowflake Sequences: managing/snowflake.md
     - Using Lolor to Manage Large Objects: managing/lolor.md
     - Using Automatic DDL Replication: managing/spock_autoddl.md
+    - Using Group Replication Slots: managing/group_slots.md
   - Adding or Removing Nodes:
     - Modifying a Cluster: modify/index.md
     - Using Z0DAN: