Merge branch 'main' into docs/husk-sweep-upstream-marker

Author: gloryfromca

CHANGELOG.md

@@ -1070,8 +1070,32 @@ Manually trigger a registered OME strategy.
 
 | Field | Type | Notes |
 |---|---|---|
-| `status` | `"ok" \| "timeout"` | Whether the strategy completed within the timeout |
+| `status` | `"ok" \| "timeout" \| "not_dispatched"` | `ok` = every dispatched run settled — a dead-lettered run still counts as settled (see `runs[*].error`); `timeout` = at least one run had not settled when `timeout` elapsed; `not_dispatched` = no strategy was dispatched (see below) |
 | `name` | `string` | Echoes the requested strategy name |
+| `dispatched` | `int` | Number of strategy routes enqueued. `0` iff `status == "not_dispatched"` |
+| `runs` | `list[RunSummary]` | One entry per strategy run *attempt*, not per dispatched route: `{run_id: string, status: string, error?: string}`. A strategy that retried before settling contributes multiple entries sharing one `event_id`. `status` is one of `running` / `success` / `failed` / `dead_letter` / `crashed`. Includes dead-lettered runs |
+
+**`not_dispatched`** means every subscriber was rejected by one of the
+four dispatch gates (`_routes_to` / `enabled` / `applies_to` /
+`Counter`). The most common cause is forgetting `"force": true` on a
+strategy that is `enabled=false` in `ome.toml` — e.g. triggering
+`reflect_episodes` without `force` while it is disabled in config
+returns `{"status": "not_dispatched", "dispatched": 0, "runs": []}`
+instead of an error.
+
+> `status: "ok"` means all dispatched strategy runs settled — including
+> runs that dead-lettered (their errors are in `runs[*].error`). It does
+> **not** mean the LanceDB index has caught up. Markdown is written
+> synchronously; the index syncs asynchronously (see
+> [Eventual consistency](#eventual-consistency)).
+>
+> If you need read-your-write semantics, poll `GET /health`'s
+> `cascade.pending` field until it reads `0` on two consecutive samples
+> (a single zero can be a false convergence — the watcher-input window
+> can briefly report an empty queue between md write and enqueue).
+
+See [docs/openapi.json](openapi.json) for the exact generated schema
+(`TriggerResponse` / `RunSummary`) behind this table.
 
 #### Errors
 

docs/api.md

@@ -3,7 +3,7 @@
   "info": {
     "title": "everos",
     "description": "md-first memory extraction framework",
-    "version": "1.2.2"
+    "version": "1.2.3"
   },
   "paths": {
     "/health": {
@@ -222,7 +222,7 @@
           "ome"
         ],
         "summary": "Trigger",
-        "description": "Manually trigger a registered OME strategy and wait for completion.",
+        "description": "Manually trigger a registered OME strategy and wait for its runs to\nsettle. Returns without waiting for the LanceDB index — the response\nreflects markdown state only; poll ``GET /health``'s ``cascade.pending``\nfor index convergence (two consecutive zero samples to guard against the\nwatcher-input window). See docs/api.md#eventual-consistency.",
         "operationId": "trigger_api_v1_ome_trigger_post",
         "requestBody": {
           "content": {
@@ -982,7 +982,7 @@
           "ome"
         ],
         "summary": "Trigger",
-        "description": "Manually trigger a registered OME strategy and wait for completion.",
+        "description": "Manually trigger a registered OME strategy and wait for its runs to\nsettle. Returns without waiting for the LanceDB index — the response\nreflects markdown state only; poll ``GET /health``'s ``cascade.pending``\nfor index convergence (two consecutive zero samples to guard against the\nwatcher-input window). See docs/api.md#eventual-consistency.",
         "operationId": "trigger_api_v2_ome_trigger_post",
         "requestBody": {
           "content": {
@@ -3089,6 +3089,36 @@
         ],
         "title": "MessageItemDTO"
       },
+      "RunSummary": {
+        "properties": {
+          "run_id": {
+            "type": "string",
+            "title": "Run Id"
+          },
+          "status": {
+            "type": "string",
+            "title": "Status"
+          },
+          "error": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Error"
+          }
+        },
+        "type": "object",
+        "required": [
+          "run_id",
+          "status"
+        ],
+        "title": "RunSummary",
+        "description": "One strategy run within a trigger response."
+      },
       "SearchAgentCaseItem": {
         "properties": {
           "id": {
@@ -4006,15 +4036,28 @@
           "name": {
             "type": "string",
             "title": "Name"
+          },
+          "dispatched": {
+            "type": "integer",
+            "title": "Dispatched"
+          },
+          "runs": {
+            "items": {
+              "$ref": "#/components/schemas/RunSummary"
+            },
+            "type": "array",
+            "title": "Runs",
+            "default": []
           }
         },
         "type": "object",
         "required": [
           "status",
-          "name"
+          "name",
+          "dispatched"
         ],
         "title": "TriggerResponse",
-        "description": "Response body for ``POST /api/v2/ome/trigger``."
+        "description": "Response body for ``POST /api/v2/ome/trigger``.\n\n``status`` distinguishes three outcomes that were previously masked as\na single ``ok``:\n\n- ``ok``: at least one strategy was dispatched and all runs settled\n  within ``timeout``. Individual run outcomes are in ``runs`` (a\n  ``dead_letter`` there is still ``ok`` at this level — the strategy\n  *ran*, it just failed permanently).\n- ``timeout``: at least one strategy was dispatched but the engine did\n  not go idle within ``timeout``. Runs may be partially complete;\n  poll ``GET /health`` for cascade convergence separately.\n- ``not_dispatched``: no strategy was dispatched — the subscriber was\n  rejected by one of the dispatch gates (``_routes_to`` / ``enabled`` /\n  ``applies_to`` / ``Counter``). Common cause: forgetting\n  ``force=true`` on a strategy that is ``enabled=false`` in ome.toml."
       },
       "UnprocessedMessageDTO": {
         "properties": {

docs/openapi.json

@@ -1,6 +1,6 @@
 [project]
 name = "everos"
-version = "1.2.2"
+version = "1.2.3"
 description = "EverOS — local-first markdown memory framework for AI agents and user chats; lightweight, dev-friendly, small-team"
 license = {text = "Apache-2.0"}
 readme = "README.md"

pyproject.toml

@@ -29,8 +29,12 @@
 # Atomic fact extraction runs per memcell. Always enabled — search
 # depends on atomic facts for MaxSim retrieval.
 
-# Foresight extraction (runs per memcell). Heavy LLM call — set
-# enabled = false to skip in evaluation / benchmark runs.
+# Foresight extraction (runs per memcell). Heavy LLM call.
+#
+# DISABLED BY DEFAULT since 1.2.3: nothing in EverOS reads foresights yet
+# — no search route surfaces them, no prompt slot consumes them — so
+# running it spends one LLM call per sender per memcell on write-only
+# data. Uncomment to opt in; it works on agent and chat input alike.
 # [strategies.extract_foresight]
 # enabled = true
 

src/everos/config/default_ome.toml

@@ -16,6 +16,8 @@
         # Frontmatter schema chassis
         BaseFrontmatter, UserScopedFrontmatter, AgentScopedFrontmatter,
         DailyLogPathMixin, SkillPathMixin,
+        # Path safety
+        sanitize_dirname,
         # Async SQLite (SQLModel / SA 2.0)
         create_system_engine, create_session_factory, session_scope,
         SQLModel, Field, Relationship, BaseTable, RepoBase,
@@ -49,6 +51,7 @@
 from .markdown import parse_frontmatter as parse_frontmatter
 from .markdown import parse_structured_entry as parse_structured_entry
 from .markdown import render_structured_entry as render_structured_entry
+from .markdown import sanitize_dirname as sanitize_dirname
 from .markdown import split_entries as split_entries
 from .memory_root import MemoryRoot as MemoryRoot
 from .memory_root import app_dir_name as app_dir_name
@@ -100,6 +103,7 @@
     "project_dir_name",
     "project_id_from_dir",
     "render_structured_entry",
+    "sanitize_dirname",
     "session_scope",
     "split_entries",
     "touch",

src/everos/core/persistence/__init__.py

@@ -21,6 +21,9 @@
         KnowledgeScopedMixin, KnowledgeDocumentPathMixin,
         KnowledgeTopicPathMixin,
     )
+
+External usage (path safety):
+    from everos.core.persistence.markdown import sanitize_dirname
 """
 
 from .entries import Entry as Entry
@@ -42,6 +45,7 @@
 from .frontmatter import dump_frontmatter as dump_frontmatter
 from .frontmatter import parse_frontmatter as parse_frontmatter
 from .parsed import ParsedMarkdown as ParsedMarkdown
+from .path_safety import sanitize_dirname as sanitize_dirname
 from .reader import MarkdownReader as MarkdownReader
 from .writer import MarkdownWriter as MarkdownWriter
 
@@ -66,5 +70,6 @@
     "parse_frontmatter",
     "parse_structured_entry",
     "render_structured_entry",
+    "sanitize_dirname",
     "split_entries",
 ]

src/everos/core/persistence/markdown/__init__.py

@@ -36,6 +36,8 @@
 import yaml
 from pydantic import BaseModel, ConfigDict
 
+from .path_safety import sanitize_dirname
+
 # ── YAML helpers ────────────────────────────────────────────────────────
 
 _DELIM = "---"
@@ -229,6 +231,15 @@ class AgentSkillFrontmatter(SkillPathMixin, AgentScopedFrontmatter):
             SKILL_DIR_PREFIX: ClassVar[str] = "skill_"
             SKILL_MAIN_FILENAME: ClassVar[str] = "SKILL.md"
             ...
+
+    ``skill_dir_name`` / ``sanitize_skill_name`` are the single
+    sanitization point both ``AgentSkillWriter`` and ``AgentSkillReader``
+    derive their ``skill_<name>`` directory segment from, and that
+    ``memory.strategies.extract_agent_skill._persist_skill`` uses to
+    sanitize LLM-emitted ``skill_name`` *before* constructing
+    ``AgentSkillFrontmatter`` — ``skill_name`` is LLM output and must not
+    reach the filesystem, or the frontmatter's traversal validator,
+    unsanitized (CWE-22).
     """
 
     SKILLS_CONTAINER_NAME: ClassVar[str]
@@ -244,6 +255,83 @@ def path_glob(cls) -> str:
             f"{cls.SKILL_DIR_PREFIX}*/{cls.SKILL_MAIN_FILENAME}"
         )
 
+    @classmethod
+    def sanitize_skill_name(cls, skill_name: str) -> str:
+        """Bare sanitized skill name (no ``skill_`` prefix).
+
+        The single sanitization point for a skill's ``name`` value itself —
+        as opposed to :meth:`skill_dir_name`, which additionally prefixes
+        it for the directory segment. Callers building
+        ``AgentSkillFrontmatter.name`` from LLM output (see
+        ``memory.strategies.extract_agent_skill._persist_skill``) route
+        through this *before* constructing the frontmatter, so
+        ``frontmatter.name`` ends up byte-identical to the directory-derived
+        name rather than merely idempotent-if-resanitized.
+
+        This is lossy: distinct raw names can collapse onto the same
+        sanitized name. Dropped punctuation, space/underscore collapse, and
+        the 50-character cap are the visible cases (``"fix django"`` and
+        ``"fix_django"`` both become ``"fix_django"``; ``"fix!django"`` and
+        ``"fixdjango"`` both become ``"fixdjango"``). The larger case is
+        every combining mark: a combining mark alone is not ``\\w``, so it
+        is stripped regardless of script, and two names that differ only in
+        their marks collide — e.g. Devanagari ``"किताब"`` and ``"कताब"``
+        both sanitize to ``"कतब"``; the same holds for Thai tone marks,
+        Hebrew niqqud, and Arabic harakat.
+
+        Case is *not* folded, which makes the collision above
+        filesystem-dependent rather than universal, and is the dimension an
+        LLM varies most freely: ``"Fix Django"`` → ``"Fix_Django"`` and
+        ``"fix django"`` → ``"fix_django"`` are two distinct sanitized
+        names, so they are two rows in LanceDB (a case-sensitive Python
+        string key) but one directory on a case-insensitive filesystem —
+        macOS APFS and Windows NTFS in their default configurations. That
+        splits the invariant this seam otherwise maintains: the surviving
+        ``SKILL.md`` carries one of the two names in its frontmatter while
+        the index still advertises both, so a search hit on the shadowed
+        name resolves to the other skill's content. On a case-sensitive
+        filesystem the same pair simply stays two independent skills.
+
+        Because ``AgentSkillWriter.write_main`` is a full-file replace and
+        the LanceDB primary key is ``f"{agent_id}_{sanitized_name}"``, a
+        collision means the later skill silently overwrites the earlier
+        one — its accumulated ``source_case_ids``, ``maturity_score``, and
+        body are lost, not merged.
+
+        This is deliberate, not an oversight — but not because a collision
+        "usually reads as an intended update". ``_persist_skill`` sanitizes
+        *before* constructing the frontmatter, so the LLM is shown the
+        already-sanitized name in ``existing_relevant_skills``; when it
+        then emits a raw name like ``"fix django"`` after having just been
+        shown ``"fix_django"``, it has affirmatively treated them as two
+        different skills, and the write silently merges them anyway.
+
+        What justifies accepting it is narrower: the two alternatives are
+        both worse here. Detecting a collision and raising would
+        reintroduce the dead-letter DoS this sanitizer was built to avoid —
+        LLM output would again decide whether a run survives. Appending a
+        disambiguating suffix is the real candidate and is left for a
+        deliberate design pass, not dismissed: it does *not* break the
+        ``frontmatter.name`` ≡ directory-suffix identity (writing
+        ``"fix_django_2"`` into both keeps that intact), but it does need a
+        collision probe on a path that currently touches no other skill,
+        and a rule for the case-insensitive-filesystem variant above where
+        the probe must compare case-folded while the key stays exact.
+        """
+        return sanitize_dirname(skill_name, fallback="unnamed")
+
+    @classmethod
+    def skill_dir_name(cls, skill_name: str) -> str:
+        """Sanitized ``skill_<name>`` directory segment (traversal-safe).
+
+        Idempotent in ``skill_name``: calling this again on an already
+        sanitized name (e.g. one recovered by walking the directory tree)
+        returns the same segment, so a reader deriving ``skill_name`` from
+        the on-disk directory and a writer deriving it from raw LLM output
+        land on the same path.
+        """
+        return f"{cls.SKILL_DIR_PREFIX}{cls.sanitize_skill_name(skill_name)}"
+
 
 class ProfilePathMixin:
     """Path strategy for single-file profile markdown.

src/everos/core/persistence/markdown/frontmatter.py

@@ -0,0 +1,83 @@
+"""``sanitize_dirname`` — the single path-safety primitive for md directory names.
+
+Several markdown layouts turn free-text into a filesystem path segment:
+knowledge document/category titles, and agent-skill names. Both sources are
+untrusted in the same way — knowledge titles come from parsed source
+documents, skill names come straight from LLM output — so a name containing
+``../`` or a path separator must never survive into a directory segment
+(CWE-22 path traversal).
+
+This module is the one place that decision is made. Callers that need a
+filesystem-safe segment from a free-text string route through
+:func:`sanitize_dirname` rather than keeping a private regex copy — see
+``writers/knowledge_writer.py`` and
+:meth:`SkillPathMixin.skill_dir_name() <.frontmatter.SkillPathMixin.skill_dir_name>`.
+Some callers (``knowledge_writer.py``) concatenate the result directly under a
+shared directory with no per-caller prefix, so the guarantee below has to hold
+on its own, without relying on a prefix like ``skill_`` to absorb a degenerate
+result.
+
+``sanitize_dirname`` is idempotent (``sanitize_dirname(sanitize_dirname(x),
+fb) == sanitize_dirname(x, fb)``): a name built by re-sanitizing an
+already-sanitized segment (e.g. one derived by walking the directory tree)
+lands on the same string as sanitizing the original raw name. That property
+is what lets a reader and a writer agree on a path even when one side has
+only the raw name and the other only the on-disk directory name.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+
+_MAX_DIRNAME_LEN = 50
+_SAFE_CHARS = re.compile(r"[^\w\-.]", re.UNICODE)
+_DEGENERATE = frozenset({"", ".", ".."})
+
+
+def sanitize_dirname(raw: str, fallback: str) -> str:
+    """Produce a safe directory/file name segment from free-text input.
+
+    * NFC-normalize first. For an ordinary decomposed (NFD) input — a base
+      letter plus a combining mark, e.g. ``"e"`` + combining acute accent —
+      this collapses to the precomposed form before the character filter
+      runs, so the accent survives (a combining mark alone is not ``\\w``
+      and would otherwise be silently stripped). This is best-effort, not a
+      guarantee: for the ~1,082 Unicode *composition exclusion* codepoints
+      (e.g. Devanagari ``क़``/``ख़``, U+0958/U+0959), NFC does the
+      opposite — it *decomposes* an already-precomposed exclusion
+      character, because recomposing it is explicitly excluded from the
+      NFC algorithm, and the resulting combining mark is then stripped just
+      the same. Normalizing here improves fidelity for the common case; it
+      does not make every Unicode script round-trip losslessly.
+    * Replace spaces with underscores.
+    * Strip characters outside ``[a-zA-Z0-9_\\-.]`` (``\\w`` is Unicode-aware,
+      so CJK and other non-ASCII scripts survive readably). Note that ``.``
+      is a *safe* character, not stripped — a run of literal dots is a legal
+      result of this step.
+    * Truncate to 50 characters.
+    * Fall back to *fallback* if the result is empty, ``"."``, or ``".."``.
+
+    Every path separator (``/``, ``\\``) is stripped by the character-class
+    filter, so no separator survives and the result is always exactly one
+    path component — it can never be split into multiple segments by a
+    downstream ``Path(...) / result``. The fallback on ``""`` / ``"."`` /
+    ``".."`` closes the remaining gap: those are the only single components
+    that resolve to *no new child* (``""`` and ``"."`` both mean "this same
+    directory", ``".."`` means "its parent") rather than a genuinely new
+    entry. With both guarantees together, ``Path(some_dir) / sanitize_dirname(raw, fb)``
+    can never escape ``some_dir`` and never silently collapses back onto it
+    or its parent — unconditionally, including for a caller with no
+    additional prefix (like ``skill_``) protecting the segment.
+
+    This function is lossy and not injective: distinct inputs can sanitize
+    to the same output (dropped characters, space/underscore collapse, and
+    truncation are all many-to-one). A caller that needs distinct outputs
+    for distinct inputs must disambiguate before or after calling this —
+    the function itself makes no such guarantee.
+    """
+    slug = unicodedata.normalize("NFC", raw)
+    slug = slug.replace(" ", "_")
+    slug = _SAFE_CHARS.sub("", slug)
+    slug = slug[:_MAX_DIRNAME_LEN]
+    return slug if slug not in _DEGENERATE else fallback