feat: Add _vision_sidecar plugin

plugins/_vision_sidecar/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vision Sidecar contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

plugins/_vision_sidecar/README.md

@@ -0,0 +1,154 @@
+# Vision Sidecar
+
+![Vision Sidecar](webui/thumbnail.jpg)
+
+Tolerant `vision_load` + optional dedicated Vision Model.
+
+## Why
+
+Two recurring pain points in Agent Zero:
+
+1. **Bare-string bug.** Core `vision_load` requires `paths` as a list. A bare string `"/a.png"` is iterated char-by-char, loads 0 images, and wastes a turn with no error.
+2. **No vision, no images.** Frontier reasoners (GLM 5.2/5.3, DeepSeek V4 Flash/Pro) are cheap and strong but have no vision. On stock A0 that means no `vision_load` at all — even though a cheap `gpt-4o-mini` or `qwen2-vl` could read the image for pennies.
+
+Vision Sidecar fixes both in one plugin.
+
+## What it does
+
+### 1. Tolerant `vision_load`
+
+`vision_load` now accepts `paths` as `string` or `list[str]`.
+
+- `{"paths": "/a.png"}` is treated as `["/a.png"]`
+- Handles harness quirks: JSON-encoded array strings (`"[\"/a.png\"]"`), quoted single paths (`"\"/a.png\""`) 
+- Wrong types return a clear tool error — never a `Message misformat`
+
+### 2. Delegated Vision Model
+
+Configure an optional **Vision Model** in **Settings → Model Presets → Vision Model**.
+
+> Optional dedicated model for vision_load — used when Main has no vision. Leave empty to use Main's vision.
+
+When set:
+
+- `vision_load(paths, query?, raw?)` materializes images, calls the Vision Model with `query + images`, and returns a **text capsule** instead of injecting `~1500 tok/image` into the main history.
+- Your Main (GLM, DeepSeek) never sees raw pixels — only ~300 tokens of focused text. Saves thousands of tokens per future turn.
+- `query` is a focused instruction: `"read the top-right error toast"`, `"locate the login button and give coordinates"`. Empty → generic precise description.
+- `raw=true` bypasses delegation and injects images directly into Main. Use for side-by-side comparison when Main must see pixels.
+- Large images over ~900 KB are auto-compressed to 1280×960 JPEG before the vision call to avoid `Request Entity Too Large` (4 MB PNG → ~250 KB).
+
+When empty: legacy path — images are injected as `RawMessage` for `chat_model.vision == true`, appearance identical to stock A0.
+
+Preset defaults for the Vision slot: **64000 context, 70% for history** (new presets only). Existing presets are untouched.
+
+## Requirements
+
+- Agent Zero. If your **Settings → Model Presets → Edit** already shows **Main / Vision / Utility / Embedding** (only if you're updating the plugin), nothing else to do.
+- If it only shows **Main / Utility / Embedding** (on any A0 instance), run the one-time Vision-slot patch below — otherwise Vision Sidecar still works, but `vision_load` falls back to tolerant direct injection (no delegation).
+- Any LiteLLM-compatible vision model for the Vision slot (tested with `openai/gpt-4o-mini`, `qwen2-vl`).
+
+## Installation
+
+### From ZIP
+
+1. Download `vision_sidecar.zip` from Releases
+2. Agent Zero → **Settings → Plugins → Install → From ZIP** → select the ZIP
+3. Add the Vision slot via the script (check below)
+4. Restart the WebUI (`Ctrl+Shift+R`)
+
+### From Git
+
+```bash
+git clone https://github.com/GreifMax/a0-vision-sidecar
+cp -r a0-vision_sidecar /a0/usr/plugins/vision_sidecar
+# restart Agent Zero
+```
+
+### Add Vision slot
+
+Since **v0.4.0** the Vision slot is applied **automatically on install** (via the plugin's `install()` hook) — no manual step. If Model Presets still shows only Main / Utility / Embedding (e.g. after an A0 core update overwrote `plugins/_model_config`), re-run it in one click:
+
+- **Settings → Plugins → Vision Sidecar → Execute** (preferred), or
+- the manual script below (same logic, also usable for `--status` / `--restore`)
+
+The patcher is **self-contained pure Python** — no git or `patch(1)` required, idempotent, and creates `.vision_sidecar.bak` backups of every modified file.
+
+```bash
+# any directory works; it auto-finds the Agent Zero root
+bash /a0/usr/plugins/vision_sidecar/scripts/enable_vision_slot.sh
+# or:  python3 /a0/usr/plugins/vision_sidecar/scripts/enable_vision_slot.py
+```
+
+Docker (run **inside** the Agent Zero container, not on the host — `plugins/` only exists in the image):
+
+```bash
+docker exec -it <agent-zero-container> bash /a0/usr/plugins/vision_sidecar/scripts/enable_vision_slot.sh
+```
+
+Options:
+
+| Command | Effect |
+| --- | --- |
+| (no args) | Apply patch (skips files already patched) |
+| `--status` | Show per-file state without changing anything |
+| `--restore` | Restore all original files from `.bak` backups |
+
+If auto-detection fails, point it at your install: `A0_ROOT=/path/to/agent-zero bash enable_vision_slot.sh`.
+
+Then restart Agent Zero and hard-refresh the browser (Ctrl+Shift+R). Model Presets will show **Main / Vision / Utility / Embedding**.
+
+> Note: A0 updates can overwrite `plugins/_model_config`. After updating, rerun the script — it is idempotent and will re-apply cleanly.
+
+## Configuration
+
+1. **Settings → Model Presets → Edit** → fill **Vision Model** with your cheap vision helper (provider + name + key). Leave empty to use Main's vision.
+2. **Main Model → Supports Vision on** → optional **Overrides Vision Model** switch appears right under it: when on, Main's native vision is always used for that preset and the Vision Model is ignored; when off (default), the dedicated Vision Model handles vision when configured. The chat model switcher hides the Vision row for presets where the override is on, and the Agent Config preset preview shows "Overwritten by Main" in place of the Vision model when the override is active.
+3. **Settings → Plugins → Vision Sidecar** → tune the delegated system prompt and timeout if needed.
+
+New presets automatically get `Vision: 64000 / 0.7`. Current presets keep their values. The override flag is per-preset (never inherited from Default).
+
+## Usage (By A0)
+
+```json
+{
+  "tool_name": "vision_load",
+  "tool_args": {
+    "paths": ["/a0/usr/uploads/screenshot.png"],
+    "query": "read the error message in the top-right"
+  }
+}
+```
+
+- With Vision Model set -> chat shows thumbnails + `N images sent, M images skipped - Description: "..."` (counts + vision-model capsule in one line). The tool step always includes a Query row (Paths / Tool Name / Query / Result), even when the call omitted `query`.
+- The delegated `vision_load` prompt declares an explicit JSON tool schema (`paths`, `query`, `raw`), so models see `query` as a real parameter. When the Main model overrides vision (or there is no Vision Model), the stock prompt is used and `query` is absent from the schema. The schema rejects unknown properties (`additionalProperties: false`), and the tool normalizes prompt-style aliases (`Prompt`, `question`, `instruction`, ...) into `query`, so the vision model always receives the intended focus text even if a model ignores the schema.
+- With `raw=true` → forces direct injection even when Vision Model is set.
+- Without Vision Model -> fully stock: stock prompt (no `query`/`raw`), `Loaded images: N` with thumbnails for Main vision, and no vision tool at all when Main has no vision.
+- With **Overrides Vision Model** on (Main vision-capable presets) -> Main's native vision is used even though a Vision Model is set; delegation (and the switcher's Vision row) is skipped for that preset.
+
+## Reliability notes
+
+- `vision_load` is safe inside the `parallel` tool: the job result is the real text capsule (or error), never a placeholder — the tool sets the authoritative response message in every outcome path.
+- Image blocks are injected into main history only when the preset's main model declares vision support (delegated and legacy paths alike). With a text-only main, delegated calls return the text capsule only and `raw=true` auto-delegates.
+- The `vision` flag is the user's declaration and governs: if it mislabels a text-only provider, the provider may reject image blocks (`400 content.type invalid`) — fix the flag in Model Presets; the text capsule still carries the answer meanwhile.
+- Inside `parallel` workers the tool writes the `Result` row of its log item. The parent job aggregator skips the body text write when the Result row already carries the same text, so the step shows the text only once (inside the Result row). Errors and tools without a Result row still get body text.
+- With no Vision Model configured and a text-only main, the tool reports loaded/skipped counts plus an explicit note that images were not injected.
+
+## File layout
+
+```
+plugin.yaml
+default_config.yaml
+LICENSE
+README.md
+thumbnail.jpg              ← plugin list image (256×256, ≤20 KB)
+helpers/vision_model.py    ← preset-aware vision dispatch + compression
+tools/vision_load.py       ← tolerant paths + delegation
+prompts/agent.system.tool.vision_load.md
+extensions/python/system_prompt/_10_vision_sidecar_guidance.py
+webui/config.html
+webui/thumbnail.jpg/png
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).

plugins/_vision_sidecar/default_config.yaml

@@ -0,0 +1,4 @@
+behaviour:
+  delegated_system: "You are a precise vision analyst. Answer only what was asked about the image(s). Be concise, factual, mention positions/coordinates when asked to locate."
+  max_tokens: 2000
+  timeout: 300

plugins/_vision_sidecar/execute.py

@@ -0,0 +1,28 @@
+"""Vision Sidecar — manual maintenance script.
+
+Triggered from Settings → Plugins → Vision Sidecar (Execute). Useful after an
+A0 update that overwrote plugins/_model_config: re-applies the Vision-slot
+patch idempotently (already-patched files are skipped). The install hook does
+this automatically on plugin install, so most users never need this.
+"""
+import sys
+from pathlib import Path
+
+
+def main() -> int:
+    try:
+        hooks = Path(__file__).resolve().parent / "hooks.py"
+        import importlib.util
+        spec = importlib.util.spec_from_file_location("vision_sidecar_hooks", hooks)
+        mod = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(mod)
+        mod.install()
+        print("Vision Sidecar maintenance completed successfully.")
+        return 0
+    except Exception as e:
+        print(f"ERROR: {e}")
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

plugins/_vision_sidecar/execute_record.json

@@ -0,0 +1 @@
+{"executed_at": "2026-08-19T18:55:52.671678+02:00", "exit_code": 0}

plugins/_vision_sidecar/extensions/python/system_prompt/_10_vision_sidecar_guidance.py

@@ -0,0 +1,21 @@
+from helpers.extension import Extension
+
+_D = "Vision Sidecar active: a dedicated Vision Model IS configured (Model Presets -> Vision Model). vision_load DELEGATES:"
+_D += chr(10) + "- send a focused query about the images (e.g. read the top-right error toast, locate the login button and give its position) - you will get a concise text capsule, not pixels."
+_D += chr(10) + "- if Main cannot see images, ~1500 tok/image stays out of your context (capsule only); if Main is vision-capable, the images are also attached to history alongside the capsule. Use raw=true only when you need pixels without a capsule (e.g. side-by-side comparison)."
+_GUIDANCE_DELEGATED = _D
+
+
+class VisionSidecarGuidance(Extension):
+    async def execute(self, system_prompt: list[str] | None = None, **kwargs):
+        # With no dedicated Vision Model the plugin is fully stock: no guidance,
+        # stock prompt, stock result format.
+        if system_prompt is None:
+            return
+        try:
+            from usr.plugins.vision_sidecar.helpers.vision_model import has_vision_model
+            if not has_vision_model(self.agent):
+                return
+        except Exception:
+            return
+        system_prompt.append(_GUIDANCE_DELEGATED)

plugins/_vision_sidecar/helpers/vision_model.py

@@ -0,0 +1,128 @@
+from __future__ import annotations
+
+from typing import Any
+
+import models
+
+DEFAULT_DELEGATED_SYSTEM = (
+    "You are a precise vision analyst. Answer only what was asked about the image(s). "
+    "Be concise, factual, mention positions/coordinates when asked to locate."
+)
+
+def get_behaviour(agent: Any = None) -> dict[str, Any]:
+    from helpers import plugins
+    cfg = plugins.get_plugin_config("vision_sidecar", agent=agent) or {}
+    b = cfg.get("behaviour") or {}
+    if not isinstance(b, dict):
+        b = {}
+    return {
+        "delegated_system": str(b.get("delegated_system") or DEFAULT_DELEGATED_SYSTEM),
+        "max_tokens": int(b.get("max_tokens") or 2000),
+        "timeout": float(b.get("timeout") or 300),
+    }
+
+def get_vision_model_config(agent: Any = None) -> dict[str, Any]:
+    """Read Vision Model from Model Presets (vision slot), not from sidecar config.
+
+    Preset storage is in _model_config plugin: PRESET_SLOT_CONFIG_SECTIONS["vision"] = "vision_model".
+    Uses get_effective_config so per-chat overrides are respected.
+    """
+    try:
+        from plugins._model_config.helpers.model_config import get_effective_config
+        cfg = get_effective_config(agent) or {}
+        vm = cfg.get("vision_model") or {}
+        if not isinstance(vm, dict):
+            return {}
+        provider = str(vm.get("provider") or "").strip()
+        name = str(vm.get("name") or "").strip()
+        if not provider and not name:
+            return {}
+        return vm
+    except Exception:
+        return {}
+
+def has_vision_model(agent: Any = None) -> bool:
+    vm = get_vision_model_config(agent)
+    if not (vm.get("provider") and vm.get("name")):
+        return False
+    try:
+        from plugins._model_config.helpers.model_config import get_chat_model_config
+        chat_cfg = get_chat_model_config(agent) or {}
+        if chat_cfg.get("vision", False) and chat_cfg.get("vision_override", False):
+            return False  # Main's native vision overrides the Vision Model
+    except Exception:
+        pass
+    return True
+
+def build_vision_model(agent: Any = None):
+    vm = get_vision_model_config(agent)
+    if not vm:
+        return None
+    from plugins._model_config.helpers.model_config import build_model_config
+    mc = build_model_config(vm, models.ModelType.CHAT)
+    mc.vision = True
+    return models.get_chat_model(mc.provider, mc.name, model_config=mc, **mc.build_kwargs())
+
+async def call_vision_model(
+    agent: Any,
+    images_a0_paths: list[str],
+    query: str,
+    delegated_system: str | None = None,
+    timeout: float = 300,
+) -> str:
+    """Call the dedicated vision preset model with images + query, return text capsule."""
+    import asyncio
+    from langchain_core.messages import HumanMessage, SystemMessage
+
+    model = build_vision_model(agent)
+    if model is None:
+        raise RuntimeError("vision_model not configured — set it in Model Presets → Vision Model")
+
+    behaviour = get_behaviour(agent)
+    system = (delegated_system or behaviour["delegated_system"]).strip() or DEFAULT_DELEGATED_SYSTEM
+    if query and query.strip():
+        user_text = query.strip()
+    else:
+        user_text = "Describe the image(s) precisely. Be concise, mention key objects, text, and layout."
+
+    content: list[dict[str, Any]] = [{"type": "text", "text": user_text}]
+    for pa in images_a0_paths:
+        url = pa
+        try:
+            import base64
+            from pathlib import Path as _Path
+            from helpers import files as _files
+            from helpers.images import compress_image as _compress
+            raw = str(pa or "").strip()
+            cand = None
+            if raw.startswith("/a0/"):
+                cand = _Path(_files.fix_dev_path(raw) if hasattr(_files, "fix_dev_path") else raw)
+                if not cand.exists():
+                    cand = _Path(raw)
+            else:
+                cand = _Path(raw)
+            if cand and cand.exists() and cand.is_file():
+                data = cand.read_bytes()
+                if len(data) > 900 * 1024:
+                    try:
+                        c = _compress(data, max_pixels=1280*960, quality=80)
+                        b64 = base64.b64encode(c).decode()
+                        url = f"data:image/jpeg;base64,{b64}"
+                    except Exception:
+                        url = pa
+                else:
+                    url = pa
+        except Exception:
+            url = pa
+        content.append({"type": "image_url", "image_url": {"url": url}})
+
+    messages = [SystemMessage(content=system), HumanMessage(content=content)]
+
+    async def _call():
+        kwargs = {"max_tokens": int(behaviour["max_tokens"])}
+        resp, _reason = await model.unified_call(messages=messages, explicit_caching=False, **kwargs)
+        return resp
+    try:
+        return await asyncio.wait_for(_call(), timeout=float(timeout or behaviour["timeout"]))
+    except asyncio.TimeoutError:
+        raise TimeoutError(f"vision_model timed out after {timeout}s")