Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion backend/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,11 @@ async def get_asset_content(asset_id: str) -> FileResponse:
abs_path = asset_store.get_absolute_path(rec)
if not abs_path.is_file():
raise HTTPException(status_code=404, detail="Asset file missing on disk")
return FileResponse(abs_path, media_type="image/png")

import mimetypes
guessed_type, _ = mimetypes.guess_type(str(abs_path))
content_type = guessed_type or ("video/mp4" if rec.media_type == "video" else "image/png")
return FileResponse(abs_path, media_type=content_type)


@app.post("/api/v1/workflow/cancel/{run_id}")
Expand Down
42 changes: 32 additions & 10 deletions docs/ACCEPTANCE_DOSSIER.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,25 +47,35 @@ All changes have been developed on focused feature branches and integrated into
| **[PR #14](https://github.com/BerryUIKI/AI-Studio/pull/14)** | `feature/m10-workflow-repair` | `feat(comfy): workflow validation and automated repair engine (M10)` | 3/3 Passed | [`60c3620`](https://github.com/BerryUIKI/AI-Studio/commit/60c3620) |
| **[PR #15](https://github.com/BerryUIKI/AI-Studio/pull/15)** | `feature/m11-discrete-gpus` | `feat(hardware): support non-NVIDIA discrete GPUs with DirectML and launch flag injection (M11)` | 3/3 Passed | [`7f700b2`](https://github.com/BerryUIKI/AI-Studio/commit/7f700b2) |
| **[PR #16](https://github.com/BerryUIKI/AI-Studio/pull/16)** | `feature/m12-cross-platform` | `feat(platform): cross-platform desktop adapters for macOS Metal and Linux (M12)` | 3/3 Passed | [`2551796`](https://github.com/BerryUIKI/AI-Studio/commit/2551796) |
| **[PR #17](https://github.com/BerryUIKI/AI-Studio/pull/17)** | `docs/dossier-roadmap-sync` | `docs: synchronize acceptance dossier and roadmap post v0.1 with dev integration commits` | 3/3 Passed | [`0a59aba`](https://github.com/BerryUIKI/AI-Studio/commit/0a59aba) |
| **[PR #18](https://github.com/BerryUIKI/AI-Studio/pull/18)** | `feature/repair-agent-safety` | `fix(workflow): enforce model family guards and ambiguity blocking in workflow repair` | 3/3 Passed | [`3cf860a`](https://github.com/BerryUIKI/AI-Studio/commit/3cf860a) |
| **[PR #19](https://github.com/BerryUIKI/AI-Studio/pull/19)** | `feature/hardware-matrix-realism` | `fix(hardware): accurately classify AMD, Intel, and Apple Silicon as candidate or source-compatible` | 3/3 Passed | [`c35f3b1`](https://github.com/BerryUIKI/AI-Studio/commit/c35f3b1) |
| **[PR #20](https://github.com/BerryUIKI/AI-Studio/pull/20)** | `feature/video-format-dossier-alignment` | `fix(video): support video containers and animated webp, fix MIME type, align dossier evidence` | 3/3 Passed | Pending |

---

## 3. Supported Versions & Hardware
## 3. Supported Versions & Hardware Evidence Tiers

### Operating Systems
- **Windows 10 / 11 (64-bit)**: Primary target platform for portable release distribution.
- **Cross-Platform Core**: macOS / Linux supported for source checkouts; verified through core portable test suite.
### Operating Systems & Platform Verification Status
- **Windows 10 / 11 (64-bit)**: **Physically Verified**. Tested on Windows 11 host with native Rust launcher, ComfyUI engine supervisor, single-instance Win32 mutex, and local inference.
- **macOS (Apple Silicon, Metal MPS)**: **Source Compatible**. Directory paths, browser launcher commands, Unix domain socket single-instance checks, and Metal fp16 flags implemented and unit-tested; native macOS binary bundling and physical Apple Silicon execution remain unverified in this session.
- **Linux (Ubuntu 22.04+)**: **Source Compatible**. Supervisor process handling, Unix domain socket locking, and headless CLI generation implemented and unit-tested; native Linux binary packaging and physical non-NVIDIA execution remain unverified in this session.

### Hardware Tiers
### Hardware Tiers & Real Evidence Separation
- **Tier 1 (Cloud-Only / Featherweight)**:
- **GPU**: None required (Zero GPU, Zero PyTorch on host).
- **RAM**: 4 GB RAM minimum.
- **Storage**: < 500 MB disk footprint.
- **Mode**: BYOK Cloud APIs (OpenAI, Fal.ai, SiliconFlow).
- **Tier 2 (NVIDIA Local Inference)**:
- **GPU**: NVIDIA RTX (Turing, Ampere, Ada Lovelace, Blackwell) with minimum 6 GB VRAM (8 GB+ recommended).
- **Driver**: NVIDIA Driver 535+ with CUDA 12.1+.
- **Engines**: ComfyUI (managed or external), Stable Diffusion WebUI (AUTOMATIC1111).
- **Mode**: BYOK Cloud APIs (OpenAI, Fal.ai, SiliconFlow). Real API calls verified.
- **Tier 2 (Physically Verified NVIDIA CUDA)**:
- **GPU**: NVIDIA RTX (RTX 3060 12GB Ampere, driver 572.70, CUDA 12.8 physically verified on host rig).
- **Engines**: ComfyUI (managed and external), Stable Diffusion WebUI (AUTOMATIC1111).
- **Tier 3 (Candidate Architecture - Unverified Physical Non-NVIDIA Discrete GPUs)**:
- **Hardware**: AMD Radeon (DirectML / ROCm) and Intel Arc (DirectML / OneAPI IPEX).
- **Status**: Detection and launch flag injection implemented and verified via unit tests; physical generation unverified in this session.
- **Tier 4 (Source-Compatible Desktop Adapters - Unverified Native Build)**:
- **Hardware**: Apple Silicon (MPS Metal).
- **Status**: Platform adapters implemented and unit-tested; native binary compilation and physical Mac runs unverified in this session.

---

Expand Down Expand Up @@ -138,6 +148,18 @@ cargo run --manifest-path launcher/Cargo.toml
3. **Task Cancellation**:
- Local engine cancellation dispatches interrupt signals (`/interrupt` on ComfyUI, `/sdapi/v1/interrupt` on SD WebUI).
- Cloud task cancellation immediately flags the task as cancelled locally, but truthfully discloses: `"Cloud cancellation requested locally. Note: external cloud providers may continue asynchronous inference or incur compute charges."`
4. **Video Generation & Container Formats**:
- Fal.ai (Fast SVD) and SiliconFlow (CogVideoX) output native MP4 (`video/mp4`, H.264).
- Standard ComfyUI (AnimateDiff/SVD) without custom nodes outputs animated WebP (`image/webp`) via `SaveAnimatedWEBP`. If `VHS_VideoCombine` with ffmpeg is installed, ComfyUI outputs native MP4.
- In the frontend canvas, MP4 and WebM containers render natively in `<video controls loop>`, while animated WebP renders via `<img>` with an "Animated WebP" badge.
- Downloads and exports strictly preserve the native container extension (`.mp4`, `.webm`, or `.webp`).
5. **Conversational Agent Boundaries**:
- The assistant operates via deterministic keyword and regular expression intent matching against predefined creative action schemas.
- It does not claim general unconstrained natural-language reasoning or autonomous arbitrary workflow synthesis.
- All execution requires explicit user review and approval via the proposal confirmation gate.
6. **Workflow Repair Ambiguity & Compatibility Guards**:
- The ComfyUI repair engine automatically reconnects slots only when an unambiguous single source exists in the DAG. If multiple sources exist, automatic connection is blocked (`substitution_blocked`) to avoid graph corruption.
- Checkpoint substitution is strictly constrained to matching architecture families (`sd15`, `sdxl`, `flux`, `svd`). Unknown or cross-family substitutions are blocked.

---

Expand Down
41 changes: 23 additions & 18 deletions docs/ROADMAP_POST_V0_1.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ This roadmap operationalizes the long-term approved technical directions for Ber
- AnimateDiff SD1.5 / SDXL macro subgraphs (`AnimateDiffLoaderWithContext`, `KSampler`, `VHS_VideoCombine` / `SaveAnimatedWEBP`).
- **Technical Boundaries**:
- Video port type conforms to the 5-type contract (`video`).
- Output stored in `AssetStore` with media type `video/mp4` or `video/webm`, content-addressed via SHA-256.
- Output stored in `AssetStore` with media type `video/mp4`, `video/webm`, or `image/webp` (for animated WebP from standard ComfyUI). Content-addressed via SHA-256.
- Video playback in GUI: native `<video>` for MP4/WebM containers; `<img>` with "Animated WebP" badge for animated WebP outputs.
- Export: downloads preserve actual container extension (`.mp4`, `.webm`, or `.webp`).
- Deterministic dirty-check caching accounts for video-specific parameters: `motion_bucket_id`, `fps`, `num_frames`, `duration`.
- **Data & Project Compatibility**:
- SQLite database persists video assets and execution records. Project schema v1 loads seamlessly; canvas renders video cards.
Expand Down Expand Up @@ -107,11 +109,12 @@ This roadmap operationalizes the long-term approved technical directions for Ber
- The UI presents the plan with an **"Approve & Run"** button and editable fields.
- The user can adjust parameters or approve execution; the Agent dispatches the action to `CreativeRunner` and presents results with next-step suggestions.
- **Supported Capability Matrix**:
- Natural-language mapping to all creative actions (txt2img, img2img, inpaint, upscale, txt2video).
- Intent classification, parameter extraction, and template matching.
- Deterministic keyword and regex mapping to creative actions (txt2img, img2img, inpaint, upscale, txt2video, img2video).
- Parameter extraction (aspect ratios, step count, seeds, engines) and template recommendation.
- **Technical Boundaries**:
- Backend `AgentService` with rule-based and LLM-assisted intent resolution.
- Backend `AgentService` with rule-based and regex intent resolution; does not claim general unconstrained natural-language understanding or arbitrary workflow synthesis.
- Strict human-in-the-loop gate: execution requires user approval token.
- Transparent plan disclosure with cost, engine, and parameter visibility.
- Never mutates engine files or executes uninspected external commands.
- **Data & Project Compatibility**:
- Conversation sessions and generated proposals are persisted in project SQLite tables.
Expand All @@ -120,35 +123,37 @@ This roadmap operationalizes the long-term approved technical directions for Ber
- UI displays proposal, parameters, and confirmation dialog.
- Execution only dispatches upon explicit user confirmation.
- **Explicit Exclusions**:
- Autonomous loop execution without user confirmation; silent billing against cloud accounts.
- Unbounded conversational reasoning without explicit action templates; autonomous loop execution without user confirmation; silent billing against cloud accounts.

---

### M10: Agent-Assisted Construction, Validation, Repair, and Execution of ComfyUI Workflows

- **User Journey**:
- Users can import complex ComfyUI workflows or ask the Agent to *"Add face detailing to my workflow"* or *"Diagnose why this workflow fails"*.
- Users can import complex ComfyUI workflows or ask the Agent to *"Diagnose why this workflow fails"*.
- The Agent analyzes the ComfyUI DAG:
- Validates node connections against port types (`MODEL`, `CLIP`, `VAE`, `LATENT`, `IMAGE`).
- Checks for missing custom node classes and missing checkpoint/LoRA models.
- Identifies topological cycles or disconnected subgraphs.
- If errors exist, the Agent proposes a concrete **Repair Diff** (e.g. *"Node 8 (VAEDecode) is missing VAE input; connect from Node 4 [slot 2]"*).
- If errors exist, the Agent proposes a concrete **Repair Diff** (e.g. *"Node 8 (VAEDecode) is missing VAE input; connect from unambiguous Node 4 [slot 2]"*).
- **Ambiguity Guards**: When multiple potential sources exist for an input slot (e.g., multiple VAE loaders or EmptyLatentImage nodes), automatic reconnection is blocked (`substitution_blocked`) rather than guessing, preventing corrupted dataflow.
- **Model Family Guards**: Missing checkpoints are substituted only within the exact same architecture family (`sd15`, `sdxl`, `flux`, `svd`); cross-family or unknown substitutions are strictly blocked.
- The user inspects the visual diff and accepts the repair.
- Once validated, the workflow can be executed directly or registered into the `TemplateRegistry`.
- **Supported Capability Matrix**:
- ComfyUI DAG syntax parser, topological sort validator, and port type-checker.
- Common pattern repairs: missing VAE decode, mismatched latent dimensions, missing LoRA bypass.
- Model dependency resolver: checks against indexed local models.
- ComfyUI DAG syntax parser, topological sort cycle detector, and port type-checker.
- Safe pattern repairs: unambiguous single-source slot reconnections, same-family model substitution.
- Model dependency resolver: checks against indexed local models with family inference.
- **Technical Boundaries**:
- Pure Python DAG analysis engine in `backend/app/core/workflow_validator.py` and `backend/app/core/workflow_repair.py`.
- Sandboxed validation without running untrusted Python node code.
- **Data & Project Compatibility**:
- Validated workflows can be saved as custom templates in `TemplateRegistry`.
- **Acceptance Criteria**:
- Automated tests verify detection of broken connections, missing models, and invalid node types.
- Repair engine fixes broken workflows and produces valid, executable graphs.
- Automated tests verify detection of broken connections, missing models, cycles, and type mismatches.
- Repair engine fixes broken workflows safely and blocks ambiguous or cross-family mutations.
- **Explicit Exclusions**:
- Arbitrary Python code execution inside custom nodes; automated downloading of untrusted git repositories.
- Arbitrary Python code execution inside custom nodes; automated downloading of untrusted git repositories; guessing ambiguous wiring.

---

Expand All @@ -158,12 +163,12 @@ This roadmap operationalizes the long-term approved technical directions for Ber
- Users with AMD Radeon or Intel Arc discrete GPUs run Berry AI Studio.
- The Environment Manager detects the GPU hardware, reports detected VRAM, and displays the appropriate acceleration backend (DirectML / ROCm / OneAPI IPEX).
- Engine supervisors automatically pass vendor-optimized launch flags to ComfyUI (e.g., `--directml`, `--use-split-cross-attention`, `--lowvram`).
- Unsupported or untested configurations display clear performance caveats and recommendation to use Cloud BYOK if local performance is inadequate.
- Candidate configurations are clearly labeled as "Candidate Architecture (Unverified on Physical Hardware in Current Session)" with guidance to use Cloud BYOK if local performance is inadequate.
- **Supported Capability Matrix**:
- *Reference Hardware*:
- *Candidate Reference Hardware*:
- AMD Radeon RX 7900 XTX / 7800 XT / 6700 XT (Windows DirectML, Linux ROCm 6.x).
- Intel Arc A770 / A750 (Windows DirectML, Linux IPEX/OneAPI).
- *Status Classification*: Explicitly distinguished as "Verified on Reference Hardware" vs. "Experimental / DirectML Emulated".
- *Status Classification*: Explicitly distinguished as "Verified on Physical Hardware" (NVIDIA CUDA) vs. "Candidate Architecture (Unverified on Physical Hardware in Current Session)" vs. "Experimental / DirectML Emulated".
- **Technical Boundaries**:
- Extend `backend/app/runtime/hardware.py` to detect AMD GPUs via `rocm-smi` (Linux) and WMI `Win32_VideoController` (Windows), and Intel Arc GPUs via WMI.
- Update `HardwareReadiness` schema with vendor classification (`nvidia`, `amd`, `intel`, `apple_silicon`, `cpu_only`).
Expand All @@ -173,9 +178,9 @@ This roadmap operationalizes the long-term approved technical directions for Ber
- **Acceptance Criteria**:
- Hardware detector correctly identifies AMD and Intel GPUs and reports VRAM.
- Engine supervisor injects `--directml` or vendor flags when non-NVIDIA GPU is selected.
- `docs/SUPPORT_MATRIX.md` documents verified vs. unverified combinations.
- `docs/SUPPORT_MATRIX.md` documents verified vs. candidate and unverified combinations without claiming physical verification for unverified hardware.
- **Explicit Exclusions**:
- Custom kernel compilation; driver installation scripts; obsolete legacy GPUs lacking FP16 support.
- Custom kernel compilation; driver installation scripts; claiming physical hardware verification without physical generation run logs.

---

Expand Down
23 changes: 17 additions & 6 deletions frontend/src/components/canvas/ImageCardNode.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,18 @@ export const ImageCardNode = memo(({ id, data, selected }: NodeProps) => {
const { nodes } = useCanvasStore();

const isVideo = cardData.mediaType === 'video' || Boolean(cardData.videoUrl);
const targetMediaUrl = cardData.videoUrl || cardData.imageUrl || '';
const isVideoContainer = targetMediaUrl.toLowerCase().match(/\.(mp4|webm|mov)(\?|#|$)/) !== null;

const handleExport = (e: React.MouseEvent) => {
e.stopPropagation();
const link = document.createElement('a');
link.href = cardData.videoUrl || cardData.imageUrl;
const ext = isVideo ? 'mp4' : 'png';
link.href = targetMediaUrl;
let ext = isVideo ? (isVideoContainer ? 'mp4' : 'webp') : 'png';
const match = targetMediaUrl.match(/\.([a-zA-Z0-9]+)(?:\?|#|$)/);
if (match && ['mp4', 'webm', 'mov', 'png', 'jpg', 'jpeg', 'webp', 'gif'].includes(match[1].toLowerCase())) {
ext = match[1].toLowerCase();
}
link.download = `${cardData.label || 'berry_asset'}.${ext}`;
document.body.appendChild(link);
link.click();
Expand Down Expand Up @@ -111,22 +117,27 @@ export const ImageCardNode = memo(({ id, data, selected }: NodeProps) => {

{/* Media Preview Container */}
<div className="relative bg-slate-950 flex items-center justify-center min-h-[220px] max-h-[360px] overflow-hidden">
{isVideo ? (
{isVideo && (isVideoContainer || !targetMediaUrl.toLowerCase().match(/\.(webp|gif)(\?|#|$)/)) ? (
<video
src={cardData.videoUrl || cardData.imageUrl}
src={targetMediaUrl}
controls
loop
playsInline
className="w-full h-auto object-contain max-h-[360px]"
/>
) : (
<img
src={cardData.imageUrl}
src={targetMediaUrl || cardData.imageUrl}
alt={cardData.label || 'Generated creative asset'}
className="w-full h-auto object-contain select-none pointer-events-none"
className="w-full h-auto object-contain select-none"
loading="lazy"
/>
)}
{isVideo && !isVideoContainer && (
<div className="absolute top-2 left-2 px-2 py-0.5 rounded-md bg-purple-950/80 backdrop-blur text-[10px] font-medium text-purple-300 border border-purple-800/60">
Animated WebP
</div>
)}
{p && (
<div className="absolute bottom-2 left-2 px-2 py-0.5 rounded-md bg-slate-950/75 backdrop-blur text-[10px] font-mono text-slate-300 border border-slate-800">
{p.dimensions}
Expand Down
Loading