Make skills aggregator-compatible: co-located evals + self-contain complexa-sweep

.claude/skills/complexa-design/evals/evals.json

@@ -0,0 +1,60 @@
+{
+  "skill_name": "complexa-design",
+  "evals": [
+    {
+      "id": "complexa-design-001",
+      "prompt": "Run complexa design for a PDL1 binder using beam-search and refold with AF2. I want at least 50 designs generated.",
+      "expected_output": "The agent executed the complexa-design pipeline for a PDL1 protein binder target, running preflight checks, selecting the appropriate search_binder_local_pipeline.yaml config, launching the design with beam-search and AF2 refold, and reporting how many designs passed filtering.",
+      "assertions": [
+        "The agent ran the preflight script (bash .claude/skills/_shared/scripts/preflight.sh) and checked preflight.json for GPU availability and AF2_DIR",
+        "The agent selected configs/search_binder_local_pipeline.yaml as the pipeline config for protein binder design",
+        "The agent constructed and executed a complexa design command with beam-search parameters and the PDL1 target task_name",
+        "The agent reported the number of designs that passed and referenced success rate or per-design CSV output",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-design",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-design-002",
+      "prompt": "I have a small molecule target (ATP) and I want to design proteins that bind it tightly. Can you set up and run a de novo design campaign for this?",
+      "expected_output": "The agent recognized this as a ligand binder design task, ran preflight validation including RF3 checkpoint checks, selected the ligand binder pipeline config, executed the complexa design run for an ATP-binding protein, and reported results including pass rates.",
+      "assertions": [
+        "The agent ran the preflight script and verified RF3_CKPT_PATH and RF3_EXEC_PATH are present in the environment for ligand binder generation",
+        "The agent selected the ligand binder pipeline configuration (not the default protein binder config)",
+        "The agent executed the complexa design command with appropriate parameters for small-molecule/ligand binding",
+        "The agent summarized the design campaign results including success rate or number of passing designs",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-design",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-design-003",
+      "prompt": "We're working on an enzyme engineering project. We have a catalytic motif (AME) that needs to be scaffolded near a ligand binding site. The ligand is a substrate analog. Can you run the full design-to-evaluation pipeline and tell me the diversity of the resulting designs?",
+      "expected_output": "The agent executed the AME motif scaffolding pipeline with ligand context, validated that the complexa_ame checkpoint and RF3 dependencies were available, ran the full four-stage pipeline (generate, filter, evaluate, analyze), and reported FoldSeek/MMseqs diversity metrics alongside success rates.",
+      "assertions": [
+        "The agent ran preflight checks and confirmed availability of ckpts.complexa_ame and RF3 environment variables",
+        "The agent selected the AME/enzyme scaffolding pipeline configuration appropriate for motif + ligand design",
+        "The agent executed the complexa design command and waited for all four stages (generate, filter, evaluate, analyze) to complete",
+        "The agent reported diversity metrics (FoldSeek or MMseqs) and success rates from the analysis stage",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-design",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-design-004",
+      "prompt": "Can you help me set up a PostgreSQL database with replication across three nodes for our production environment?",
+      "expected_output": "The agent recognized this as a database administration task unrelated to protein design and did not invoke the complexa-design skill, instead providing guidance on PostgreSQL replication setup or directing the user to appropriate resources.",
+      "assertions": [
+        "The agent did not run the complexa preflight script or any complexa design commands",
+        "The agent did not reference protein binder design, flow matching, or refold pipelines",
+        "The agent addressed the PostgreSQL replication question directly or asked clarifying questions about the database setup",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": null,
+      "expected_script": null
+    }
+  ]
+}

.claude/skills/complexa-evaluate-pdbs/evals/evals.json

@@ -0,0 +1,59 @@
+{
+  "skill_name": "complexa-evaluate-pdbs",
+  "evals": [
+    {
+      "id": "complexa-evaluate-pdbs-001",
+      "prompt": "I want to use the complexa-evaluate-pdbs skill to score my binder designs in /data/designs/pdl1_binders/ using AF2 refolding. Can you run that for me?",
+      "expected_output": "The agent ran the complexa-evaluate-pdbs workflow by executing preflight checks, selecting the evaluate_from_pdb_dir.yaml config with colabdesign as the folding backend, ran complexa analysis on the specified PDB directory, and reported pass-rate metrics including i_pAE, pLDDT, and scRMSD from the resulting CSV.",
+      "assertions": [
+        "The agent executed the preflight.sh script to verify GPU availability and required environment variables",
+        "The agent ran complexa analysis with configs/evaluate_from_pdb_dir.yaml and ++sample_storage_path=/data/designs/pdl1_binders/ with colabdesign as the binder_folding_method",
+        "The agent parsed the output CSV and reported pass-rate summaries against protein_binder thresholds",
+        "The agent generated or referenced an eval_manifest.json for reproducibility",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-evaluate-pdbs",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-evaluate-pdbs-002",
+      "prompt": "I have a folder of BindCraft outputs at ~/bindcraft_results/her2/ and I need to compute interface pAE and i_pTM for all of them. Can you also check designability with ProteinMPNN?",
+      "expected_output": "The agent identified this as a PDB evaluation task, ran preflight checks, configured complexa analysis with the evaluate_from_pdb_dir.yaml config pointing to the BindCraft output directory, included designability metrics via inverse_folding_model=soluble_mpnn, and reported per-PDB interface pAE, i_pTM, and designability scRMSD results.",
+      "assertions": [
+        "The agent ran preflight.sh to check GPU, disk, and tool availability before launching the evaluation",
+        "The agent selected the protein_binder result_type and evaluate_from_pdb_dir.yaml config for third-party BindCraft outputs",
+        "The agent included ++metric.inverse_folding_model=soluble_mpnn to enable ProteinMPNN designability scoring",
+        "The agent reported interface pAE, i_pTM, and designability scRMSD metrics from the results CSV",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-evaluate-pdbs",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-evaluate-pdbs-003",
+      "prompt": "We just received 50 hand-curated binder decoys from a collaborator targeting EGFR. They're in /shared/collab_decoys/egfr_v2/. I need to validate them before our meeting tomorrow — can you refold with RoseTTAFold3 and give me pass rates? We need at least 20% of designs passing on scRMSD and i_pAE.",
+      "expected_output": "The agent performed a full complexa evaluate-pdbs workflow using RF3 as the folding backend on the collaborator's EGFR decoys, verified RF3 environment variables were set, ran complexa analysis, and reported whether the 20% pass-rate threshold was met for scRMSD and i_pAE metrics.",
+      "assertions": [
+        "The agent ran preflight.sh and verified RF3_CKPT_PATH and RF3_EXEC_PATH were available in the environment",
+        "The agent ran complexa analysis with ++metric.binder_folding_method=rf3_latest and ++sample_storage_path=/shared/collab_decoys/egfr_v2/",
+        "The agent parsed the result CSV and explicitly reported pass-rates for scRMSD and i_pAE against the protein_binder thresholds",
+        "The agent compared the observed pass-rates to the user's stated 20% requirement and communicated whether the designs met that bar",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-evaluate-pdbs",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-evaluate-pdbs-004",
+      "prompt": "How do I install RoseTTAFold3 and set up the checkpoint paths on my new workstation?",
+      "expected_output": "The agent recognized this as an installation/setup question rather than an evaluation task, and either provided general guidance on RF3 installation or directed the user to the complexa-setup skill instead of running complexa-evaluate-pdbs.",
+      "assertions": [
+        "The agent did not run complexa analysis or any evaluation workflow",
+        "The agent provided installation guidance or directed the user to the complexa-setup skill for environment configuration",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": null,
+      "expected_script": null
+    }
+  ]
+}

.claude/skills/complexa-setup/evals/evals.json

@@ -0,0 +1,60 @@
+{
+  "skill_name": "complexa-setup",
+  "evals": [
+    {
+      "id": "complexa-setup-001",
+      "prompt": "Set up complexa on this machine. I just cloned the repo and need everything configured from scratch.",
+      "expected_output": "The agent executed the complexa-setup skill end-to-end, running preflight checks, initializing the .env via `complexa init`, configuring user-specific paths, downloading model weights, and validating the environment, producing a replayable setup artifact.",
+      "assertions": [
+        "The agent ran `bash .claude/skills/_shared/scripts/preflight.sh` and read the resulting preflight.json",
+        "The agent executed `complexa init` to create the .env file with the appropriate runtime selection",
+        "The agent asked the user for environment-specific values (e.g., LOCAL_CODE_PATH, DATA_PATH) and edited the .env file accordingly",
+        "The agent ran `complexa validate env` to confirm the environment was correctly configured",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-setup",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-setup-002",
+      "prompt": "I need to download the AF2 and ProteinMPNN model weights for my protein design pipeline. I already have a .env but I'm not sure if my checkpoints are complete.",
+      "expected_output": "The agent identified this as a model weight download task within the complexa-setup skill, checked the current download status, and ran the appropriate `complexa download` commands to fetch missing AF2 and ProteinMPNN checkpoints.",
+      "assertions": [
+        "The agent ran preflight checks to assess disk space and existing checkpoint status",
+        "The agent executed `complexa download --status` or equivalent to determine which model weights were already present",
+        "The agent ran `complexa download` with appropriate flags to fetch the missing AF2 and ProteinMPNN checkpoints",
+        "The agent confirmed successful download by validating the environment or checking file existence",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-setup",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-setup-003",
+      "prompt": "I just spun up a new A100 instance on AWS and git cloned proteina-complexa. When I try to run a design job it says '.env not found'. How do I get this thing working? I want to use Docker as my runtime.",
+      "expected_output": "The agent recognized the fresh-instance scenario, performed the full first-time setup workflow selecting Docker as the runtime, created and configured the .env, downloaded necessary model weights, and validated the environment so the user's design jobs could proceed.",
+      "assertions": [
+        "The agent ran the preflight script to check GPU availability, VRAM, and disk space on the new instance",
+        "The agent executed `complexa init docker` (or equivalent Docker runtime selection) to generate the .env file",
+        "The agent edited the .env file with user-specific paths using file write operations",
+        "The agent ran `complexa download` to fetch required model checkpoints and then validated the environment",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": "complexa-setup",
+      "expected_script": null
+    },
+    {
+      "id": "complexa-setup-004",
+      "prompt": "Can you help me analyze the binding affinity results from my last Complexa design run? The output CSV is in ./results/run_042/scores.csv and I want to know which candidates had the best dG values.",
+      "expected_output": "The agent recognized this as a results analysis task unrelated to environment setup, and assisted with reading and interpreting the design run output CSV rather than invoking the complexa-setup skill.",
+      "assertions": [
+        "The agent read the scores.csv file from the specified path to examine binding affinity results",
+        "The agent identified and ranked candidates by dG values without running any setup or initialization commands",
+        "The agent did not execute `complexa init`, `complexa download`, or `complexa validate env`",
+        "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+      ],
+      "expected_skill": null,
+      "expected_script": null
+    }
+  ]
+}

.claude/skills/complexa-sweep/SKILL.md

@@ -215,4 +215,4 @@ Refer to [`.claude/skills/_shared/reference/hardware.md`](../_shared/reference/h
 
 For per-axis reference (typical ranges, cost, what gets better/worse), see [reference/sweep_axes.md](reference/sweep_axes.md).
 
-For the user-facing sweep system overview (config generation, output layout), see [`docs/SWEEP.md`](../../../docs/SWEEP.md).
+For the user-facing sweep system overview (config generation, output layout), see [`docs/SWEEP.md`](reference/SWEEP.md).