fix: custom model objects with id field skipped in resolveModelCollections

[frantj]

Fixes #22

When defining a custom model in a blueprint using object syntax with multiple fields (e.g. id, url, modelName, inherit), the model is silently skipped during resolution and a warning is logged instead.

Note: No Blueprint ID applicable — this is a code-level bug in resolveModelCollections, reproducible with any blueprint defining a custom model object with an id field.

To reproduce

Define a custom model in a blueprint:

models:
  - id: my-custom-agent
    url: http://localhost:3001/v1
    modelName: gpt-4o

Run an evaluation. The model is not included — instead you see:

Invalid object entry in models array: Must have exactly one key (the provider). Found 3 keys. Skipping: {"id":"my-custom-agent","url":"...","modelName":"gpt-4o"}

Expected behavior

Custom model objects with an id field should be passed through directly and included in the evaluation.

Root cause

In src/cli/commands/run-config.ts, resolveModelCollections checks for a generic single-key provider object before checking for a custom model object with an id property. Since a custom model has multiple keys, it fails the single-key check and is skipped.

The fix is to check for modelEntry.id first:

} else if (typeof modelEntry === 'object' && modelEntry !== null && !Array.isArray(modelEntry) && modelEntry.id) {
    // Custom model definition — pass through directly
    normalizedConfigModels.push(modelEntry);
} else if (typeof modelEntry === 'object' && modelEntry !== null && !Array.isArray(modelEntry)) {
    // Provider shorthand — expect exactly one key
    ...

Additional context

No error is thrown — the silent skip makes this hard to diagnose without reading the warning carefully. Fix is available in frantj/wevalapp@339c52a.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.