EvalPort export adapter for benchmark results / TestCase suites?

[adhabnr-ux]

Idea: EvalPort export adapter for MASArena's result/results files

I've been building EvalPort (TestCase/Grader/Result/ResultSet/GraderResult — an interchange format + Python/TS SDKs so eval data isn't locked to one framework), and MASArena's output shape maps onto it unusually cleanly, so I wanted to float a small adapter rather than just open a PR unannounced.

What I looked at: mas_arena/evaluators/registry.py's normalization_keys (id/problem/solution), BaseEvaluator/MathEvaluator.evaluate(), and BenchmarkRunner._finalize_benchmark() in mas_arena/benchmark_runner.py, which already writes {"summary": summary, "results": all_results} to results/{benchmark}_{agent_system}_{timestamp}.json — that's basically a ResultSet in a different vocabulary:

MASArena (benchmark_runner.py)	EvalPort (evalport-sdk / openeval.types)
a problems row, keyed via normalization_keys (id, problem, solution)	TestCase(id, input, expected_output, graders=[...])
result_entry from _process_one_problem: problem_id, prediction, score, is_correct, duration_ms, llm_usage	Result(test_case_id, passed, actual_output, duration_ms, grader_results=[GraderResult(...)])
summary dict (benchmark, agent_system, accuracy, total_problems, timestamp, ...)	ResultSet(suite_id, run_id, started_at, results=[...], summary={...})

A converter would sit next to the evaluators as a thin, optional package (no core-file changes needed) — same shape as the existing autogen-openeval-adapter: plain dict/attr access via _get(), zero hard dependency on MASArena internals beyond the JSON shape you already write.

# masarena_openeval_adapter/__init__.py  (sketch)
from openeval.types import TestCase, Grader, Result, GraderResult, ResultSet, OPENEVAL_VERSION

def suite_from_problems(problems: list[dict], benchmark_config: dict, suite_id: str):
    """problems: rows loaded from data/{benchmark}_test.jsonl.
    benchmark_config: BENCHMARKS[name] from mas_arena.evaluators (has normalization_keys)."""
    keys = benchmark_config["normalization_keys"]
    return [
        TestCase(
            id=str(p[keys["id"]]),
            input=p[keys["problem"]],
            expected_output=str(p[keys["solution"]]),
            graders=[Grader(id="gr_score", type="custom",
                             params={"handler": f"mas_arena:{suite_id}"})],
        )
        for p in problems
    ]

def resultset_from_output_json(output_json: dict) -> ResultSet:
    """output_json: the dict MASArena already writes to
    results/{benchmark}_{agent_system}_{timestamp}.json."""
    s = output_json["summary"]
    results = [
        Result(
            test_case_id=str(r["problem_id"]),
            passed=r.get("is_correct", r.get("score", 0) == 1),
            actual_output=r.get("prediction", ""),
            duration_ms=r.get("duration_ms"),
            grader_results=[GraderResult(
                grader_id="gr_score", type="custom",
                score=r.get("score"), passed=r.get("is_correct", False),
            )],
            metadata={"agent_system": r.get("agent_system"), "llm_usage": r.get("llm_usage", {})},
        )
        for r in output_json["results"]
    ]
    return ResultSet(
        version=OPENEVAL_VERSION,
        suite_id=s["benchmark"], run_id=s["timestamp"],
        started_at=s["timestamp"], results=results,
        provider={"agent_system": s["agent_system"]},
        summary={"accuracy": s["accuracy"], "total": s["total_problems"], "errored": s["errored"]},
    )

type="custom" on the grader is deliberate, not laziness — several evaluators here (math_evaluator.py's symbolic/SymPy equivalence check, humaneval/mbpp/swebench's code execution) do real work that EvalPort's built-in grader types (exact_match, regex, etc.) don't faithfully represent, so custom + a params.handler back-reference seemed more honest than mislabeling them.

Would something like this be useful as a standalone adapter package (own repo, pip install-able, MASArena stays a soft optional dependency), or would you rather it live under mas_arena/ directly if there's interest? Happy to put together a working version either way — just didn't want to show up with an unsolicited PR before checking it's wanted.

— Sahi, independent contributor (not affiliated with this project)