Optional EvalPort interop for Examples and evaluation results

[adhabnr-ux]

Hi Arize team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM evaluation datasets and results (a TestCase/Suite/ResultSet schema with a real validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first before writing any code. (I know Phoenix already has an EvalPort adapter — this is specifically about client_python's own arize.experiments/arize.evaluators/arize.datasets modules for the Arize AX platform, which are a separate object model.)

I installed arize (8.48.0) and read the real classes rather than guessing:

# arize/experiments/types.py
@dataclass(frozen=True)
class Example:
    id: ExampleId = field(default_factory=str)
    updated_at: datetime = field(default_factory=datetime.now)
    input: Mapping[str, JSONSerializable] = field(default_factory=dict)
    output: Mapping[str, JSONSerializable] = field(default_factory=dict)
    metadata: Mapping[str, JSONSerializable] = field(default_factory=dict)
    dataset_row: Mapping[str, JSONSerializable] = field(default_factory=dict)

# arize/experiments/evaluators/types.py
class AnnotatorKind(Enum):
    CODE = "CODE"
    LLM = "LLM"

@dataclass(frozen=True)
class EvaluationResult:
    score: float | None = None
    label: str | None = None
    explanation: str | None = None
    metadata: Mapping[str, JSONSerializable] = field(default_factory=dict)

An Example (from arize.datasets/arize.experiments) maps directly onto an EvalPort TestCase — input/output/metadata line up almost one-to-one — and an EvaluationResult (from arize.experiments.evaluators, produced by a CODE- or LLM-kind evaluator) maps onto an EvalPort grader result: score/label cover both the numeric and pass/fail cases EvalPort's spec supports, and explanation carries straight through as the grader's rationale.

Two ways I could see this landing, and I don't have a strong preference — happy to do whichever you'd rather maintain:

1. A standalone arize-openeval-adapter package in the EvalPort repo (same shape as the existing adapters there — a to_openeval()/from_openeval() pair depending on arize as a normal package dependency). Zero footprint on this repo beyond maybe a link from your docs if you're open to it.
2. A small optional module inside this repo if you'd rather it live here, gated so it doesn't affect anyone not using it.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.