Can you add the type hint maybe?

This doesn't look correct to me – self._task is a Callable, not an Awaitable (Task / Coroutine / something else) meaning it is not correct to await it (and that's why you had to add this type: ignore I'm guessing?)

I was wondering how somehow this wasn't showing in tests's type checking but the fact they receive llmobs as an untyped fixture hides that problem. Trying to run the same code with the "real" LLMObs thing, you get an error if you try to pass a coroutine.

I think this absolutely needs to be changed.

yes I very much agree, I just realized this last night. The type hint for task does not fit the async case, and we have a directive for mypy to ignore that line in our code, but I think this should be handled differently. I was trying to get mypy to show errors in a test script I have where I try using local editable install of ddtrace -- it seems mypy is actually surprising permissive on allowing async function with similar signature where the type is Callable, but I'd rather have the correct type

I was thinking of either having separate async_task argument to Experiment init, so the idea is the user will either only one of task and async_task.

Another thought is that task arg could accept either sync or async task, use inspect.iscoroutinefunction() to determine if it's async.

In either case, the user gets a friendly error if they call run() / arun() methods when it doesn't match the task type.

Do you know what other APIs (e.g. Braintrust) do? Maybe there's a good pattern to re-use there. To be honest, I don't have a strong preference and it's an issue that I've encountered in the past (without a clear-cut solution).

inspect.iscoroutinefunction()

I remember I think at some point seeing someone use that and although it works, it gets somewhat hard to properly type the code when relying on this pattern.

having separate async_task argument

I think this is OK but again, typing this properly is somewhat complicated; I guess you'd have to use @override and make sure this stays maintained over time because it's easy to get wrong (also, relying on @override for typing makes everything more complicated because you have to maintain separate function declarations and definitions...)

I'd recommend a type hint here

Any reason not to use an f-string here?

Are those really Tasks? I think they're just coroutines.

good call, Claude used the name task but I would usually in the past have used the name coros. I think I've seen that in a lot of examples of async code in the past.