We just started using aimock in a Playwright E2E test and it works really well.
In a multi-turn conversation, we encountered a scenario where we need a side-effect from a previous tool call in order to generate the fixture response of the next tool call.
In the application feature we're testing the same side-effect is awaited and right after that the next agent run is triggered.
Currently the fixtures don't seem to allow for an async flow so that this could result in a race condition.
For now everything works smoothly, but I want to prevent any test flakiness as early as possible.
Proposal
Maybe I'm overlooking something and the race condition cannot occur, but I think the following proposal would add a lot of flexibility for complex test scenarios.
Before
The promise is awaited, but if the conversation were to take another turn before the fixture is registered, this would result in an error because no fixtures matches.
const createdEntity = await createEntityPromise;
llmock.on(
{toolCallId: "toolu_bdrk_01LBX7TNNo3ipEV61uVXmvE5"},
{
content: `Perfect! 🎉 The entity "${entityTitle}" was successfully created. Now we're creating a sub-entity:`,
toolCalls: [
createAssistantToolCall({
name: "createSubEntity",
id: "toolu_bdrk_01BHXUVJqE99iEaED6BqJthD",
args: {
parentId: createdEntity.id, // this seems risky
elements: [{title: firstSubEntityTitle}],
},
}),
],
},
);After
If the fixture matches, the callback function is invoked so that async operations can gracefully finish before the response is returned.
llmock.on(
{toolCallId: "toolu_bdrk_01LBX7TNNo3ipEV61uVXmvE5"},
async () => {
const createdEntity = await createEntityPromise;
return {
content: `Perfect! 🎉 The entity "${entityTitle}" was successfully created. Now we're creating a sub-entity:`,
toolCalls: [
createAssistantToolCall({
name: "createSubEntity",
id: "toolu_bdrk_01BHXUVJqE99iEaED6BqJthD",
args: {
parentId: createdEntity.id, // this seems fine
elements: [{title: firstSubEntityTitle}],
},
}),
],
}
},
);
We just started using
aimockin a Playwright E2E test and it works really well.In a multi-turn conversation, we encountered a scenario where we need a side-effect from a previous tool call in order to generate the fixture response of the next tool call.
In the application feature we're testing the same side-effect is awaited and right after that the next agent run is triggered.
Currently the fixtures don't seem to allow for an async flow so that this could result in a race condition.
For now everything works smoothly, but I want to prevent any test flakiness as early as possible.
Proposal
Maybe I'm overlooking something and the race condition cannot occur, but I think the following proposal would add a lot of flexibility for complex test scenarios.
Before
The promise is awaited, but if the conversation were to take another turn before the fixture is registered, this would result in an error because no fixtures matches.
After
If the fixture matches, the callback function is invoked so that async operations can gracefully finish before the response is returned.