Skip to content

Commit a05cd33

Browse files
authored
fix(tool-loader): enable image output from declarative tools via data URI (#62)
* fix(tool-loader): support image output from declarative tools via data URI Declarative tools that return image data were silently falling through to text handling, causing models to receive raw base64 as plain text instead of an image content block. Detect data URIs (data:image/<type>;base64,...) in tool output and return a proper ImageContent block. Works across all providers via pi-ai's shared transform-messages layer; non-vision models receive a graceful placeholder automatically. * fix(tool-loader): address image URI review comments - Scan lines for data URI instead of requiring stdout to start with it, so stray log output from scripts doesn't silently break detection - Add mime type allowlist (jpeg, png, gif, webp) and empty data guard; unsupported types fall through to text handling instead of erroring silently - Add comment explaining the magic number 5 (skipping "data:" prefix)
1 parent e4f3af9 commit a05cd33

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

‎src/tool-loader.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,28 @@ function createDeclarativeTool(
118118

119119
// Try parsing JSON output
120120
let text = stdout.trim();
121+
122+
// Detect data URI — works regardless of field naming in user scripts.
123+
// Scan lines so stray log output before the URI doesn't break detection.
124+
const dataUriLine = text.split("\n").find(
125+
line => line.startsWith("data:image/") && line.includes(";base64,"),
126+
);
127+
if (dataUriLine) {
128+
const commaIndex = dataUriLine.indexOf(",");
129+
// Slice past the leading "data:" prefix (5 chars) to get "image/jpeg;base64"
130+
const mimeType = dataUriLine.slice(5, commaIndex).split(";")[0];
131+
const data = dataUriLine.slice(commaIndex + 1).replace(/\s/g, "");
132+
const allowedMimeTypes = ["image/jpeg", "image/png", "image/gif", "image/webp"];
133+
if (data && allowedMimeTypes.includes(mimeType)) {
134+
resolve({
135+
content: [{ type: "image", data, mimeType }],
136+
details: undefined,
137+
});
138+
return;
139+
}
140+
// Fall through to text/JSON handling if data is empty or mime type unsupported
141+
}
142+
121143
try {
122144
const parsed = JSON.parse(text);
123145
if (parsed.text) text = parsed.text;

0 commit comments

Comments
 (0)