[Teams Chatbot] Enable image input through Teams connector downloads by JialinHuang803 · Pull Request #17428 · Azure/azure-sdk-tools

Jialin Huang (JialinHuang803) · 2026-10-29T02:20:20Z

Summary

Enable the Teams chatbot to use inline screenshots when answering channel messages.
Addresses Azure/azure-sdk-pr#2316.

The Logic App downloads images through its existing Teams connection; the backend receives image bytes and does not need Graph download permissions.
Image recognition happens in the hosted agent's vision-capable model.

Data flow: Teams HTML → image attachment URLs → connector downloads → base64 data URIs → frontend → backend → model input_image.

Changes by component

1. Backend: activity converter

services/activity_converter_service.py

Extract <img src> references from message HTML and add deduplicated image/* attachments to the Bot Framework activity.
Resolve relative URLs and verify that Graph hosted-content images belong to the source message, including thread replies.
Preserve the original message text and HTML attachment; do not download images at this stage.

2. Logic App: image downloads

pipelines/logicapp/template.json

Keep the existing text-based intention check before activity conversion and image downloading.
Send messages without image attachments directly to the bot.
For image-bearing messages, select the first four image attachments, then download their Graph hosted-content URLs through the Teams connector.
Use sequential loops and reset the download array per message to prevent batched messages from mixing image results.
Replace successful, size-bounded downloads with base64 data URIs; keep unresolved URLs when a download fails or exceeds the size guard.
Hide image-bearing payloads in action history and send the bot a complete JSON POST, without Logic Apps' chunked-upload protocol.

3. Frontend: Teams bot

src/index.ts, src/input/, src/models/RAGModel.ts, src/backend/rag.ts

Raise the JSON body limit to 30 MiB to accommodate four base64-encoded images plus activity metadata.
Forward connector-supplied data URIs through the existing prompt.images → additional_infos[].link contract.
Omit image bytes from activity/prompt/payload logs and avoid logging Axios request bodies on failures.
Exclude inline data URIs from Azure Table prompt storage without changing current-turn forwarding; retain remote image URLs.

4. Backend: chat service and image validation

services/chat_service.py, utils/teams_image.py

Accept connector-supplied data URIs without authentication or another download.
Enforce four image attempts per request and 5 MiB decoded per image, with strict base64, supported MIME-type, and file-signature checks.
Send valid images as Responses API input_image items, with guidance to treat instructions inside images as untrusted content.
Skip unavailable or excess images and add a notice instructing the model to disclose missing image context rather than guess.
Preserve the existing native Bot Framework download path, with bounded downloads and no redirects; do not download Graph or SharePoint URLs in the backend.

5. Teams manifest and environment tags

Update the manifest to schema 1.25 and declare supportsChannelFeatures: "tier1".
Update frontend image tags to dev-0.0.32, preprod-0.0.15, and prod-0.0.14.

Scope and limitations

Supported image types: PNG, JPEG, GIF, and WebP.
Uploaded SharePoint file attachments and forwarded images belonging to another message are not supported.
The four-image limit selects attachments in order; a failed download does not cause the fifth attachment to be substituted.
Inline image bytes are not retained in frontend Table Storage history or replayed from that stored prompt after recovery.
Latency: An image download takes approximately 3 seconds in observed testing; downloads are sequential, so posts containing images incur additional latency before bot processing begins, and the download time accumulates with each image. Actual timing varies.

Validation

Backend: 165 targeted tests passed across converter, image utility, and chat-service suites.
Frontend: 21 tests passed across the changed transport, prompt, and conversation suites; build succeeded, including after merging latest main.
Earlier dev testing confirmed a single-image connector-to-bot flow and successful replies.
The latest revisions, multiple-image messages, and partial download failures have not been reverified end to end in a live Teams flow.

Testing: Teams Post Teams Post
Logic app run history

Preserve Teams image attachments, download Graph hosted content through the Logic App Teams connection, and forward validated inline image bytes to Foundry. Handle payload size, logging, and conversation storage constraints. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/chat_service_test.py

Copilot

🟡 Changes recommended

HTTPX INFO logging exposes full preauthenticated SharePoint/OneDrive download URLs and their access-granting query parameters in backend logs despite the exception-logging safeguard.

1 open issue

Prevent HTTPX from logging preauthenticated download URLs · New

What changed in this PR

Adds image context to the Teams QA bot’s answer-generation flow, using the existing Teams connector to download Graph-hosted images and forward them to Foundry.

Changes:

Preserves inline images and uploaded image references, with authenticated backend downloads as a fallback.
Adds image format validation, four-image and 5 MiB limits, and notices for unavailable image context.
Increases the bot’s request-body limit, reduces image payload logging, and excludes inline bytes from conversation storage.

File	Description
`tools/sdk-ai-bots/azure-sdk-qa-bot/test/ragImages.test.ts`	Tests payload forwarding and log redaction.
`tools/sdk-ai-bots/azure-sdk-qa-bot/test/promptGenerator.test.ts`	Tests connector-image prompt handling.
`tools/sdk-ai-bots/azure-sdk-qa-bot/test/messageBodyParser.test.ts`	Tests request-body size limits.
`tools/sdk-ai-bots/azure-sdk-qa-bot/test/logicAppImages.test.ts`	Tests connector download configuration and image preparation.
`tools/sdk-ai-bots/azure-sdk-qa-bot/test/conversation.test.ts`	Tests inline-byte exclusion from storage and updates constructor mocks.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/models/RAGModel.ts`	Replaces image-bearing logs with summaries.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/input/PromptGenerator.ts`	Omits image URLs from raw-prompt logs.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/input/messageBodyParser.ts`	Defines the 30 MiB JSON parser.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/input/ConversationHandler.ts`	Excludes inline image bytes from stored prompts.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/index.ts`	Enables the larger request-body parser.
`tools/sdk-ai-bots/azure-sdk-qa-bot/src/backend/rag.ts`	Redacts inline payloads and limits Axios error logging.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/utils/teams_image.py`	Validates image data and handles bounded authenticated downloads.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/teams_image_test.py`	Covers authentication, validation, redirects, and image forwarding.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/chat_service_test.py`	Tests model image content, limits, and failure notices.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/activity_converter_service_test.py`	Tests image extraction, source validation, and deduplication.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/services/chat_service.py`	Enforces image-count limits and reports unavailable context.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/services/activity_converter_service.py`	Preserves supported inline and file-image attachments.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/README.md`	Documents permissions, deployment, testing, and limitations.
`tools/sdk-ai-bots/azure-sdk-qa-bot-agent/pipelines/logicapp/template.json`	Downloads connector images and forwards data URIs with protected action history.

_{Review effort: Balanced}

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replace foreach result aggregation with four guarded download slots, preserving secure image payloads through direct capture outputs. Continue after failed or unused slots and cover the secured-output topology with regression tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep image preparation in one scope, start all four guarded downloads after filtering, and join direct secured capture outputs. Gate bot invocation on successful activity preparation so failed downloads do not block a ready reply. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Invoke the bot directly after successful image preparation while retaining partial download fallback and secured HTTP inputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Check converted attachments before entering image processing, and send non-image activities directly to the bot with secured request inputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 29, 2026 02:20

Jialin Huang (JialinHuang803) requested review from Jiaqi Zhang (JiaqiZhang-Dev) and Crystal YU (chunyu3) as code owners October 29, 2026 02:20

Copilot started reviewing on behalf of Jialin Huang (JialinHuang803) October 29, 2026 02:20

View session

Merge latest main into Teams image recognition branch

9364068

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # tools/sdk-ai-bots/azure-sdk-qa-bot-agent/tests/chat_service_test.py

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread tools/sdk-ai-bots/azure-sdk-qa-bot-agent/utils/teams_image.py Resolved

Jialin Huang (JialinHuang803) and others added 14 commits October 29, 2026 11:08

Merge latest main into Teams image recognition branch

f2a71dc

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Fix Teams activity body forwarding and expose workflow failures

13163be

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Disable unsupported Logic Apps chunked uploads to Teams bot

1ebc687

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Simplify Teams image downloads with a protected sequential array

58992be

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Remove redundant prepared activity validation

27c5270

Invoke the bot directly after successful image preparation while retaining partial download fallback and secured HTTP inputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Bypass image resolution for Teams messages without images

2ea82b0

Check converted attachments before entering image processing, and send non-image activities directly to the bot with secured request inputs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Clarify stored image filtering and remove per-message image log

66a61b7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Inline Teams message JSON parser configuration

fd2ef49

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Consolidate Teams bot image tests and update channel manifest

0685070

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Simplify Teams image handling after backend review

8c2eabd

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Update Teams bot image tags for all environments

2b5b084

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge latest main into Teams image recognition

0c06e6b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Jiaqi Zhang (JiaqiZhang-Dev) reviewed Nov 2, 2026

View reviewed changes

Comment thread tools/sdk-ai-bots/azure-sdk-qa-bot-agent/pipelines/logicapp/template.json Resolved

Comment thread tools/sdk-ai-bots/azure-sdk-qa-bot-agent/README.md Resolved

Renhe Li (lirenhe) reviewed Nov 3, 2026

View reviewed changes

Comment thread tools/sdk-ai-bots/azure-sdk-qa-bot-agent/services/activity_converter_service.py Resolved

Renhe Li (lirenhe) reviewed Nov 3, 2026

View reviewed changes

Comment thread tools/sdk-ai-bots/azure-sdk-qa-bot/src/index.ts Resolved

Jialin Huang (JialinHuang803) and others added 3 commits November 5, 2026 11:09

Remove Teams image README additions

63abb34

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Handle image preparation failures and clarify image limits

d6b9af2

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Separate bot invocation from image resolution scope

536ea14

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Jiaqi Zhang (JiaqiZhang-Dev) approved these changes Nov 5, 2026

View reviewed changes

Navigation Menu

Repository navigation

[Teams Chatbot] Enable image input through Teams connector downloads - #17428

[Teams Chatbot] Enable image input through Teams connector downloads#17428
Jialin Huang (JialinHuang803) wants to merge 19 commits into
mainfrom
jialinhuang/teams-image-recognition

Jialin Huang (JialinHuang803) commented Oct 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Jialin Huang (JialinHuang803) commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes by component

1. Backend: activity converter

2. Logic App: image downloads

3. Frontend: Teams bot

4. Backend: chat service and image validation

5. Teams manifest and environment tags

Scope and limitations

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jialin Huang (JialinHuang803) commented Oct 29, 2026 •

edited

Loading