Test Coverage & Performance Improvements

[asdek]

Description of the Change

Problem

PentAGI codebase had several quality and performance issues requiring attention:

Test Coverage Gaps:

• Multiple core packages lacked unit tests (config, terminal, server/response, server/context, graph/context, embeddings)
• Existing community-contributed tests had inconsistent style and incomplete coverage
• Tests didn't follow project conventions (missing edge cases, no t.Parallel(), hardcoded assertions)

HTTP Client Timeout Missing:

• No timeout configured for external API calls, causing indefinite hangs when APIs stopped responding
• Affected all 17 LLM provider integrations and search tools
• Goroutine leaks from hung connections

Tool Call ID Detection Inefficiency:

• Full detection algorithm ran on every PentAGI restart (6-10 LLM calls per provider)
• Wasted tokens and time validating known templates already verified by developers
• Slow startup experience for users

AWS Bedrock Compatibility:

• Converse API returned ValidationException when conversation history contained tool blocks but current turn provided no tools
• Required toolConfig field missing in certain scenarios

Solution

Comprehensive Test Coverage:

• Added 200+ unit tests across 6 core packages with consistent table-driven approach
• Standardized test style: t.Parallel() for independent tests, require.* for critical assertions, comprehensive edge cases
• Brought community tests to project standards while maintaining their value
• Packages now have hermetic, production-ready test suites

HTTP Client Timeout Configuration:

• Added HTTP_CLIENT_TIMEOUT environment variable (default: 600 seconds)
• Applied to all external API calls (LLM providers, search engines, embeddings)
• Prevents indefinite hangs, improves reliability
• Integrated into installer wizard UI with validation

Tool Call ID Template Optimization:

• Implemented testTemplate() fast-path validation
• Single LLM call validates known templates vs 6-10 calls for full detection
• 83-90% reduction in startup time and token usage for 8 major providers
• Automatic fallback to full algorithm if validation fails

Bedrock toolConfig Fix:

• Reconstruct minimal tool definitions from conversation history when needed
• Ensures toolConfig always present when required by Converse API
• Fixes ValidationException for tool-heavy conversations

Dependency Updates:

• Updated OpenTelemetry SDK to v1.39.0 (rollback from v1.40.0 for compatibility)
• Updated golang.org/x/crypto to v0.46.0
• Updated google.golang.org/grpc to v1.79.3
• Frontend: dompurify 3.3.3, immutable 5.1.5

Closes #160

Type of Change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Configuration change
• 🧪 Test update
• 🛡️ Security update

Areas Affected

• Core Services (Frontend UI/Backend API)
• AI Agents (Researcher/Developer/Executor)
• Security Tools Integration
• Memory System (Vector Store/Knowledge Base)
• Monitoring Stack (Grafana/OpenTelemetry)
• Analytics Platform (Langfuse)
• External Integrations (LLM/Search APIs)
• Documentation
• Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: Latest development (feature/next_release)
Docker Version: 24.0.x+
Host OS: Linux/macOS
LLM Provider: OpenAI, Anthropic, AWS Bedrock
Go Version: 1.24.1

Test Steps

1. Run all backend unit tests: cd backend && go test ./...
2. Test HTTP timeout with slow external API (verify 600s default, custom values, zero=unlimited)
3. Restart PentAGI and measure tool call ID template detection time (should be <1s for known providers)
4. Test AWS Bedrock with multi-turn tool conversations (no ValidationException)
5. Verify all installer wizard fields work correctly
6. Run hermetic config tests with ambient environment variables set

Test Results

• ✅ All 200+ new unit tests pass
• ✅ Existing tests continue to pass
• ✅ HTTP timeout prevents indefinite hangs (verified with mock slow server)
• ✅ Tool call ID detection: <1s vs ~5-10s on startup for Anthropic/OpenAI
• ✅ Bedrock toolConfig issue [Bug]: When using Bedrock, toolConfig is not defined causing failures. #160 resolved
• ✅ Config tests hermetic (pass regardless of shell environment)
• ✅ Backend: go fmt, go vet, golangci-lint clean
• ✅ Frontend: npm run lint passed

Security Considerations

No Security Impact:

• Test additions don't affect runtime behavior
• HTTP timeout improves DoS resistance (prevents resource exhaustion from hung connections)
• All changes maintain existing security model
• Community test contributions reviewed and sanitized

Performance Impact

Improvements:

• Tool Call ID Detection: 83-90% faster startup for major providers (1 LLM call vs 6-10)
• HTTP Timeout: Prevents goroutine leaks from hung connections, improves resource management
• Test Suite: No impact on runtime (development-time only)

Token Savings:

• ~500-1000 tokens saved per provider on PentAGI restart (tool call ID validation)
• Multiplied across 8 providers = 4000-8000 tokens saved per restart

Documentation Updates

• README.md updates - HTTP_CLIENT_TIMEOUT variable documented
• API documentation updates
• Configuration documentation updates - backend/docs/config.md enhanced
• GraphQL schema updates
• Other: Installer wizard localization strings, .env.example

Deployment Notes

New Environment Variable (Optional):

# HTTP client timeout for external APIs (default: 600 seconds, 0 = unlimited)
HTTP_CLIENT_TIMEOUT=600

Configuration Notes:

• Variable can be set via .env file or installer wizard
• Default 600s suitable for most LLM operations
• Set to 0 to disable timeout (not recommended in production)
• Applied to: LLM providers, search engines, embedding APIs, external tools

Compatibility:

• ✅ Fully backward compatible
• ✅ No breaking changes
• ✅ Existing deployments work without modification
• ✅ New tests don't affect runtime

Deployment Steps:

1. Pull latest changes
2. Update .env if custom timeout needed
3. Rebuild: docker compose build (dependency updates)
4. Restart: docker compose up -d

Checklist

Code Quality

• My code follows the project's coding standards
• I have added/updated necessary documentation
• I have added tests to cover my changes
• All new and existing tests pass
• I have run go fmt and go vet (for Go code)
• I have run npm run lint (for TypeScript/JavaScript code)

Security

• I have considered security implications
• Changes maintain or improve the security model
• Sensitive information has been properly handled

Compatibility

• Changes are backward compatible
• Breaking changes are clearly marked and documented
• Dependencies are properly updated

Documentation

• Documentation is clear and complete
• Comments are added for non-obvious code
• API changes are documented

Additional Notes

Key Changes by Category

Test Coverage Improvements

Added 200+ Unit Tests (PRs #198, #199, #200, #201, #202, #213, #214):

• pkg/version - Binary version, develop mode detection, binary name
• pkg/config - Configuration loading, defaults, env overrides, Installation ID handling
• pkg/terminal - Markdown detection, interactive prompts, context cancellation, JSON/markdown output
• pkg/server/response - HttpError type, 78 predefined errors, dev/production mode response handling
• pkg/providers/embeddings - All 7 providers (OpenAI, Ollama, Mistral, Jina, Huggingface, GoogleAI, VoyageAI)
• pkg/graph/context - User ID/type/permissions helpers, validation functions, admin regex
• pkg/server/context - Gin context helpers (GetInt64, GetUint64, GetString, GetStringArray, GetStringFromSession)

Test Quality Standards Applied:

• Table-driven tests with comprehensive edge cases
• t.Parallel() for independent test execution
• require.* for critical assertions (prevents panics)
• Hermetic tests (clearConfigEnv helper prevents ambient environment interference)
• Proper context cancellation testing (io.Pipe vs strings.NewReader)
• Global state restoration (version.PackageVer save/restore pattern)

Contributors: @mason5052

Performance Optimizations

Tool Call ID Template Fast-Path (Commit 55e0ac5):

• File: backend/pkg/providers/provider/agents.go
• New testTemplate() function validates known templates with single LLM call
• Integrated after cache lookup, before full detection algorithm
• Providers optimized: OpenAI, Anthropic, Bedrock, Gemini, DeepSeek, Kimi, Qwen, GLM (8/10)
• Result: 83-90% reduction in startup time and token usage

HTTP Client Timeout (PR #205, Commit 47ae6fe):

• Files: backend/pkg/system/utils.go, backend/pkg/config/config.go
• Added HTTP_CLIENT_TIMEOUT env var (default 600s, 0=unlimited)
• Applied to all http.Client instances via GetHTTPClient()
• Installer wizard integration with validation (must be >= 0)
• Documentation in README.md, backend/docs/config.md
• Result: Prevents indefinite hangs, improves reliability
• Contributor: @liri-ha (original), enhanced by maintainers

Bug Fixes

Bedrock toolConfig Requirement (PR #196):

• File: backend/pkg/providers/bedrock/bedrock.go
• Functions restoreMissedToolsFromChain(), collectToolUsageFromChain(), inferSchemaFromArguments()
• Reconstruct minimal tool definitions when conversation contains toolUse/toolResult blocks
• Ensures Converse API requirement met: toolConfig must be defined when using tool blocks
• Result: Fixes ValidationException in multi-turn tool conversations
• Contributor: @manusjs

Dependencies

Go Modules (PR #215, Commit 47ae6fe):

• go.opentelemetry.io/otel/sdk v1.36.0 → v1.39.0 (rolled back from v1.40.0 for compatibility)
• golang.org/x/crypto v0.44.0 → v0.46.0
• google.golang.org/grpc v1.73.0 → v1.79.3

Frontend Dependencies (PR #207):

• dompurify 3.3.1 → 3.3.3 (security patches)
• immutable 3.7.6 → 5.1.5 (major version update)

Contributors

This release includes contributions from:

• @mason5052 - Comprehensive test coverage (7 packages, 200+ tests), test quality standards
• @liri-ha - HTTP client timeout implementation
• @manusjs - Bedrock toolConfig fix
• @asdek (Dmitry Ng) - Test review and refinement, tool call ID optimization, integration

Special thanks to community contributors for improving PentAGI's code quality!

Merged Pull Requests

• fix(bedrock): always include toolConfig when messages contain toolUse/toolResult blocks #196 - Bedrock toolConfig requirement fix
• test: add unit tests for pkg/version package #198 - Unit tests for pkg/version
• test: add unit tests for pkg/config package #199 - Unit tests for pkg/config
• test: add unit tests for pkg/terminal package #200 - Unit tests for pkg/terminal
• test: add unit tests for pkg/server/response package #201 - Unit tests for pkg/server/response
• test: add unit tests for pkg/providers/embeddings package #202 - Unit tests for pkg/providers/embeddings
• fix(system): add configurable timeout to HTTP client #205 - HTTP client timeout configuration
• chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates #207 - Frontend dependency updates (dompurify, immutable)
• test: add unit tests for pkg/graph context helpers #213 - Unit tests for pkg/graph/context
• test: add unit tests for pkg/server/context helpers #214 - Unit tests for pkg/server/context
• chore(deps): bump the go_modules group across 1 directory with 3 updates #215 - Go module updates (OpenTelemetry, crypto, gRPC)

Issues Addressed

• [Bug]: When using Bedrock, toolConfig is not defined causing failures. #160 - AWS Bedrock ValidationException with toolUse/toolResult blocks
• Agent delegation context canceled — detached commands inherit parent context #176 - Context canceled on agent delegation (HTTP timeout as contributing factor)