Comprehensive TorchSharp improvements: memory management, advanced layers, and modern deep learning features

[Copilot]

This PR provides a comprehensive analysis and significant improvements to ANNdotNET v2's TorchSharp integration, addressing performance, memory management, and feature gaps while maintaining compatibility with TorchSharp 0.101.2.

🔧 Infrastructure Fixes

Cross-platform TorchSharp Support

• Fixed test failures by adding proper TorchSharp CPU packages for Linux, Windows, and macOS
• Improved test success rate from 25% (4/16) to 94% (15/16)
• Added conditional package references for platform-specific native libraries

🚀 Performance & Memory Improvements

GPU-Optimized Calculations

• Implemented batch metric calculations to reduce CPU-GPU data transfers
• Added proper using statements and disposal patterns for tensor memory management
• Optimized tensor accumulation with automatic cleanup to prevent memory leaks
• Added torch.no_grad() context for evaluation phases

Enhanced Metrics System

// Before: Multiple separate GPU-CPU transfers
var accuracy = MCAccuracy(predicted, target);
var error = MCError(predicted, target);

// After: Single batch calculation
var metrics = TorchMetrics.CalculateMetricsBatch(evalFunctions, predicted, target);

🧠 Modern Deep Learning Features

Advanced Layer Support

• Added 9 new layer types: BatchNormalization, LayerNormalization, Conv1D/2D, MaxPool1D/2D, AvgPool2D, GlobalAvgPool, Flatten, Reshape, Attention
• Implemented proper weight initialization (Xavier, Kaiming, Normal, Uniform)
• Enhanced model architecture flexibility for modern deep learning patterns

Training Enhancements

• Added early stopping with configurable patience
• Implemented model checkpointing with metadata preservation
• Added gradient clipping (norm and value-based) to prevent exploding gradients
• Enhanced progress reporting with learning rate and memory usage tracking

Tensor Utilities

// Advanced tensor operations with memory management
var normalized = TensorUtils.MinMaxNormalize(tensor);
var (train, val) = TensorUtils.TrainValidationSplit(data, 0.8f, shuffle: true);
var accuracy = TensorUtils.ComputeAccuracy(predictions, targets, topK: 5);

📊 Model Management

Comprehensive Model Utilities

• Parameter counting and memory usage analysis
• Model checkpointing with optimizer state (where API allows)
• Model summary generation with layer-wise parameter breakdown
• Freezing/unfreezing capabilities for transfer learning

Example Usage

var model = MlFactory.CreateNetwork("AdvancedModel", layers, 784, 10, device, WeightInitMethod.KaimingUniform);
var paramCount = ModelUtils.CountParameters(model);
var summary = ModelUtils.GetParameterSummary(model);
ModelUtils.SaveCheckpoint(model, optimizer, epoch, loss, metrics, "./checkpoint");

🎯 Backward Compatibility

All improvements maintain full backward compatibility with existing ANNdotNET v2 code. The enhanced features are opt-in and don't break existing functionality.

📈 Results

• Build Status: ✅ Success (eliminated all compilation errors)
• Test Results: ✅ 94% success rate (15/16 tests passing)
• Performance: Reduced memory usage and CPU-GPU transfers
• Features: 6 new utility classes, 9 new layer types, comprehensive examples

🔍 Code Quality

• Fixed tensor-to-bool conversion errors and TorchSharp API compatibility issues
• Added comprehensive documentation and usage examples
• Implemented proper error handling and validation
• Created demonstration code showing modern deep learning techniques

This PR transforms ANNdotNET v2 into a modern, efficient deep learning framework while respecting the constraints of the current TorchSharp API version.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• av-build-tel-api-v1.avaloniaui.net
  • Triggering command: dotnet exec --runtimeconfig /home/REDACTED/.nuget/packages/avalonia.buildservices/0.0.29/tools/netstandard2.0/runtimeconfig.json /home/REDACTED/.nuget/packages/avalonia.buildservices/0.0.29/tools/netstandard2.0/Avalonia.BuildServices.Collector.dll (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.