Improve memory efficiency and resource cleanup

[Nk-rodrigues]

Summary

This PR improves memory efficiency and resource cleanup across ingestion/query flows by avoiding quadratic string building, reducing peak allocations, and ensuring database/client resources are properly released.

Motivation

The API can process large uploads and many chunks per file. A few code paths could retain resources (clients/pools) across restarts or build large intermediate strings/buffers. This PR focuses on reducing peak RAM and preventing resource retention without changing the public API.

What changed

• main.py
  • Close the async PostgreSQL pool on shutdown.
  • Close vector store resources on shutdown (SQLAlchemy engine or Mongo client).
  • Truncate request bodies for validation logging to avoid excessive memory usage when logging large payloads.
• app/services/mongo_client.py
  • Always close MongoClient in the health check to avoid accumulating connections.
• app/services/vector_store/factory.py
  • Cache MongoDB clients to avoid creating a new client on each vector store construction.
• app/services/vector_store/extended_pg_vector.py
  • Add close() to dispose the SQLAlchemy engine.
• app/services/vector_store/atlas_mongo_vector.py
  • Add close() to close the underlying MongoDB client.
• app/utils/document_loader.py
  • Stream CSV encoding conversion instead of reading the full file into memory.
  • Refactor process_documents() to use list accumulation + join.
• app/routes/document_routes.py
  • Refactor extract_text_from_documents() to use list accumulation + join.
  • Mutate Document metadata in-place to avoid duplicating Document objects.
  • Ensure UploadFile objects are closed in inally blocks.
• tests/services/test_vector_store.py
  • Add regression tests for the new close() hooks.

Notes

• Changes are intended to be behavior-preserving, focused on reducing memory overhead and ensuring resources are released deterministically.

Checklist

• Scoped change, no API contract changes intended
• Resource cleanup is handled on shutdown
• Improvements documented in code via docstrings/comments