[Bug] Pemja reference leaks in action execution and resource bridges

[joeyutong]

Search before asking

• I searched existing issues.

Related Action report: #1048. Fix: #1058.

Description

Repeated Python Action execution and resource calls create Pemja PyObject handles that own native Python references. Dropping the Java local variable does not release that reference; PyObject.close() performs the native decref. Handles left unclosed can retain Events, coroutines, messages, documents, and resource objects after Java has finished using them.

The affected lifetimes are:

• Actions: converted Events, returned coroutine wrappers, wrappers retrieved for each poll, ignored yielded/returned values, and interpreter globals left after completion.
• Resource calls: temporary ChatMessage/tool wrappers, input/output Documents and normalized vector queries, formatted MCP messages, and Mem0 MemorySet/MemorySetItem wrappers.
• Resource shutdown: chat/embedding connection and setup roots, vector-store roots, and MCP Server/Tool/Prompt roots. MCP discovery registers tools and prompts but leaves the Server wrapper outside the resource cache's shutdown path.

The Python connection's logical close() and the Java PyObject handle's native-reference release are separate operations. Calling the former through a Tool or Prompt does not release the Server wrapper's own handle.

This issue covers temporary values consumed by framework calls and root handles released during normal resource shutdown. Resource construction/initialization/discovery failure cleanup and ownership of arbitrary outgoing Tool results or custom metadata values are outside its scope.

Observed behavior

Local Flink reproductions show native-reference retention after forced Java/Python garbage collection:

Path	Workload	Retention when the corresponding reference is not released
Converted Action Event	100 records, 1 MiB payload each	100 Events and 100 MiB of payload
Coroutine wrapper returned by an Action or retrieved for polling	100 records, 1 MiB payload each; each boundary measured separately	100 Events and 100 awaitables retaining 100 MiB
Completed-coroutine global	100 completed Actions	100 globals and 100 awaitables
Ignored yielded or returned output	100 records, 1 MiB payload each; each output path measured separately	100 output objects and 100 MiB
Chat bridge	200 iterations, 256 KiB per tracked object	600 messages, 200 tool values, and 200 MiB
Vector bridge	200 iterations, 256 KiB per tracked object	600 documents, 200 normalized handles, and about 100 MiB
Resource recreation	200 resources, 256 KiB each	200 resources and 50 MiB; no logical close calls

Each ownership boundary was measured in a fresh TaskManager. Live-object counts and reachable payload bytes are the evidence here; these are bug-reproduction observations, not memory measurements of PR #1058.

How to reproduce

1. Use an affected build to repeatedly execute Python Actions or invoke Python-backed chat/vector resources in a long-lived interpreter.
2. Give tracked Python objects a measurable payload and keep weak references to them. For resource recreation, also count logical close calls.
3. Allow each operation to complete, then force Java/Python garbage collection and count surviving objects. For async Actions, count completed-coroutine globals as well.
4. Exercise the ownership boundaries separately to distinguish an unclosed Java handle from an interpreter-global reference.

Expected behavior:

• Temporary handles are released after use or conversion, including nested handles in a wholly discarded Action output.
• A pending coroutine remains alive and resumable through its interpreter-global reference; completion removes that global.
• call_python_awaitable retains its (finished, yielded_or_returned_value) contract.
• Normal resource shutdown invokes Python close() and releases each wrapper's native handle once, including the MCP Server wrapper used for discovery.
• Outgoing Tool results and custom metadata values preserve their existing representation and ownership.

Version and environment

• Reported deployment: Flink Agents release-0.3.
• Local reproduction: Flink 2.2.0, Pemja 0.5.7, Java 17, CPython 3.11.15, macOS.
• Implementation and verification: [runtime][python] Manage Pemja references across actions and resources #1058.

Are you willing to submit a PR?

• I'm willing to submit a PR!