Skip to content

Missing mp.set_start_method in examples involving DragonExecutionBackendV3 #70

Description

@BenCarter44

Currently, when running 02-workload-heterogeneous.py in the /examples directory via dragon, I am unable to get the output of the bash ComputeTasks or Python ComputeTasks that include a process template.

Output:

Task task.000001: DONE output: 'nid004193' Stdout: 'Native Function'  Return 'nid004193'
Task task.000002: DONE output: ''  Stdout:   Return 0
Task task.000003: DONE output: ''  Stdout:   Return [0, 0, 0, 0]
Task task.000004: DONE output: 0   Stdout:   Return 0
Task task.000005: DONE output: [0, 0, 0, 0]  Stdout:   Return [0, 0, 0, 0]

I noticed that mp.set_start_method("dragon") was missing in the file. Adding that in fixed the output capture of all the ComputeTasks that define a process template (Task2-5).

Output after adding the mp.set_start_method('dragon"):

Task task.000001: DONE output: 'nid004193' Stdout: 'Native Function'  Return 'nid004193'
Task task.000002: DONE output: 'nid004193' Stdout: 'nid004193'  Return '0'
Task task.000003: DONE output: 'nid004193'
nid004193
nid004193
nid004193' Stdout: 'nid004193
nid004193
nid004193
nid004193'  Return '[0, 0, 0, 0]'

Task task.000004: DONE output: '0'  Stdout: 'Single Function'  Return '0'
Task task.000005: DONE output: '[0, 0, 0, 0]' Stdout: 'Parallel Function
Parallel Function
Parallel Function
Parallel Function'  Return '[0, 0, 0, 0]'

From my reading into Dragon and Rhapsody, this makes sense, as without the line, leftover artifacts from python's standard multiprocessing are being mixed in.


Also, when a process_template is used, although a ComputeTask may be a Python function, output is only accessible via stdout, not the return value.

For reference: I added a print statement to each of the Python task functions to test stdout:

    async def single_function():
        import socket
        print("Single Function", flush=True)
        return socket.gethostname()

    async def parallel_function():
        import socket
        print("Parallel Function", flush=True)
        return socket.gethostname()

    async def native_function():
        import socket
        print("Native Function", flush=True)
        return socket.gethostname()

And edited the final print-out in the for to be:

    for t in tasks:
        result = t.return_value if t.function else t.stdout  # <-- Need to add a check here for process_templates??
        print(f"Task {t.uid}: {t.state} output: '{result}' stdout: '{t.stdout}'")  

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions