feat(submitit): add wait_for_completion option for fire-and-forget job submission

[ListIndexOutOfRange]

Summary

Adds a wait_for_completion config option to the submitit launcher. When set to False, jobs are submitted and the launcher returns immediately without waiting for results.

Motivation

This is useful for long-running jobs (e.g., model training) when we don't want the submitting process to block.

Usage

hydra:
  launcher:
    wait_for_completion: false

Example output

[HYDRA] Submitted job 16136162
[HYDRA]   stdout: /path/to/.submitit/16136162/16136162_0_log.out
[HYDRA]   stderr: /path/to/.submitit/16136162/16136162_0_log.err

Design consideration

It seems like this feature was requested a few times and my approach is suspiciously simple, so I guess I'm missing something.

For now, I wanted this PR to be non-invasive. It does not require modifications to Hydra core. The thing is, as noted in #2479, "Hydra's BasicSweeper collects the returned values from each job in the sweep". Thus, I chose to return placeholders values, such as JobStatus.COMPLETED. A more invasive, perhaps cleaner way, of doing things would be to create a new JobStatus, like SUBMITTED or INPROGRESS ?

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

I guess a simple test with wait_for_completion=True is enough. I'll add it later if we agree on the usefulness and design.

Related Issues and PRs

• Add an option to exit after the submitit launcher has scheduled the run on slurm #2479: feature request
• No block slurm #2965: closed, unmerged
• submitit launcher async support #3017: closed, unmerged