Harden response to `kbase` site jobs with missing htcondor cluster ID

[MrCreosote]

kbase site job submission proceeds as below:

• add subjob documents to Mongo
• add the main job document to Mongo in CREATED
• submit the job to htcondor
• add the htcondor cluster ID to the main job and set it to DOWNLOAD_SUBMITTED
• otherwise set the job to ERROR

Mongo or network failures can leave the job in several states:

1. Healthy in DOWNLOAD_SUBMITTED
2. ERROR with or without submission to htcondor
3. CREATED with or without submission to htcondor

All of these are very unlikely and need mongo or network downs at precisely the right instant to trigger them. In particular, the CREATED states need for mongo to be up to save the main job but down when updating the job state seconds (at most) later. Note there is a legitimate period where the job should exist in the CREATED state with no htcondor ID.

1. Needs no action obviously
2. In this case the dead subjob detector will eventually set all the subjobs to error, which will cause running htcondor procs, if any, to fail when trying to update their state in the server and be held. The htcondor missing stats dectector will poll and log forever since there's no htcondor ID.
3. In this case the dead subjob detector will will poll and log forever since there's no htcondor ID.

To fix this, add a _get_condor_cluster_id(job: AdminJobDetails) -> int method to the KBaseRunner. Anytime an operation needs the condor ID, it should call this method.

• If the condor ID is in the job, just return it.
• Query htcondor for the CTS job id vs. the cluster ID with a limit of 1.
  • Note this is much more expensive than querying by the indexed cluster ID.
  • If the query returns a value
    • Add it to the job with a check that the cluster ID list hasn't changed since pulling the job. The MongoDAO should throw an error if this occurs.
    • If no error occurs, return the cluster ID.
    • If an error occurs, pull the job and return the cluster ID from the new version of the job.
      • If there is still no cluster ID something very wrong is happening -> ValueError and bail.
  • Otherwise, if the job is older than time T (1 to 5 minutes?) set the job to error with an appropriate error message. We assume here that the submission never reached condor and never will. Throw an error to that effect
  • Otherwise, throw an error (or return a value) to the effect that the job may still be submitting to htcondor. The calling method can decide whether to poll or not.

In the htcondor stats backfiller, if an error is thrown such that the job will never have a htcondor ID set all the subjobs' htcondor stats to missing.