Rebuild the schedd on a htcondor error

[MrCreosote]

If HTCondor is shut down, the Schedd built by the CTS on kbase job flow startup can no longer contact it. It needs to be rebuilt from scratch, so the client probably needs a method to rebuild a schedd.

Something like

class SchedProxy:
    def __init__(self, factory):
        self.factory = factory
        self.schedd = factory()

    def refresh(self):
        htcondor.SecMan().invalidateAllSessions()
        self.schedd = self.factory()

    def query(self, *args, **kwargs):
        try:
            return self.schedd.query(*args, **kwargs)
        except htcondor.HTCondorIOError:
            self.refresh()
            return self.schedd.query(*args, **kwargs)

[MrCreosote]

probably worth it to test the schedd connectivity before trying to create a job

import htcondor2

schedd = htcondor2.Schedd()
try:
    jobs = schedd.query(limit=1)
    print("Connected to schedd OK")
except Exception as e:
    print(f"Connection failed: {e}")

Do this by calling a method on condor.py in kbase.py preflight method

[MrCreosote]

Or schedd.ping()