Make Edge Worker using async loop

[jscheffl]

So far the Edge worker running remote is using a single threaded loop to fetch, process, monitor tasks and upload results logs as well as heartbeat. This limits the parallelism achieveable on a Edge Worker - how many tasks can be handled in parallel.

With this PR a major refactor is made to use Python AsyncIO for handling of tasks in order to improve concurrency with many tasks.

Note: Before merging this needs to have proper review and testing as a lot of logic and libraries change with the risk of degraded quality/stability as of implementation glitched. UPDATE: WIP status left, did a lot of (local) testing and I think it is ready now.

Tested with 100 concurrent tasks (mostly sleeping) on my machine and worker used only 10-20% of CPU (compared to 10GB RAM all the task needed). So with the re-implementation in AsyncIO the worker scales very much more than before where I considered 10-15 tasks can be handled.

FYI @dheerajturaga - As also being a user, committer status still pending but looking forward for a review!
@dabla As you do a lot of AsyncIO on your side, looking forward for a review from you as well!
@AutomationDev85 - Would also like your feedback!

[dabla]

That will be a nice improvement @jscheffl. Looking forward to this!

[dabla]

I've just checked the code and I saw maybe a possible "improvement" regarding following code in worker:

if worker_info.state == EdgeWorkerState.MAINTENANCE_REQUEST:
                logger.info("Maintenance mode requested!")
                EdgeWorker.maintenance_mode = True
            elif (
                worker_info.state in [EdgeWorkerState.IDLE, EdgeWorkerState.RUNNING]
                and EdgeWorker.maintenance_mode
            ):
                logger.info("Exit Maintenance mode requested!")
                EdgeWorker.maintenance_mode = False
            if EdgeWorker.maintenance_mode:
                EdgeWorker.maintenance_comments = worker_info.maintenance_comments
            else:
                EdgeWorker.maintenance_comments = None
            if worker_info.state == EdgeWorkerState.SHUTDOWN_REQUEST:
                logger.info("Shutdown requested!")

We could encapsulate the checks of the state in the WorkerInfo dataclass through properties, then above code would be easier to read and to test also, as then you can test the check of state directly in the WorkerInfo dataclass. So I would add following properties in WorkerInfo dataclass:

@property
def is_maintenance(self) -> bool:
     return self.state == EdgeWorkerState.MAINTENANCE_REQUEST

@property            
def is_running_or_idle(self) -> bool:
     return self.state in [EdgeWorkerState.IDLE, EdgeWorkerState.RUNNING]

@property
def is_shutdown(self) -> bool:
     return self.state == EdgeWorkerState.SHUTDOWN_REQUEST

Then you could rewrite following as below and add dedicated tests in WorkerInfo dataclass for above properties:

if worker_info.is_maintenance:
                logger.info("Maintenance mode requested!")
                EdgeWorker.maintenance_mode = True
            elif (
                worker_info.is_running_or_idle
                and EdgeWorker.maintenance_mode
            ):
                logger.info("Exit Maintenance mode requested!")
                EdgeWorker.maintenance_mode = False
            if EdgeWorker.maintenance_mode:
                EdgeWorker.maintenance_comments = worker_info.maintenance_comments
            else:
                EdgeWorker.maintenance_comments = None
            if worker_info.is_shutdown:
                logger.info("Shutdown requested!")

WDYT?

[jscheffl]

WDYT?

Yes, the code is not perfect and the state management has grown over time. I am also not 100% happy about it, if you have a good idea and some time... like with all: contributions are welcome.

[dabla]

Really like the refactorings being done here, looking forward to test it once it's done ;-)

[jscheffl]

Almost ready to review, need to hunt for one bug that I saw (and had no time, was distracted by family) hope I can make it ready for review by EOB today.