[ader1990]

In case of the node removal, the workflow was the following:

1. remove the node from microcluster
2. remove the machine

This workflow has the big problem that step 1 destroys the node
without cleaning it up first (without cordon, drain, volume detachment).

Also, another major problem is that step 1 is async, and step 2 can
actually be done in a more eager manner, and the underlying hardware aka machine
can be wiped out before the actual remove of the node by snap-k8s can be
done safely.

In addition, step 1 will block the capi operator in doing the cordon, drain and
volume detachment, as the node will be removed in an eager manner as well.

This what-if workflow has to be changed to one that performs in a logical safe way:

1. add the pre-remove annotation for all machines
2. remove the machine -> this triggers the machine removal -> capi makes sure that the
   cordon, drain, volume detachment is done safely. After the cordon, drain, volume detachment
   is complete, capi waits for the pre-remove annotation to be removed.
3. ck8s machine controller waits until the cordon, drain, volume detachment is done by
   veryfing the conditions. Then, it verifies that CAPI set the waiting for pre-remove hook to
   be executed. Once CAPI set that condition on the machine, ck8s machine controller takes over
   and issues the remove the node from microcluster. Then, it verifies that the node has been
   properly removed. Then, it remove the pre-remove anotation from that machine.

There is currently a gap in snap-k8s API - there is currently no API to verify that the node
has been properly removed. This feature needs to be implemented. Until then, ck8s machine
controller can only blindly try to execute the node removal a few times, making sure that
the node has been removed via this "hack".