Startup is slow with large number of additional namespaces

[ianlewis]

We have observed that with a large number of namespaces given to --additional-namespaces, the time of the controller is quite long. Currently it takes ~1s per namespace to start up. We have ~300 namespaces given to --additional-namespaces so startup takes ~5min.

There are a number of problems that arise because of the large startup:

• The HTTP server is only started after all namespace watches have been started. This means that liveness probes will fail during this time even though the controller is making progress.
• Startup probes can help alleviate the problem above. However, in our case we dynamically add namespaces to --additional-namespaces so we need to restart the controller to update the namespaces list.
• Due to the large startup time the controller lags in responsiveness to SealedSecret and Secret change events.

A few improvements come to mind:

• Improve the startup time of watches per namespace.
• Move start up of the HTTP server (at least for probes) earlier to before the namespace watch creation loop. Liveness probes can succeed, and Readiness probes can fail during this time. After the watches are started, the readiness probe can return successfully.

Related:

• Controller restart loop on first install of the Helm chart #697