Node Startup Taint Feature Inquiry

[jackcasey-visier]

Summary

Hello :) We're a historically AWS shop, currently going multi-cloud and starting to run our workloads on Azure. So far the Azure Lustre CSI Driver has been working great!

One issue we ran into, is scheduling pods onto a node where the CSI failed to install. For context, the failure was due to these errors:

Unpacking amlfs-lustre-client-2.15.4-42-gd6d405d:amd64 (5.15.0-1084-azure) ...
Errors were encountered while processing:
/tmp/apt-dpkg-install-5LAgDm/00-lustre-client-2.15.4-42-gd6d405d_1_amd64.deb
needrestart is being skipped since dpkg has failed
Reading package lists...
E: Sub-process /usr/bin/dpkg returned an error code (1)

This was a one-off issue, and not an issue in itself, the problem was pods attempting to schedule onto this node, and hanging until manually fixed.

How we've managed this in AWS, is a startup taint the CSI driver removes upon successful initialization. Here are the docs for reference:

https://github.com/kubernetes-sigs/aws-fsx-csi-driver/blob/master/docs/install.md#configure-node-startup-taint

This ensures we don't schedule onto broken nodes, which get replaced by the autoscaler auto-magically :)

Proposal / Feature Request

The proposal is to add equivalent logic to the Azure Lustre CSI Driver.

Users would taint nodes at startup with something like:

azurelustre.csi.azure.com/agent-not-ready:NoExecute

This taint would be removed only upon successful init of the CSI Driver.

Summary

I'd be keen to know if something like this has been discussed, and if the community/maintainers are interested!

[dabradley]

This is a good request. I've created an item in our internal tracker for it, but I can't say when we might be able to fit it into our roadmap just yet.

[jackcasey-visier]

@dabradley Please let me know if you/the maintainers are open to it, we'd be happy to provide the contribution!

[dabradley]

Thank you for the offer but I've already got a lot of this done and am testing it out locally. We've got a number of smaller improvements like this that we should be able to release shortly, I'll be sure to update with progress!

[jackcasey-visier]

@dabradley Love to hear it! Happy to help test if needed. Otherwise, we're eager to get this out :) Thank you

[dabradley]

@jackcasey-visier If you have the opportunity, we'd appreciate you testing out the PR for this: #232

We've followed the general approach of other Azure CSI drivers, so let us know if there's anything about your use case that isn't covered by these changes.

Be aware that there have been a few other changes made to our development branch to help with readiness at startup as well.

[jackcasey-visier]

@dabradley Great news, thanks! I'll push out your changes in our dev environment, and report back if we run into any challenges. I should be able to within the next few days :)

Thanks again!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dabradley]

/remove-lifecycle stale

[dabradley]

@jackcasey-visier Sorry for the delay, unfortunately, we don't have an easy way to produce publicly available preview images before a release, but we've included this change in the latest v0.3.1 release (along with other improvements) if you wanted to try it. It follows the same idiom of the other CSI drivers for this task, so it should do what you need.