Guidance on how to handle (expiring) tokens

[Fricounet]

Hello team 👋 I'm raising this issue mainly to spark a discussion around how the nydus-snapshotter handles tokens it needs in order to fetch images from private registries.
In my current setup, I use the CRI interceptor method to catch creds from the kubelet in a kubernetes environment.

I have the following issues with this method:

1. creds are only stored in the memory of the snapshotter. In case of restart, the creds are completely lost. One consequence of this is that attempting to recreate a nydusd daemon for an image will fail if the blobs are missing from disk since the new nydusd won't be able to fetch them from the registry
2. creds are never refreshed for a specific ref
3. inside nydusd, there is no real refresh mechanism either. The existing refresh token thread will only attempt to refresh the registry oauth token it got but once the main token nydusd was given in its config file expires, then nydusd can't access the registry anymore
4. from 3. the only method i see to avoid a situation where we get stuck with an IO request that can't complete is by using prefetch. But then we lose part of the benefits of Nydus because we end having to fetch the whole image on disk, even if only 5% is really used
5. an unpleasant consequence of 4. is that if a prefetch fails for some reason (TOOMANYREQUESTS from the registry for instance), then it won't get retried. And if the application end up needing the file after the token expired, well good luck with that

Based on the different concerns above, I'd like to ask you if we have any solution available currently to address those, or do you have ideas on how we could improve the situation?

But maybe I'm the only one that has to deal with expiring tokens 😅

Thanks in advance for your answers!