New ssh-init initContainer not working as intended, causes startup issues

[reegnz]

I've just upgraded to a recent renovate version and I'm seeing the following error from the init-container:

cp: target '/home/ubuntu/.ssh': Permission denied
chown: cannot access '/home/ubuntu/.ssh/*': Permission denied
chmod: cannot access '/home/ubuntu/.ssh/*': Permission denied

The chart version I'm using is 39.182.4, as far as I've seen this ssh-init functionaity was introduced about 2 weeks ago.

I assume the reason is that the entire pod runs as non-root, but the issue this change was introduced for is because the files in the secret are owned by root. You can't change the file owned by root to not be owned by root with a non-root user though. :P

[reegnz]

PR that introduced the issue is #2314.

@maxnitze can you advise how to fix this? IMHO the init container still must run as root to be able to change the permissions, your solution didn't consider that I believe.

[maxnitze]

Hey! I did not have the time to update to the new chart version myself and I'm still running my custom adaptions.

Let me check, why this is failing in the released chart.

[maxnitze]

Can you share your values? I assume you are running the CronJob as the root user? There might be an issue in that case. The init-container has a hardcoded

chown -R ubuntu:ubuntu /home/ubuntu/.ssh/*
chmod 600 /home/ubuntu/.ssh/*

Which should probably take the securityContext into consideration.

My values include

renovate:
  securityContext:
    runAsNonRoot: true
    runAsUser: 12021 # ubuntu
    runAsGroup: 12021 # ubuntu

So both the init-container as well as the renovate container run with the ubuntu user. Could be a quick-fix for you?

[viceice]

I can rollback the change if required. let me know 🤗

[reegnz]

So this was our current config for the cronjob permissions, we didn't run it as root as that's not a great practice and our org doesn't like to do it. We have used the following security context:

renovate:
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    runAsUser: 1000

I've tried with the ubuntu user and group and that seems to have fixed the issue, thanks @slamdev !

[reegnz]

I guess a caveat or maybe enforcing the init container to always run with the ubuntu user values would be helpful.
Maybe in general if the chart would set up the cronjob to run as non-root which would be great.

Should I open a ticket for running renovate as non-root by default (eg. enforcing with the k8s manifest, not just trusting the container)?

[maxnitze]

The securityContext might have been the problem for you then. If I look at the container, there is no user with id 1000:

usr@nb [02:42:06 PM] [~] 
-> % docker run --rm --entrypoint /bin/sh renovate/renovate:39.182.4 -c 'cat /etc/passwd' 
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
_apt:x:42:65534::/nonexistent:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
ubuntu:x:12021:0::/home/ubuntu:/bin/bash
usr@nb [02:42:12 PM] [~] 
-> % 

I think it would make sense to run both containers as user 12021 by default. This could be a breaking change though. I'm not sure if the cache is still accessible if you change the default user of the system. Has anyone experience with it?

[maxnitze]

The more I think about it the more sense it does make, from my perspective.

So we should probably set a default security context (even though it is breaking!) and also adapt the init-container to not hardcode the username and group, but rather do something like:

chown -R {{ .Values.renovate.securityContext.runAsUser }}:{{ .Values.renovate.securityContext.runAsGroup }} /home/ubuntu/.ssh/*
chmod 600 /home/ubuntu/.ssh/*

I find it a bit confusing though, that the securityContext is configurable, but the files are mounted to /home/ubuntu. Maybe we should switch to a more generic place, like /opt/ssh?

Another thing that comes to my mind: Should we make the read-only filesystem by default as well? This is also best practice. And I think we already have everything in volumes, right?