[BUG] Docker Container Not Starting Due To API Token Error?

[reed-write]

I have just noticed that my cup docker container will not restart and have found this from the log :

docker logs --tail 50 cup

INFO Starting server, please wait...
ERROR GET https://gcr.io/v2/token?&service=gcr.io&scope=repository:zenika-hub/alpine-chrome:pull: Unexpected error: {"errors":[{"code":"DENIED","message":"This API method requires billing to be enabled. Please enable billing on project #24742942950 by visiting https://console.developers.google.com/billing/enable?project=24742942950 then retry. If you enabled billing for this project recently, wait a few minutes for the action to propagate to our systems and retry."}]}

I might be wrong but i think something needs to changed by the developer of cup due to the api used reaching capacity?

[sergi0g]

Hey @reed-write, thanks for reporting this! I think there were problems with that image in particular in the past, which were since resolved. I'll try to take a look tomorrow and compare with a docker client. The debugging I can do with gcr is limited unfortunately.

In case you see this message before I have time to take a look and the issue still occurs, can you confirm if you are able to make a fresh pull of it with the docker CLI? Thanks!

[reed-write]

Hey @reed-write, thanks for reporting this! I think there were problems with that image in particular in the past, which were since resolved. I'll try to take a look tomorrow and compare with a docker client. The debugging I can do with gcr is limited unfortunately.

In case you see this message before I have time to take a look and the issue still occurs, can you confirm if you are able to make a fresh pull of it with the docker CLI? Thanks!

I've just tried :

docker pull gcr.io/zenika-hub/alpine-chrome:latest

from terminal

and this was returned :

Error response from daemon: error from registry: This API method requires billing to be enabled. Please enable billing on project #24742942950 by visiting https://console.developers.google.com/billing/enable?project=24742942950 then retry. If you enabled billing for this project recently, wait a few minutes for the action to propagate to our systems and retry.

I hope this helps? Unless I have miss understood what you were asking me to do.

[jrhedman]

I've just tried :

docker pull gcr.io/zenika-hub/alpine-chrome:latest

from terminal

and this was returned :

Error response from daemon: error from registry: This API method requires billing to be enabled. Please enable billing on project #24742942950 by visiting https://console.developers.google.com/billing/enable?project=24742942950 then retry. If you enabled billing for this project recently, wait a few minutes for the action to propagate to our systems and retry.

Chiming in since I had this issue too (among a number of other things that all decided to catch on fire today... never just one...)

Actually if you're getting this error trying to pull it too... I'm leaning more towards this being an issue on Zenika's side than on cup.

[jrhedman]

For the time being, we may need to revert this to pull from the docker hub itself zenika/alpine-chrome

[reed-write]

I tried pre-pulling zenika/alpine-chrome:latest from Docker Hub and retagging it locally as gcr.io/zenika-hub/alpine-chrome:latest, but CUP still crashes on boot.
It looks like CUP makes an explicit API call to https://gcr.io/v2/token... to fetch the manifest/token before running the container, so it fails on the network request before it ever checks local Docker images. An update to point the registry reference to Docker Hub (zenika/alpine-chrome) or ghcr.io in the source code seems to be required?

[sergi0g]

Alright, thanks for confirming! Cup and Docker are both doing the right thing, Google is the one which doesn't want to play nice with their weird policies. Zenika will have to fix this.

You can resolve this issue permanently by switching your references over to a ghcr or Docker Hub version and deleting the gcr tag.

On Cup's side, no I won't introduce an override in the source code, I haven't done that for anything before and one day it'll break anyway. What I can do is make Cup handle this failure gracefully instead of crashing. I'll try to see to it soon!

[reed-write]

I have been able to fix but was a slightly strange process. Here is what i did.

Checking my cup docker-compose file this was it's contents:

services:
  cup:
    image: ghcr.io/sergi0g/cup:latest
    container_name: cup
    restart: unless-stopped
    command: -c /config/cup.json serve
    environment:
      - CUP_REFRESH_INTERVAL=0 0 * * * *    # <-- Added an extra 0 for seconds

    ports:
      - 8085:8000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config:/config

The compose file already used GHCR.

With a little help from my friend.... Chat GPT, i discovered that....

nano config/cup.json

{ "$schema": "https://raw.githubusercontent.com/sergi0g/cup/main/cup.schema.json", "version": 3, "theme": "blue", "showContainersWithNoUpdates": true }

confirms cup.json is not the source of the GCR reference. CUP is discovering it from another Docker container.

So i used the below to find the container using it :

docker ps -a --format 'table {{.Names}}\t{{.Image}}' | grep gcr.io

this turned out to be this container :

karakeep-chrome-1

then on my karakeep compose file I amended the line :

image: gcr.io/zenika-hub/alpine-chrome:124

to

image: zenika/alpine-chrome:124

then for karakeep

docker compose pull docker compose up -d --force-recreate

deleted the obsolete local GCR image:

docker image rm gcr.io/zenika-hub/alpine-chrome:TAG

finally i restarted the cup docker container

docker restart cup

Now all is all well with Cup being back to normal. (And Karakeep running as normal as far as i can see)

Hope this makes sense. I wouldn't call myself a developer more of a Self-hosting Enthusiast. I am a but confused how the cup container was broken by something referenced in another docker container / docker compose file. But I suppose this is the interesting learning part of self-hosting!

[jrhedman]

I am a but confused how the cup container was broken by something referenced in another docker container / docker compose file. But I suppose this is the interesting learning part of self-hosting!

So how cup works, is it queries the docker daemon (what is managing your containers) and asks for a list of containers and where they came from (sort of like doing docker image list). It then reaches out to all those registries and says "Hey, what's the latest version of this container?" and pulls details about what it finds (not the actual container) like the version, hash (cryptographic fingerprint) and compares it against the hashes of what currently exists on your system. This is how it can let you know about a new "latest" container, or if you have your version pinned, it will let you know that there's a newer version available and what that version number is.

So what happened in this case, is that cup was reaching out to gcr.io/zenika-hub/alpine-chrome to pull the latest container they have. Well.. either someone at zenika forgot to pay the bill, or something else happened but, that registry isn't active currently and only they are able to fix it.

What you did was change the registry to zenika/alpine-chrome, not to different right? so gcr.io is Google's registry that you can host containers on. if you don't add a domain name like what you changed it to, docker will default to their own registry (docker.io).
(If you see ghcr.io, that one is hosted on GitHub itself)

So now cup will check this location instead of the Google one. You can kind of think of it like different mirrors for downloading a file, if one url is broken, you can probably use a different one. Containers like this are pretty popular/common so they deploy onto multiple locations you can see on their docker hub they have domains you can use for eu, asia, etc. but for now, all their Google ones aren't working.

Extra Credit: You noticed that cup was still having issues after you changed the compose file and had karakeep running with the new container right? That goes back to how cup inventories your containers. It's not looking at your compose files, it's looking at whatever images you have on your system, running or not, and looks to see if there are any updates. That's why it continued to crash until you deleted the old image. Even though it wasn't running, it was still there so cup dutifully continued to see if it had any updates. This is a good thing, you can bring a container down for any number of reasons so it's good to still track if there are any updates for what you currently have installed.

Sorry for a long winded response, but I felt you deserved a proper answer. Keep being curious, and keep sharing bugs! Maybe one day you'll feel confident to try tackling some yourself. It's things like this that help us all make better tools 🥇

[jrhedman]

On Cup's side, no I won't introduce an override in the source code, I haven't done that for anything before and one day it'll break anyway. What I can do is make Cup handle this failure gracefully instead of crashing. I'll try to see to it soon!

@sergi0g, I wonder if you can implement some try/catch for this and maybe have it display a warning sign or maybe place images that had a fault at the top of the list in red and in the dropdown you could display the returned error message?

It's a rare enough occurrence something like this ever happens, but it'll be good to have some way to quickly see what failed and why.

Thanks for such an invaluable tool!

[sergi0g]

@reed-write, sorry if I didn't explain it well enough, I had basically said what you ended up doing:

You can resolve this issue permanently by switching your references over to a ghcr or Docker Hub version and deleting the gcr tag.

Maybe "deleting the gcr tag" wasn't too obvious, sorry for wasting your time like that!

I hope @jrhedman cleared it up for you, that's a very nice explanation of what Cup does (thanks! 😉).

I wonder if you can implement some try/catch for this and maybe have it display a warning sign or maybe place images that had a fault at the top of the list in red and in the dropdown you could display the returned error message?

This is already being done for a bunch of other stuff in Cup and you'll get warnings in the logs instead of errors. The images will show up as unknown in the web UI and you can click them to see a message about what went wrong to help you debug.

The difference is that I don't show stuff in red, nor at the top, because in most cases, like this one, the user isn't the cause of the problem and showing big red warnings they have no influence over will just make the user experience worse IMO.

The plan is to gracefully handle this failure too, just like the others. I'm a bit busy these days, sorry for the delays. I'll try to get it fixed soon.

[ThatLurker]

In case someone is here because of karakeep using the zenika-hub/alpine-chrome image they migrated to their own https://docs.karakeep.app/next/administration/chrome-image-migration/