First step: open the admin Allocator Logs page or run lablink logs,
then run the built-in checks and read their output:
lablink doctor # operator-side prerequisites, for the provider you configured
lablink status # allocator health and client inventory
lablink client doctor # on a BYO client box: registration, container, log shipperlablink doctor branches on your provider — Docker checks under manual, and OpenTofu, AWS credentials, S3 and AMI checks under aws. Most problems below are named in its output.
Run tofu output commands below from the initialized infrastructure
directory: lablink-infrastructure/ in a template checkout, or
~/.lablink/deploy/<deployment_name>/<environment>/ for the CLI.
!!! info "Which port?"
Inside the AWS host, the allocator container listens on 5000. Caddy
proxies host port 80 to it for none, letsencrypt, and cloudflare;
acm uses an ALB. With domain-based HTTPS, port 5000 may be bound to
loopback, so test through the configured public URL from outside. A
manual-provider compose stack exposes host port 80.
??? note "Permission denied connecting to the Docker daemon"
Got permission denied while trying to connect to the Docker daemon socket
Add yourself to the `docker` group:
```bash
sudo usermod -aG docker $USER
newgrp docker
docker ps
```
??? note "Cannot connect to the Docker daemon"
Cannot connect to the Docker daemon at unix:///var/run/docker.sock
Linux — start the service:
```bash
sudo systemctl start docker
sudo systemctl enable docker
```
macOS / Windows — start Docker Desktop.
??? note "Permission denied (publickey)" In order, the three things that cause this:
1. **Key permissions too open** — SSH silently refuses a world-readable key.
```bash
chmod 600 ~/lablink-key.pem
ls -l ~/lablink-key.pem # -rw-------
```
2. **Wrong key** — the deployment key opens the allocator, while client
VMs use the allocator-generated keypair. For the allocator, re-extract
it from OpenTofu state:
```bash
tofu output -raw private_key_pem > ~/lablink-key.pem
chmod 600 ~/lablink-key.pem
```
3. **Wrong user** — it's `ubuntu` on the Ubuntu AMIs LabLink uses, not `ec2-user`.
??? note "SSH connection timed out" Timeout means the packets aren't arriving at all — a closed port or a stopped instance, not an auth problem.
```bash
aws ec2 describe-instances --instance-ids <id> \
--query 'Reservations[0].Instances[0].State.Name'
tofu output ec2_public_ip
```
The template already allows inbound SSH on port 22; check the instance
state, effective security group, and VPC network ACL before changing rules.
??? note "How do I get the client-VM SSH key?"
Client VMs use a different key from the allocator. On an AWS deployment,
first SSH to the allocator with its deployment key. In the production
allocator container, the client key is the lablink_private_key_pem
OpenTofu output. Save it to a private file on the allocator host:
```bash
sudo docker ps --format '{{.ID}} {{.Image}}' # find the allocator container ID
sudo docker exec <allocator-container-id> sh -lc \
'cd /app/.venv/lib/python*/site-packages/lablink_allocator_service/terraform && tofu output -raw lablink_private_key_pem' \
> ~/lablink-client-key.pem
chmod 600 ~/lablink-client-key.pem
```
The client-VM public IP comes from the allocator's client inventory or
the client OpenTofu outputs, not `ec2_public_ip` in the deployment state.
??? note "OpenTofu init fails: bucket does not exist"
lablink setup creates the S3 state bucket and DynamoDB lock table and
writes the bucket name into your config. If either is missing, re-run:
```bash
lablink setup
```
The CLI bucket name is `lablink-tf-state-<account-id>`.
??? note "Error acquiring the state lock" A previous OpenTofu run died without releasing its DynamoDB lock.
First make sure nothing is actually still running:
```bash
aws dynamodb scan --table-name lock-table --region us-west-2
ps aux | grep tofu
```
Entries with an `Info` field are real locks; the rest are digests. Copy the exact `LockID` — it does **not** always carry an `-md5` suffix — and delete it:
```bash
aws dynamodb delete-item \
--table-name lock-table \
--key '{"LockID":{"S":"<exact-lock-id-from-scan>"}}' \
--region us-west-2
```
On Windows PowerShell, put the key in a `key.json` file and pass `--key file://key.json`.
Common lock paths:
- Template infrastructure: `<bucket>/<env>/terraform.tfstate`
- CLI infrastructure: `lablink-tf-state-<account-id>/<deployment_name>/<env>/terraform.tfstate`
- Client VMs: `<bucket>/<deployment_name>/<env>/client/terraform.tfstate`
If you get `AccessDeniedException: not authorized to perform: dynamodb:GetItem`, the allocator's IAM role is missing `dynamodb:GetItem`, `PutItem` and `DeleteItem` on `table/lock-table`. Add them and redeploy.
**Prevention:** let OpenTofu runs finish. Don't terminate instances from the console mid-apply, and use the destroy workflow rather than deleting resources by hand.
??? note "Resource already exists"
Inspect the resource and state before retrying. For a separate deployment,
choose a different deployment_name or environment in the config and
workflow inputs. See Template Deployment.
??? note "Resources won't destroy cleanly"
Inspect the failed OpenTofu destroy output and check whether client VMs
still exist. The template destroy workflow removes client VMs first. For
orphaned resources after a failed destroy, preview the
template cleanup script or run
lablink cleanup --dry-run on the CLI path before deleting anything.
??? note "GitHub Actions: could not assume role with OIDC"
bash aws iam list-open-id-connect-providers aws iam get-role --role-name github-actions-lablink \ --query 'Role.AssumeRolePolicyDocument'
The trust policy must name your repository (`repo:<owner>/<repo>:*`), and the role ARN in the workflow must match your account ID.
??? note "GitHub Actions: workflow won't trigger"
Check the branch name and path filters against the workflow's on: block, confirm Actions are enabled for the repository, and validate the YAML:
```bash
yamllint .github/workflows/*.yml
```
??? note "Browser shows connection refused" Work outward from the container:
```bash
ssh -i ~/lablink-key.pem ubuntu@<ip>
sudo docker ps # is it running?
sudo docker logs -f <container> # did it crash on startup?
curl localhost:5000 # does it answer locally?
```
The template's security group already allows 80, 443, 5000, and 22, but
HTTPS modes may bind 5000 only to loopback. Check Caddy, the public URL,
and the effective security group rather than opening another port.
??? note "Browser cannot reach the HTTP site (no SSL provider)"
Symptoms: "This site can't be reached", ERR_CONNECTION_REFUSED on a deployment using ssl.provider: "none".
Your browser cached an HSTS policy from a previous HTTPS deployment on the same hostname, so it silently upgrades to HTTPS — and port 443 is closed.
Quickest workarounds: use a private/incognito window, or hit the IP directly.
To clear it properly:
=== "Chrome / Edge"
1. Open `chrome://net-internals/#hsts` (`edge://` on Edge)
2. Under **Delete domain security policies**, enter your domain
3. Click **Delete**, then reload with an explicit `http://`
=== "Firefox"
1. Close all Firefox windows
2. Delete `SiteSecurityServiceState.txt` from your profile directory
(`~/Library/Application Support/Firefox/Profiles/` on macOS,
`%APPDATA%\Mozilla\Firefox\Profiles\` on Windows,
`~/.mozilla/firefox/` on Linux)
3. Restart and reload with an explicit `http://`
=== "Safari"
1. Quit Safari
2. `rm ~/Library/Cookies/HSTS.plist`
3. Restart and reload with an explicit `http://`
Expected behaviour with `ssl.provider: "none"` is a "Not Secure" badge — no certificate is issued and traffic is unencrypted. For HTTPS, set `ssl.provider` to `letsencrypt`, `cloudflare` or `acm`. See [Configuration → SSL Options](configuration.md#ssltls-options-ssl).
??? note "HTTP works but HTTPS doesn't" Caddy cannot get a certificate until DNS resolves to the allocator, so check that first:
```bash
nslookup <your-domain> 8.8.8.8
sudo systemctl status caddy
sudo journalctl -u caddy -f
```
In the Caddy log, `certificate obtained successfully` means it worked; `challenge failed` means DNS isn't ready; `timeout` means ports 80/443 aren't reachable. Caddy retries every 2 minutes on its own, so a few minutes of patience often fixes it.
Verify the Caddyfile proxies to the container's real port:
```bash
cat /etc/caddy/Caddyfile
# <your-domain> {
# reverse_proxy localhost:5000
# }
sudo systemctl restart caddy
```
See [Configuration](configuration.md#ssl-providers) for the Let's Encrypt
duplicate-certificate limit during repeated test deployments.
??? note "Domain doesn't resolve to the allocator"
```bash
# Does the record exist?
aws route53 list-resource-record-sets --hosted-zone-id
--query "ResourceRecordSets[?Name=='.']"
# Does it match the actual IP?
tofu output ec2_public_ip
dig <your-domain> +short
```
If the record is missing, re-run `tofu apply`. If it exists but isn't propagating, give it 5–15 minutes and flush your local cache (`sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder` on macOS, `sudo systemd-resolve --flush-caches` on Linux, `ipconfig /flushdns` on Windows).
??? note "DNS record landed in the wrong hosted zone"
OpenTofu's zone lookup matched the parent zone (e.g. example.com instead of lablink.example.com). Pin it explicitly:
```yaml
dns:
enabled: true
terraform_managed: true
domain: "test.lablink.example.com"
zone_id: "Z0123456789ABCDEFGHIJ" # force the correct zone
```
Delete the stray record from the wrong zone, then re-apply.
??? note "Verifying a deployment end to end"
The template repo ships a script that checks DNS, HTTP, and SSL together. It lives in scripts/, not lablink-infrastructure/:
```bash
./scripts/verify-deployment.sh prod # reads config.yaml + tofu outputs
./scripts/verify-deployment.sh <domain> <ip> # or pass them explicitly
./scripts/verify-deployment.sh --ci prod # no ANSI colors, for CI logs
```
Green checks are pass, yellow warnings usually mean "not ready yet — wait and retry", red is a real failure. The three common ones are a DNS timeout (wait longer), HTTP not responding (check the allocator container logs), and SSL not ready (check the Caddy logs).
??? note "NS delegation not working"
nslookup returns your registrar's nameservers instead of AWS, so records in Route53 are never consulted.
```bash
dig NS lablink.example.com # should list four awsdns nameservers
aws route53 get-hosted-zone --id <zone-id> --query 'DelegationSet.NameServers'
```
Add those four as NS records for the `lablink` subdomain at whoever hosts the parent domain, TTL 300, then re-check after 5–15 minutes.
If you have both a parent and child zone in Route53 fighting each other, keep the one LabLink manages and delete the duplicate — `aws route53 list-hosted-zones --query 'HostedZones[*].[Name,Id]' --output table` shows them all.
??? note "Clicking Create New VM Instance does nothing"
bash sudo docker logs -f <allocator-container> sudo docker exec <allocator-container> aws sts get-caller-identity sudo docker exec <allocator-container> tofu version
If `get-caller-identity` fails, the allocator's IAM instance profile isn't attached or lacks EC2 permissions (`RunInstances`, `DescribeInstances`) and VPC permissions (`CreateSecurityGroup`).
To run OpenTofu by hand inside the container:
```bash
sudo docker exec -it <allocator-container> bash
cd /app/.venv/lib/python*/site-packages/lablink_allocator_service/terraform
tofu init && tofu plan
```
??? note "VM created but never appears in the allocator"
Clients register themselves — the allocator does not insert rows when OpenTofu finishes. Each client posts to /api/v1/clients/register, and the allocator stores its hostname alongside an argon2 hash of a freshly minted client_secret. A VM that never registers is a VM that never reached that endpoint.
```bash
# What the allocator has
sudo docker exec <allocator-container> psql -U lablink -d lablink_db \
-c "SELECT hostname, status, inuse FROM vms;"
# What the allocator saw
sudo docker logs <allocator-container> | grep -E "clients/register|heartbeat"
# What the client tried (use the allocator-generated client key)
ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip>
sudo docker logs <client-container>
# Can the client reach the allocator at all?
curl -i http://<allocator-ip>/api/health # IP-only mode; use the configured HTTPS domain otherwise
```
If `/api/health` fails from the client, stop debugging the client — it's the security group or the allocator's listening port.
!!! danger "Don't INSERT rows by hand"
A row created with just `(hostname, inuse)` has a null `client_secret_hash`, so that client can never authenticate for heartbeats, status or logs. It will look present in the admin panel and be permanently broken. Let the client register, or re-provision it.
??? note "Failed VMs aren't being rebooted"
The allocator runs an AutoRebootService that sweeps every 60 seconds for VMs that are in error, running but GPU-Unhealthy, stuck initializing over 25 minutes, stuck rebooting over 10 minutes, or running but silent for 3 minutes with no heartbeat.
Assigned VMs get a warm `sudo reboot` that keeps their container state.
Unassigned VMs get a cold reboot (`docker rm -f`, `cloud-init clean`, then
`reboot`) that reprovisions from scratch. EC2 stop/start is used only if
SSH fails. There are at most 3 attempts with a 300-second cooldown; an
exhausted VM becomes `error` and its seat is released.
```bash
sudo docker logs <allocator-container> | grep -i reboot
sudo docker exec <allocator-container> psql -U lablink -d lablink_db -c \
"SELECT hostname, status, reboot_count, last_reboot_time FROM vms WHERE reboot_count >= 3;"
```
There is no admin reset endpoint for an exhausted VM. Review its logs and
replace it if needed; an out-of-band EC2 reboot alone does not reset the
allocator's attempt counter.
!!! note "BYO boxes are never rebooted"
Only providers that can recover hosts participate. The `manual` provider can't, so bring-your-own machines are left alone.
??? note "CUDA not available on a client VM"
bash ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip> nvidia-smi docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi sudo docker inspect lablink-client --format '{{.HostConfig.Runtime}}'
The client container is launched with `--runtime=nvidia`; a Docker
`default-runtime` setting is not required. If `nvidia-smi` fails on the
host, use an AMI with NVIDIA drivers. Confirm the instance type has a GPU.
??? note "Start here: lablink client doctor"
Run it on the client box. It checks three things — that ~/.lablink/client.env exists with this box's credentials, that the lablink-client container is running, and that the log shipper is forwarding to the allocator.
Most failures are fixed by re-running `lablink client register`. See [Bring-Your-Own Clients](cli/byo-clients.md).
??? note "The client registers but participants can't reach the desktop"
Almost always the wrong manual.connectivity mode for your network:
- `lan_direct` puts the participant's browser onto the client's LAN IP. It **cannot** serve off-LAN participants, no matter what exposure mode you pair it with.
- `mesh_overlay` joins the client to a Tailscale tailnet and proxies the byte path through the allocator's nginx.
- `reverse_tunnel` has the client dial out and hold one connection open — for networks that won't carry Tailscale or boxes that can't accept inbound connections.
See [Pick a connectivity mode](cli/byo-clients.md#pick-a-connectivity-mode).
??? note "A reverse-tunnel client logs Invalid status code: 524 every ~2 minutes"
Expected churn when manual.connectivity=reverse_tunnel is paired with
manual.participant_exposure=cloudflare_tunnel. The wstunnel client keeps
one "await" connection open for the server to hand new tunnels to, and the
server holds that HTTP upgrade pending until it's needed. Cloudflare's
proxy gives an origin 120 seconds to answer, then kills the pending
request with its own 524 status — so the client logs the error and
re-dials within a second, forever.
Established desktop sessions ride their own, fully-upgraded connections
and are **not** affected (measured: sessions survive many churn cycles and
400+ seconds of idle). The cost is a ~1-second window every ~2 minutes in
which a brand-new participant connection stalls until the client
re-attaches, plus this log noise. There is no fix on the free Cloudflare
plan (the 120s timeout is configurable only for Enterprise), and wstunnel
upstream documents CDN buffering as incompatible with its protocol —
treat the message as noise unless it stops being periodic.
??? note "Sessions on a laptop client drop when the lid closes (or randomly)" A bring-your-own client on a laptop dies with the machine: Windows sleep/Modern Standby suspends the WSL2 VM, which freezes Docker, the tunnel, and KasmVNC in one stroke — every participant session drops and the allocator eventually marks the client unhealthy. Wi-Fi power saving can do the same without sleep.
Before serving real participants from a laptop, disable sleep while
plugged in (Windows: `powercfg /change standby-timeout-ac 0`) and turn
off Wi-Fi power management. The viewer auto-reconnects when the machine
comes back, but anything running in the session meanwhile was frozen.
??? note "A mesh-overlay client looks healthy but is unreachable"
Re-registering a mesh_overlay client mints a new Tailscale node. MagicDNS suffixes the name (-1, -2, …) because the old node is offline but still holds the original name, so the allocator's stored hostname points at a machine that no longer answers — while the client's own logs look perfectly fine.
Delete the stale machine in the Tailscale admin console, then re-register. Use `lablink client reset-overlay` when you deliberately want a fresh node identity; note it requires the `lablink-client` container to be gone first, since Docker won't remove an attached volume.
??? note "Allocator compose stack won't come up"
bash docker compose ps docker compose logs -f allocator curl -i http://localhost/api/health
`lablink doctor` under the `manual` provider checks the Docker side for you.
Symptoms specific to running your own research image instead of the default — see Adapting for Your Software.
??? note "The client container never starts on a fresh VM"
bash ssh -i ~/lablink-client-key.pem ubuntu@<client-vm-ip> sudo docker ps -a # did it exit, or was it never created? sudo docker logs <client-container>
An image that exits immediately is almost always a missing dependency or a
broken entrypoint — reproduce it locally with the same command before
blaming the deployment:
```bash
docker run --rm -it your-image:tag bash
```
If the container was never created, the failure is earlier than your image:
read `sudo cat /var/log/cloud-init-output.log` for the pull, and confirm the
tag in `machine.image` exists and is publicly readable (private registries
need credentials baked into the AMI).
??? note "machine.repository isn't cloned onto the desktop"
Check the URL, that the repository is public (LabLink clones over HTTPS with
no credentials), and that the VM has outbound network access. The clone runs
at startup, so its output is in the client container logs.
??? note "Your software isn't found inside the session"
The desktop session doesn't inherit the Dockerfile's build-time shell. Set
PATH (and any activation, e.g. a virtualenv) via ENV in the image rather
than in ~/.bashrc or a RUN line, then verify from inside the running
container:
```bash
sudo docker exec <client-container> which your-software
```
??? note "Cannot connect to PostgreSQL"
psycopg2.OperationalError: could not connect to server. The container's start.sh starts Postgres and blocks on pg_isready before Flask boots, so a refused connection normally means the cluster died rather than that it was never started.
```bash
sudo docker exec <container> pg_isready -U lablink
sudo docker exec <container> psql -U lablink -d lablink_db -c "SELECT 1;"
sudo docker exec <container> tail -f /var/log/postgresql/postgresql-*-main.log
```
To restart the cluster in place:
```bash
sudo docker exec -it <container> pg_ctlcluster 17 main restart
```
??? note "Running out of database connections"
The allocator reports its own connection usage — check /admin (a line under the title) or hit the endpoint directly:
```bash
curl -u admin:<password> https://<allocator-domain>/api/health/connections
```
At the critical level, new connections can be refused, which fails VM registration and admin actions. Raise Postgres `max_connections` and `LABLINK_DB_POOL_MAX_SIZE` together, and check for connections stuck idle in transaction.
??? note "Container runs but Flask doesn't start"
bash sudo docker logs <container> # port in use, import error, bad config? sudo docker exec <container> cat /config/config.yaml sudo netstat -tulpn | grep 5000 sudo docker restart <container>
A config that fails schema validation stops the app before it serves anything — the log names the offending key.
# Host health
df -h; free -h; top
sudo docker ps -a
sudo docker stats
sudo netstat -tulpn
# Allocator on an AWS host (manual-provider host port is 80)
curl localhost:5000/api/health
sudo docker logs -f <allocator-container>
# Database
sudo docker exec <container> pg_isready -U lablink
sudo docker exec <container> psql -U lablink -d lablink_db -c "SELECT hostname, status, inuse FROM vms;"
# AWS reachability from inside the container
sudo docker exec <container> aws sts get-caller-identity-
lablink doctorrun, output read - Container running (
docker ps) and answering at the host port for your provider - Public URL reaches port 80 or 443 as configured; SSH reaches port 22
- SSH key is mode 600
- AWS credentials resolve (
aws sts get-caller-identity) - OpenTofu state not locked, S3 bucket exists
- Config passes validation — no unknown keys
Still stuck? Search existing issues, then open a new one with the lablink doctor output, the relevant container logs, what you expected, and what you tried.
- Prerequisites — what each deployment path actually needs
- Bring-Your-Own Clients — the manual provider end to end
- Configuration — every config key and its valid values
- Deployment — deployment and teardown
- Database — schema and direct access
- Security & Access — auth, SSH, and SSL posture