Persistent Linux "Jails" on TrueNAS CE (formerly TrueNAS Scale) to install software (K3s, Docker, Portainer, Podman, etc.) with full access to all files via bind mounts.
Jip-Hop previously announced that his repo was no longer going to be maintained with the final release being v2.1.1. It was last tested to work with TrueNAS SCALE v24.10.0. However, with the release of TrueNAS CE v25.10 (Goldeye), support for older NVIDIA GPUs broke due to iXsystems migrating from the NVIDIA Proprietary driver to the NVIDIA Open Kernel driver. The reason for this migration was to support NVIDIA Blackwell (RTX 50 Series) GPUs and newer. A kind patron who goes by zzzhouuu created a TrueNAS NVIDIA Driver Build repo to help maintain support for legacy NVIDIA GPUs. Additional information and instructions on zzzhouuu's site, TrueNAS GPU Drivers.
In addition to legacy NVIDIA GPUs breaking, the NVIDIA Open Kernel driver also broke the bind mounts of the NVIDIA libraries for newer NVIDIA GPUs as well. This is because the path in which the libraries are stored have changed; they are now located directly in the base System Extensions directory, /usr/share/truenas/sysext-extensions.
This fork of Jip-Hop's original Jailmaker repo was created to help continue his efforts to help restore support for NVIDIA GPUs:
- Support for newer NVIDIA GPUs using the Open Kernel driver
- Support for legacy NVIDIA GPUs using the Proprietary driver
Why continue the effort?
- iXsystems announced that they have decided to drop Incus support in TrueNAS 26, and will be reverting back to libvirt.
- Running Docker apps in a self-contained environment could make things easier with managing containerized apps, especially if you need to install OS packages, such as cron or logrotate, that can't be installed directly on the TrueNAS host.
There won't be a lot of effort put into Jip-Hop's brilliant Jailmaker script; however, minimal effort will be given as time permits to help address issues that may arise whenever something stops functioning after upgrading to a new version of TrueNAS CE.
USING THIS SCRIPT IS AT YOUR OWN RISK! IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
TrueNAS CE can create persistent Linux Jails with systemd-nspawn. This script helps with the following:
- Setting up the Jail so it won't be lost when you upgrade TrueNAS CE
- Choosing a distro (Debian 12 strongly recommended, but Ubuntu, Arch Linux or Rocky Linux seem good choices too)
- Will create a ZFS dataset for each Jail if the
jailmakerdirectory is a dataset (easy snapshotting) - OPTIONAL: Configuring the Jail so you can run Docker inside it
- OPTIONAL: GPU passthrough (Intel, NVIDIA with drivers bind mounted from the host, AMD reportedly works too)
- Starting the Jail with your config applied
Beginning with 24.04 (Dragonfish), TrueNAS CE (previously TrueNAS Scale) officially includes the systemd-nspawn containerization program in the base system. Technically there's nothing to install. You only need the jlmkr.py script file in the right place. Instructions with screenshots are provided on the TrueNAS website. Start by creating a new dataset called jailmaker with the default settings (from TrueNAS web interface). Then login as the root user and download jlmkr.py.
cd /mnt/mypool/jailmaker
curl --location --remote-name https://raw.githubusercontent.com/Jip-Hop/jailmaker/main/jlmkr.py
chmod +x jlmkr.pyThe jlmkr.py script (and the Jails + config it creates) are now stored on the jailmaker dataset and will survive updates of TrueNAS CE. If the automatically created jails directory is also a ZFS dataset (which is true for new users), then the jlmkr.py script will automatically create a new dataset for every Jail created. This allows you to create a snapshot for each individual Jails For legacy users (where the jails directory is not a dataset) each Jail will be stored in a plain directory.
Optionally you may create a shell alias for the currently logged in (admin) user to conveniently run jlmkr.py without having to change into the jailmaker directory or specify the full absolute path. I suggest to create the jlmkr alias like this:
echo "alias jlmkr=\"sudo -E '/mnt/mypool/jailmaker/jlmkr.py'\"" >> ~/.bashrcPlease replace /mnt/mypool/jailmaker/ with the actual path to where you stored jlmkr.py. If you're using zsh instead of Bash, then you should replace .bashrc in the command above with .zshrc. If you've created the alias, you may use it instead of ./jlmkr.py.
The alias will be available the next time you load the shell, but to use the alias immediately you can source ~/.bashrc or source ~/.zshrc, as appropriate.
To view the list of available commands, you may run the following command
./jlmkr.py helpor
./jlmkr.py --helpTo view the commands with more details, add the --verbose (or -v) flag.
./jlmkr.py help --verboseWhen creating a Jail, you'll be given the opportunity to specify the Distro and Release to use. You may also view a list of available LXC images in advance by using the following command
./jlmkr.py imagesJailmaker also allows you to create a Jail using cloud-init. To view a list of cloud-init variants, add the --cloud flag to the command.
./jlmkr.py images --cloudCreating a Jail with the default settings is as simple as:
./jlmkr.py create --start myjailBy default, the hostname in the Jail will default to the Jail name; however, you may set a custom internal hostname.
./jlmkr.py create --start --hostname myhostname myjailYou may also specify a path to a config template, for a quick and consistent Jail creation process. The templates folder contain sample templates that can be used as a starting point.
./jlmkr.py create --start --config /path/to/config/template myjailOr you can override the default config by using flags.
./jlmkr.py create --start --distro ubuntu --release jammy myjail --bind-ro='/mnt'Or you can override the default config with the same flags but install the cloud-init variant.
./jlmkr.py create --start --cloud --distro ubuntu --release jammy myjail --bind-ro='/mnt'Or you can override more options in the config template using other flags as well. See ./jlmkr.py create --help for all available options. Anything passed after the Jail name will be passed to systemd-nspawn when starting the Jail. See the systemd-nspawn manual for available options, specifically Mount Options and Networking Options are frequently used.
./jlmkr.py create --start --config /path/to/config/template --distro ubuntu --release jammy myjail --bind-ro='/mnt'Omitting the Jail name will start an interactive session that will allow you to configure the Jail manually. You'll be presented with questions, which will guide you through the process.
./jlmkr.py createAfter answering the questions, a new Jail will be created and will start automatically if you answered "Yes" to the question about starting the Jail immediately.
The priority of the options when creating a Jail are handled in the following order.
| Stage | Behavior | Precedence |
|---|---|---|
| Default Config | An immutable config that's baked directly into the Jailmaker script. | Lowest |
| Config Template | User-provided config template that can be specified using the --config (or -c) flag. The options in the template will override values in the default config; however, the default config can likewise be overriden using CLI flags (or interactive inputs) when creating a new Jail. |
|
| CLI Flags | Flags that can be set when executing the jlmkr.py create command. The flags can be used to override both the default config and whatever is set in the config template. See jlmkr.py create --help for all available flags. |
|
| Interactive Inputs | By ommitting the Jail name when executing the jlmkr.py create command, an interactive session will begin. Any value entered in any of the inputs will have the highest priority and will override all CLI flags, options in the config template, and options in the default config. |
Highest |
# Call startup using the absolute path to jlmkr.py
/mnt/mypool/jailmaker/jlmkr.py startupIn order to start Jails automatically after TrueNAS boots, run /mnt/mypool/jailmaker/jlmkr.py startup as Post Init Script with Type Command from the TrueNAS web interface. This will start all the Jails with startup=1 in the config file.
If you need NVIDIA GPU Passthrough support on TrueNAS CE Goldeye or newer, use this command instead to automatically start the GPU in Persistence Mode:
# Start NVIDIA GPU in persistence mode, then call startup using the absolute path to jlmkr.py
nvidia-persistenced && /mnt/mypool/jailmaker/jlmkr.py startupThis will help avoid repetitively initializing the GPU whenever it's needed. This is useful for apps like Beszel. If you have no plans to run apps that need to constantly query the GPU, but only plan to use the GPU on demand for transcoding only, such as for Jellyfin, Plex, Immich, etc., then you can omit the nvidia-persistenced command from the Post Init Script.
./jlmkr.py start myjailSee list of Jails (including running, startup state, GPU passthrough, distro, and IP).
./jlmkr.py listYou may want to execute a command inside a Jail, for example manually from the TrueNAS shell, a shell script or a CRON job. The example below executes the env command inside the Jail.
./jlmkr.py exec myjail envThis example executes bash inside the Jail with a command as additional argument.
./jlmkr.py exec myjail bash -c 'echo test; echo $RANDOM;'./jlmkr.py edit myjailOnce you've created a Jail, it will exist in a directory inside the jails dir next to jlmkr.py. For example /mnt/mypool/jailmaker/jails/myjail if you've named your Jail myjail. You may edit the Jail configuration file using the ./jlmkr.py edit myjail command. This opens the config file in your favorite editor, as determined by following Debian's guidelines on the matter. You'll have to stop the Jail and start it again with jlmkr for these changes to take effect.
Delete a Jail and remove its files (requires confirmation).
./jlmkr.py remove myjailRename a Jail, its dataset/directory, and its internal hostname (requires confirmation).
./jlmkr.py rename oldjailname newjailname./jlmkr.py stop myjail./jlmkr.py restart myjailSwitch into the Jail's shell.
./jlmkr.py shell myjail./jlmkr.py status myjailView a Jail's logs.
./jlmkr.py log myjailBeginning with 25.10 (Goldeye), TrueNAS CE replaced the NVIDIA Proprietary driver with Open Kernel driver. You may replace the Open Kernel driver with the Proprietary driver (thanks to zzzhouuu's manually compiled GPU driver extensions) using the following command, which restore functionality to legacy NVIDIA GPUs, such as Pascal, Maxwell, Volta, etc.
./jlmkr.py nvidia --action installExecute the following command to restore the Open Kernel driver.
./jlmkr.py nvidia --action uninstallView the current installation status, version installed, and persistence mode status
./jlmkr.py nvidia --action statusView the verbose details of the driver that's currently installed
./jlmkr.py nvidia --action versionNOTE: The commands above will have no effect on TrueNAS version older than 25.10. When gpu_passthrough_nvidia is set, Jailmaker will automatically install the NVIDIA Proprietary driver for systems having a legacy NVIDIA GPU. Jailmaker will base this on the compute capability of the NVIDIA GPU; if the value must be lower than 7.5. The NVIDIA Proprietary driver will only work for GPUs having an architecture older than Blackwell (i.e. Turing, Ada Lovelace, etc.). For systems running a Blackwell-based GPUs or newer, the NVIDIA Open Kernel driver must be used.
Expert users may use the following additional commands to manage Jails directly: machinectl, systemd-nspawn, systemd-run, systemctl and journalctl. The jlmkr script uses these commands under the hood and implements a subset of their functions. If you use them directly you will bypass any safety checks or configuration done by jlmkr and not everything will work in the context of TrueNAS CE.
By default, the root user in the Jail with uid 0 is mapped to the host's uid 0. This has obvious security implications. If this is not acceptable to you, you may lock down the Jails by limiting capabilities and/or using user namespacing or use a VM instead.
Secure computing mode (seccomp) is a Linux kernel feature that
restricts programs from making unauthorized system calls. This means that when seccomp is enabled there can be times when a process that runs inside a Jail will be killed with the error "Operation not permitted." In order to find out which syscall needs to be added to the --system-call-filter= configuration you can use strace.
For example:
# /usr/bin/intel_gpu_top
Failed to initialize PMU! (Operation not permitted)
# strace /usr/bin/intel_gpu_top 2>&1 |grep Operation\ not\ permitted
perf_event_open({type=0x10 /* PERF_TYPE_??? */, size=PERF_ATTR_SIZE_VER7, config=0x100002, sample_period=0, sample_type=0, read_format=PERF_FORMAT_TOTAL_TIME_ENABLED|PERF_FORMAT_GROUP, precise_ip=0 /* arbitrary skid */, use_clockid=1, ...}, -1, 0, -1, 0) = -1 EPERM (Operation not permitted)
write(2, "Failed to initialize PMU! (Opera"..., 52Failed to initialize PMU! (Operation not permitted)
The syscall that needs to be added to the --system-call-filter option in the jailmaker config in this case would be perf_event_open. You may need to run strace multiple times.
The seccomp feature is important for security, but as a last resort can be disabled by setting seccomp=0 in the Jail config.
By default, a Jail will use the same networking namespace, with access to all (physical) interfaces the TrueNAS host has access to. No further setup is required. You may download and install additional packages inside the Jail. Note that some ports are already occupied by TrueNAS CE (e.g. 443 for the web interface), so your Jail can't listen on these ports.
Depending on the service this may be OK. For example Home Assistant will bind to port 8123, leaving the 80 and 443 ports free from clashes for the TrueNAS web interface. You can then either connect to the service on 8123, or use a reverse proxy such as Traefik.
But clashes may happen if you want some services (e.g. Traefik) inside the Jail to listen on port 443. To work around this issue when using host networking, you may disable DHCP and add several static IP addresses (aliases) through the TrueNAS web interface. If you set up the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the Jail to listen on. Or another alternative would be to set up two separate VLANS: one for TrueNAS and a separate VLAN for your Jails and would help avoid IP and traffic conflicts.
See the networking docs for more advanced options (bridge and Macvlan networking).
Using the Docker config template is recommended if you want to run Docker inside the Jail. You may of course manually install Docker inside a Jail. But keep in mind that you need to add --system-call-filter='add_key keyctl bpf' (or disable seccomp filtering). It is not recommended to use host networking for a Jail in which you run Docker. Docker needs to manage iptables rules, which it can safely do in its own networking namespace (when using bridge or Macvlan networking for the Jail).
Additional documentation can be found in the docs directory (contributions are welcome!).
TODO: write comparison between systemd-nspawn (without jailmaker), LXC, VMs, Docker (on the host).
The rootfs image jlmkr.py downloads comes from the Linux Containers Image server. These images are made for LXC. We can use them with systemd-nspawn too, although not all of them work properly. For example, the alpine image doesn't work well. If you stick with common systemd based distros (Debian, Ubuntu, Arch Linux, etc.) you should be fine.
When in need of help or when you think you've found a bug in Jailmaker, please start with reading this.