Arkitekt Deployment Strategies

[jhnnsrs]

This is a place to discuss deployment strategies for an arkitekt server and options beyond Konstruktor and the new blok cli builds. It would be super nice to have something like the super cool forklift project to easily deploy an arkitekt server on hardware and to hide the microservice architecture a bit away.

[ethanjli]

Based on our discussion in openUC2/imswitch-os#7, here's my current (very limited) understanding of the current conditions of the project which have implications for deployment strategies - please correct any mistakes I make so I can understand the situation better 🙂

1. System components are (micro)services organized in a polyrepo structure where each service has its own Git repo, its own Docker Container image, and its own Docker Compose app. You do not intend to merge services into a smaller collection of multi-service repos (or even into a monorepo).
2. Currently your priority is making it easy+simple to deploy Arkitekt to a single container host (i.e. computer or cloud VM)
3. Services can evolve independently of each other. But they have some constraints (e.g. one service depends on an API provided by some other service), so that they can't be mixed-and-matched in totally arbitrary ways (e.g. one service being deployed in the absence of an appropriate version of an API provided by some other service).
4. Services may have some configs (set as environment variables) which are shared across most/many deployments of Arkitekt, rather than being deployment-specific. For example, feature flags.
5. There are various deployment-specific config settings shared among services. Some of these are secrets, while others are not. The services are written to load non-secret config settings from environment variables.
6. It would be fine if secrets are generated before deployment and then passed to the services (so that the overall system does not require each service to generate its own secrets and then share the secrets with other services while the services are running).
7. The services can be (re)written to load secrets from files.

Managing shared configs & secrets

Under the assumptions listed above, it might make sense to set non-secret config variables in files which are passed to Docker Compose, e.g.:

• as environment variable files which are loaded by Docker Compose via its env_file feature; or
• as YAML or JSON config files which are mounted into the service's container in order to be loaded+parsed by each service.

For services on the same container host, they can just share access to the same file(s). For services on different container hosts, either you can use Docker Compose (with or without Forklift) and just have something which mounts remotely-stored configs into the local filesystem (e.g. via rclone mount or something more sophisticated); or you can use Docker Swarm with Docker Configs; or you can use equivalent/alternative functionalities from Nomad or Kubernetes (but note that such systems don't natively support Docker Compose files).

Similarly, it might make sense to save secrets in files which are passed to Docker Compose's secrets feature , and then loaded+parsed by each service. Then you would share the secrets either via a remote/shared filesystem (potentially harder to make secure) or via a more complicated secrets manager, e.g. via something like age and/or sops. On a cluster of multiple container hosts, something like Docker Swarm secrets might make more sense.

The secret & non-secret files could be saved within some predictable path, e.g. /var/lib/arkitekt.

I've probably missed something in my assessment of Arkitekt's architecture and deployment requirements which might make these baseline recommendations not quite appropriate or not quite feasible - if so, please let me know so that I can adjust accordingly! If you can describe a few services and the specifics configs and secrets they need to share (as a microcosm of the overall Arkitekt system), we could also talk about solving the problem for those specific services so that we can have a more concrete discussion.

The potential role of Forklift

Forklift is primarily aimed at improving the user/developer experience related to items 1-4 above, i.e. integrating and version-controlling combinations of versions of disparate software services, and their associated general (i.e. not specific to a single deployment) configs. Items 5-7 above are intended to be solved with orthogonal systems (e.g. functionality already provided via Docker Compose) under the assumption that they can be made to interact/combine well with Forklift (even if Forklift's design must be adjusted to make that true).

Note that Forklift is designed to make system state changes (roughly) atomic at the level of a single container host (i.e. computer or VM), not a cluster of container hosts. Note also that Forklift currently deploys Compose apps via Docker Compose rather than Docker Swarm.

One way Forklift might provide some benefits to Arkitekt could be via the following architecture:

1. The Git repo for each (micro)service exports a Forklift package which is designed to be combined with other Forklift packages following certain kinds of constraints (e.g. API dependencies).
2. You make a new Git repo which integrates all the Forklift packages into a complete deployment of all services together on a single container host.

Depending on project requirements, other architectures may also make sense - with their own trade-offs.

[ethanjli]

In openUC2/imswitch-os#7 (comment), @beniroquai proposed having a ready-to-deploy OS image for running Arkitekt on a single computer (or VM). I think that would be great as a way of making it easier to try out Arkitekt (e.g. in a cloud VM). The use (and usefulness) of such an OS image could be somewhat independent of exactly how services are baked into that OS image (e.g. with Forklift or not, with one secrets-management strategy or another), so I'm making a separate comment thread for this topic under the overall discussion item of "Arkitekt deployment strategies".

If I had to be the person building, maintaining, and sysadmin'ing such an OS image, my initial attempt would be based on something like Fedora CoreOS using the Bootable Containers approach. That approach is how I currently build a custom Fedora Kinoite-based OS image for my laptop via GitHub Actions, and how I make automatic OS updates safe and undo-able on my laptop. The resulting OS images can be hosted as OCI images for free on any OCI image registry (such as the GitHub Container Registry, and probably also Docker Hub if it doesn't have size limits for container images), and they can also be exported into various formats (which would need to be hosted in some other way).

[jhnnsrs]

Thanks so much for the thoughtful summary—this really captures the current state of Arkitekt better than I’ve managed to explain it myself in the past 😅 I'd love to offer a few clarifications and background notes. A lot of this has grown organically over time, so criticism is very welcome!

1. Service Modularity & Polyrepo Structure

Yes, the intention from the beginning was to keep services independently deployable and composable. Most services are maintained and (supposedly) tested in isolation, and they can be used on their own. For example

• You can deploy just mikro (microscopy storage) without rekuest (app assignment) or fluss (workflow orchestration).
• elektro can run standalone for electrophysiological data.
• You can integrate a service into your own stack using its public API and JWT-based auth.

While the polyrepo approach has its downsides (and I’m not in love with the sprawl, and it must be overwhelming in the start), merging everything into a monorepo would be difficult given how diverse the dependencies are—e.g., omero-ark vs. kraph. Still, I’m open to revisiting this if a compelling case arises.

2. Deployment Focus: Local First, Federated Later

A smooth, local "try-it-out" experience is still the top priority—especially for new users. That said, it currently requires more effort than I'd like. Distributed multi-machine deployments were part of the way-back vision, and the stack is really well-suited to it, but in practice, most use cases don’t yet demand that level of scale. Another idea is pushing it to "federated arkitekt" but thats still in the pipeline. And i truly believe local-single-machine-first.

3. Minimal Inter-Service Dependencies

Services are deliberately designed to be self-contained. They rely only on common backend infrastructure:

• PostgreSQL for data
• S3/MinIO for blob storage
• Redis (optionally) for queues or caching

There is no direct service-to-service API communication. Instead, apps (which act as intermediaries) consume and integrate data from services externally. For example, an app might associate a microscopy image from mikro with a knowledge graph in kraph.

This decoupling gives a lot of flexibility, though it does push complexity into the app layer—something we're always balancing.

4. Just a few words on Auth: JWT-Based Auth

All services rely on JWTs for authentication. They don’t need to know where a token came from—just that it’s valid. Currently, tokens are issued by the lok service and provide some info on which "app" was granted the rights and which scopes the app has (i.e can it delete data).

Importantly, lok can also be hosted globally, like at https://go.arkitekt.live, which opens up the possibility of federated Arkitekt deployments. In that model, services across machines or even institutions connect via overlay networks (e.g., Tailscale or Zerotier), while sharing a common identity/auth backend.

5. Current Deployment Approach: Shared Config & Compose Generation

Deployments currently rely on a autogenerated shared directory that contains:

• Configuration files
• Secrets (mainly only JWT signing keys)
• docker-compose.yml file

This setup is powered by two tools:

• Konstruktor – a desktop app that generates standard deployments (currently supports the legacy Arkitekt stack only)
• Blok – a CLI tool meant to allow flexible service selection and config. It works, but I’ll admit its design is rough and could use a proper rethink.

Why did I design it that way?
So that we dont spin up infra (like Postgres or MinIO) per service. Instead, we:

1. Register infrastructure needs centrally (e.g., DBs, buckets, queues)
2. Generate Compose files and secrets based on those requirements
3. Store everything in a common structure

6. Where Forklift Could Really Help

Honestly, Forklift looks like a great fit for all things packaging, and some modular setup for a cloud or local vm. I think your idea of giving users a prebuild ubuntu / fedora image would be amazing <3

Thanks again for diving in so deeply—this kind of feedback and perspective is super valuable. 🙏

[beniroquai]

One maybe very obvious side-comment would be to differentiate between host and client-side OS images.
Right now I'm confronted with rebuilding the setup of robots and microscopes that interact with arkitekt - again. For these it would be great to have one (?) Arkitekt client os, where the actual service (e.g. robot API adapter...) can be swapped easily (i.e. with an APP store infracstructure?).
On the other hand, an Arkitekt server OS would be cool to quickly spawn a local (?) instance of all the tools necessary to connect multiple of such clients into one workflow. Not usre if that matters? The client OS would ideally run on low resource hardware like a pi, the server on more powerful machines.

Anyway, great stuff of which I understood - not everything :D

[jhnnsrs]

I think both ideas are very promising!

I’ve been thinking a bit more about the ClientOS concept. Currently, everything related to an Arkitekt app can be bundled into a Docker container, since the apps primarily just communicate with APIs. This is also how we handle plugins internally—each is a container that can run on any Docker host with access to the Arkitekt server. The app store, in this context, functions as a central repository of all installable apps and provides an interface to deploy them on connected Docker hosts via deployer apps, which are essentially wrappers around a local Docker client. (You can check out arkitektio/deployer for more implementation details.)

Based on this, I could imagine a potential Arkitekt-based "swapping" mechanism that works like this:

• A Docker engine runs on the ClientOS.
• A deployer app is installed on each device.
• This setup allows for deploying new apps directly onto the local Docker engine.

✅ Potential benefits

• Hot-swapping staging apps becomes significantly easier.
• We could leverage the Arkitekt app store, which I'm planning to invest a lot of time into going forward 😄
• Remote management: With deployers in place, apps could be deployed, updated, or restarted remotely—very helpful for unattended setups.

❌ A few concerns to consider

• Performance: Running Docker on every type of embedded device might introduce unnecessary overhead. While something like a Raspberry Pi can handle this reasonably well, not all devices are as capable. I’ve been exploring rewriting parts of the Arkitekt client SDK in Rust to improve performance in resource-constrained environments. A more traditional "OS-style" deployment would likely be more efficient in such cases.
• SD Card Simplicity: There’s a certain elegance in having a prebuilt image you can flash onto an SD card. This approach is simple, especially for initial setup. Whether or not it’s the better solution depends on how frequently new devices are introduced and need configuration—but since we're dealing with hardware, that might not be all that often.
• Storage footprint: Docker layers and images can accumulate over time and may pose an issue on devices with limited storage.
• Hardware access: Direct access to hardware from within containers (e.g., GPIO, cameras) often requires special setup or elevated privileges, which could complicate things on embedded platforms.

[ethanjli]

Here are my follow-up comments/questions on a server-side OS image:

• Polyrepo: yeah, this makes sense to me! I'm generally a polyrepo person, too.

  • I think a small-but-useful improvement could be to make https://github.com/arkitektio/.github/tree/main/profile/README.md (or some other docs page) include links to the repositories it describes, and to fix missing descriptions for the repo list at https://github.com/orgs/arkitektio/repositories.
  • One thing which has been challenging for me is that most repos have fun names like "fluss" or "rath" or "elektro", as opposed to semantic names like "deployer" or "init-container - and I think fun names don't really scale well for conveying meaning, at least not when there are >>10 repos with fun names.
  • If you're able to merge even a few repos together into logical groupings (e.g. a ts-sdk monorepo only consisting of all the codebases which currently have -ts-suffixed repo names, or a core-sdk-ts monorepo only consisting of the TypeScript SDKs of all the core libraries, or a fakts monorepo which just has both the Python and Typescript SDKs since I assume they evolve together), I think that might help a lot with making https://github.com/arkitektio/.github/tree/main/profile/README.md easier to maintain and keep up-to-date, and also with making the "fun repo names" a bit less overwhelming.

• Local deployment first: got it, I agree with this priority.

• Service dependencies: are you bringing up Docker containers for PostgreSQL, MinIO, and (optionally) Redis as part of the single-machine Arkitekt deployment? Or are you instead using external hosted services to provide those for your services? If it's the former, we can use Forklift to statically enforce such dependencies (e.g. so that several of your services cannot be deployed without a declaration of a PostgreSQL deployment to support them).

• Auth: ok, that's helpful to know!

• Shared config & compose generation: from your description, it sounds like it'll probably be feasible to separate Docker Compose app declarations from configs/secrets, so that we have a Git repo of Docker Compose yml files (which then would not be auto-generated) and any static config files, and then a separate directory of deployment-specific configs and secrets. If different deployments will require different combinations of Docker Compose apps, that is something Forklift can support (though note that we'd be using Forklift in a slightly different way than how the PlanktoScope OS uses it).

Here are my follow-up comments on client-side OS images:

• App store: one of my goals for Forklift is for it to be infrastructural plumbing to help people build their own federated plugin/"app store" ecosystems based on Git repos - so it might be interesting to eventually brainstorm how Forklift might be useful as lower-level plumbing in Arkitekt's app store/Blok/Konstruktor/etc. Is there currently an up-to-date design document (or other document) which describes the overall vision/objectives/architecture for Arkitekt's app store?
• Add-on/plugin/app "deployer": in Forklift's design there's no inherent distinction between a Docker Compose app for a core service vs. a Docker Compose app for a plugin/addon running on the same computer; this sounds potentially compatible with Arkitekt's container-based approach to apps/plugins and Arkitekt's goal of swapping an existing app with a substitute. However, in Forklift the deployment of each app must be statically specified/persisted in a filesystem (preferably a Git repo) - Forklift's unidirectional flow of information is: static configs on filesystem -> live state of Docker Engine, and remote management of deployments follows the GitOps model (in which changes to a deployment are pushed to a Git repo as file modifications, and then a program/person on the remote computer would automatically or manually pull and apply the changes). This might or might not be a subtly different approach from arkitektio/deployer - I didn't understand what I was looking at when I skimmed that repo's app.py, but it sounds like you might be envisioning a somewhat different flow of information (e.g. dynamically getting information from a GraphQL API and/or Docker Engine's API) compared to Forklift.
• SD card simplicity: yes, in the PlanktoScope project it's been very helpful to have a ready-to-use SD card image. I think the guiding consideration here should be what we want the UX to be like for provisioning new client devices into the cluster of connected machines.
• Maybe not using Docker: Forklift can be used to manage the deployments of static binaries without the Docker Engine (but note that currently Forklift's binary is big because the binary-building process statically links the Docker Compose library into the Forklift binary, whether or not you want to do anything with Docker Compose). I do this in the PlanktoScope OS for deploying a low-level tool (PlanktoScope/machine-name) used by a custom systemd service which needs to run before Docker. The binaries can be hosted online either as a direct HTTP/HTTPS download (e.g. from GitHub Releases), as files in HTTP/HTTPS-downloadable .tar/.tar.gz/etc. archives, or in OCI container images (which for this purpose is basically just a .tar archive with a built-in mechanism for automatic platform/CPU architecture selection, and I download OCI container images without using Docker). Even so, Forklift is targeted for embedded Linux, not for further-constrained platforms such as Zephyr.
• Docker's storage footprint: you can at least add a regularly-scheduled systemd service to run docker image prune to negate any accumulation of unnecessary image layers over time.
• Hardware access through Docker: I haven't containerized any hardware-specific apps yet (this is on my wishlist for PlanktoScope OS), but from my understanding it may be sufficient to mount the appropriate filesystem paths (e.g. in /dev, /sys, or /run/udev), set the appropriate kernel capabilities, and/or to run with full privileges. In this case Docker is being used more for its non-security-related features, which may or may not be worth the extra complexity from using Docker. The preliminary opinion of the incoming maintainer of PlanktoScope OS is that for deploying hardware-bound programs, Docker's benefits aren't worth Docker's complexity, but that maybe a simpler container system could be acceptable.