Delamain

Live

Delamain is my homemade rackmount AI workstation, running a private AI stack of my own. It's named after the AI in Cyberpunk 2077 that runs a taxi company and decides where every ride goes, which is a fair description of its job: I open a chat from my phone or laptop anywhere in the world, a small router model decides whether the request needs a quick answer or a coding model, and the whole conversation stays on hardware I own. Nothing is exposed to the public internet.

The hardware

I built Delamain by hand around an older dual-Xeon workstation board, giving it a second life as a home for local AI. Dual-socket platforms like this one bring plenty of cores, memory capacity, and PCIe lanes to the table, and the stack is laid out to use all of it.

  • 2× Intel Xeon E5-2690 v4: 28 cores and 56 threads across two sockets, with 128 GB of DDR4 that holds most of the coding model.
  • AMD Radeon RX 6700 XT (12 GB): runs the heavy coding model.
  • AMD Radeon RX 570 (4 GB): keeps the router and the fast chat model loaded around the clock.

Every model runs on llama.cpp with its Vulkan backend, which drives both generations of AMD card, including the older RX 570 that ROCm no longer supports.

The models

  • Qwen3.6-35B-A3B: the main coding model. It's a mixture-of-experts model, so its attention layers and context live on the 6700 XT while most of its experts sit in system RAM on the CPU closest to that card. That's how a 35-billion-parameter model fits a 12 GB GPU and still generates around 17 tokens per second.
  • Liquid AI LFM2.5 (2.6B): the fast chat model for everyday questions, answering at around 77 tokens per second from the RX 570.
  • Gemma 4 (12B): a lighter coding model I can switch to by hand. It shares the 6700 XT, so selecting it swaps the main coder out.
  • Arch-Router (1.5B): a tiny model whose only job is to read each message and pick which of the others should answer it.

How a request flows

I chat through Pithagoras, a self-hosted web UI for the pi coding agent, and pick a path for each conversation. The direct path talks straight to the coder or the chat model. The auto path goes through Plano, an AI gateway that asks Arch-Router about every message: programming work goes to Qwen, and small talk, facts, and general questions go to the fast chat model. I tuned the route descriptions against a labelled set of test prompts until the router sorted all of them correctly, up from an original wording that sent git and shell questions to the chat model. Behind it all, llama-swap loads and unloads the big models on the 6700 XT on demand, and everything runs as locked-down systemd services.

Chasing down a faulty GPU

The first time everything came up, every model on the 6700 XT produced garbage, even the 1.5B router, while the same build gave correct answers on the CPU and the RX 570. A Vulkan memory test found the cause: bit errors on every pass at the card's full memory clock, always on the same data bit. Pinned down to 675 MHz, the card passed 3.6 TB of testing with zero errors and the models answer correctly. A boot-time service now applies that clock before the model server is allowed to start, so the coder never runs at the unsafe speed. Generation speed didn't change, because the experts in system RAM are the real bottleneck.

Locked down by default

  • Tailscale: Delamain itself never joins my tailnet. A dedicated Tailscale container does, serving only HTTPS to the web UI, with Funnel off and an access policy that lets only my own devices reach it.
  • Default-drop firewall: an nftables policy drops all inbound traffic from the local network except SSH.
  • Least privilege: the model servers listen on localhost only and run as an unprivileged user with systemd sandboxing, and the agent's container is deliberately denied the Docker socket so it can't take over the host.

What's next

  1. A healthy GPU: replacing the faulty 6700 XT.
  2. Tuning: retuning the model split for the new card's VRAM, potentially with larger models.
  3. Laya: integrating Laya, an open-weight decision model, as a "System One" grader that decides how hard the coding model should think and checks the fast chat model's answers, handing weak ones to the coder.