Introduction

QuakeKube

QuakeKube is a Kubernetes-ified version of Quake 3. A controller runs dedicated servers for QuakeServer objects, along with a master server that lists them, and a web app creates servers and plays them in the browser with ioquake3 built for the web with Emscripten. It’s installed with a Helm chart.

Try the web app in your browser, with nothing to install.

Limitations

This uses files from the Quake 3 Demo, which has four maps. Servers can download more maps and mods, such as CPMA or Urban Terror, when they start (see Add Mods And Maps). Mods have to ship QVMs to run in the browser, since it can’t load native game libraries.

Any non-browser Quake 3 client can connect to the servers too.

What is this project for?

This was just made for fun and learning. It isn’t trying to be a complete solution for managing Quake 3 on Kubernetes, and I am using it now as a repo of common patterns and best practices (IMO) for Go/Kubernetes projects. It has since grown a Quake 3 master server, a server that exchanges information with the game client about what dedicated game servers are available, and a controller/CRD that ties it all together.

Try It In Your Browser

Open the preview

The preview is QuakeKube’s web app with nothing to install. You create servers and pick their maps, modes and bots like you would on a cluster, then play them right in the page.

It isn’t connected to anything, though:

• Servers are records saved in your browser, up to 3 of them. A game runs in its tab while you play it, against bots.
• There’s no Kubernetes, no gateway and no other players. To play with other people, see Getting Started.
• It doesn’t use Quake 3’s files. You can play OpenArena, a free game built on the same engine, with the OpenArena Community Mappack, CPMA, running on OpenArena’s files, or Urban Terror. See the Credits.
• Creating and deleting a server only look like a pod being deployed and terminated, to show what the real thing is like. A game actually starts when you press Play.

Browsers

Browsers with JavaScript Promise Integration, like Chrome, load game files as the game needs them: about 25 MB before the first map starts, and more as you play. Other browsers download about 420 MB of game files first, and the page says so before you join, and they don’t offer Urban Terror, which is too large to download whole. Either way, the files stay in your browser’s cache for next time.

Getting Started

QuakeKube is installed with its Helm chart, which deploys:

• the controller, which runs game servers for QuakeServer objects and a Quake 3 master server that lists them for native clients
• the gateway, which serves the web app, where people create servers and play in the browser

Install

The chart is published to GitHub’s container registry and works without any configuration:

helm install quake-kube oci://ghcr.io/chrisrx/quake-kube \
  --namespace quake-kube --create-namespace

The chart is published from every change to the main branch, and deploys the newest images. helm upgrade with the same command updates an install to the newest chart and images.

Then reach the web app through a port-forward:

kubectl --namespace quake-kube port-forward service/quake-kube-gateway 8080:80

Open http://localhost:8080, pick a game, mode and map under New server, and create it. Once it’s ready, Play joins it in the browser. The games are the Quake 3 demo, and on Kubernetes 1.35 and later, Urban Terror and CPMA (see Add Mods And Maps).

The game servers use the files from the Quake 3 demo, which are covered by the Quake 3 demo EULA. Installing QuakeKube means agreeing to it. Browser clients also show it to players before downloading the files.

Native clients

Game servers and the master server get LoadBalancer Services, so native clients can reach them on clusters with load balancers, from a cloud provider or MetalLB. Without one, servers still run and play in the browser. Their Services just wait for an address. gameServers.exposure in the chart’s values switches game servers to NodePort or host ports.

Any Quake 3 client can join a server at the address in its status:

kubectl get quakeservers --all-namespaces

The controller’s master server lists every server that has an address, so clients can find them in their server browser instead. Find its address with:

kubectl --namespace quake-kube get service quake-kube-master

and set it as a master server from the client’s console:

seta sv_master1 "<master address>"

Publish the web app

To make the web app public, put it behind an ingress with a TLS certificate, e.g. from cert-manager, and turn on WebTransport. Browsers only use WebTransport from HTTPS pages. It carries game traffic in unreliable datagrams, like the UDP the game was built for, where the WebSocket it otherwise falls back to holds everything up behind a lost packet.

gateway:
  ingress:
    enabled: true
    className: traefik
    host: quake.example.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
    tls:
      - secretName: quake-kube-web-tls
        hosts: [quake.example.com]
  webtransport:
    enabled: true

helm upgrade quake-kube oci://ghcr.io/chrisrx/quake-kube \
  --namespace quake-kube --values values.yaml

WebTransport runs over HTTP/3, so the gateway serves it from its own UDP LoadBalancer Service, quake-kube-gateway-webtransport, rather than through the ingress. Browsers connect to the web app’s hostname on port 443 unless told otherwise, so either:

• give the Service the ingress controller’s IP address, so the same hostname reaches both, e.g. with MetalLB:

  gateway:
    webtransport:
      service:
        annotations:
          metallb.io/allow-shared-ip: <the ingress Service's sharing key>
          metallb.io/loadBalancerIPs: <the ingress Service's IP>
  

• or give it an address of its own and tell browsers where it is:

  gateway:
    webtransport:
      publicAddress: quake-webtransport.example.com:443
  

The gateway uses a self-signed certificate for WebTransport, which browsers accept by its hash, so it doesn’t need one from a CA. With more than one gateway replica, set gateway.webtransport.tls.secretName to a certificate browsers trust, since each replica would otherwise have its own.

By default, anyone who can reach the web app can create up to gateway.maxServers servers, and delete the ones created there. To sign people in, e.g. with Google Cloud’s Identity-Aware Proxy, and give servers owners, see Authentication. Setting gateway.maxServers to 0 turns creating servers off.

The chart can also route to the web app with Gateway API instead of an ingress, with gateway.httpRoute.

Create servers with kubectl

Servers can also be created as QuakeServer objects, in any namespace:

apiVersion: quake.kube/v1
kind: QuakeServer
metadata:
  name: deathmatch
spec:
  bot:
    minPlayers: 4
  game:
    type: FreeForAll
  maps:
    - name: q3dm17
    - name: q3dm7

kubectl apply -f deathmatch.yaml
kubectl get quakeservers

See Configuration for the rest of the spec, and Add Mods And Maps for running mods. Mods’ content images need Kubernetes 1.35, or 1.33 with the ImageVolume feature gate.

Uninstall

helm uninstall quake-kube --namespace quake-kube

Servers created from the web app are deleted with it (gateway.deleteServersOnUninstall). Ones created with kubectl are left running, along with the QuakeServer CRD, which is kept by default (crds.keep). Delete the CRD to remove them too:

kubectl delete crd quakeservers.quake.kube

How it works

QuakeKube runs ioquake3 dedicated servers on Kubernetes, and ioquake3 built with Emscripten as the browser client. It’s made of a few parts:

• The gateway serves the web app and the browser client, and carries browsers’ game traffic to the servers, since browsers can’t send UDP.
• A QuakeServer is a game server’s spec, stored in the Kubernetes API. The web app creates them, and so can kubectl.
• The controller turns each QuakeServer into a running game server pod, and runs the master server native clients find servers through.
• Content, a mod and its maps, comes from content images or downloads, and browsers download the same files from the server.

Controller

The controller reconciles QuakeServer objects. For each one it creates:

• a Secret with the server’s configuration and a generated rcon password
• a Deployment running the game server
• a Service for the gateway to reach the server inside the cluster
• a Service exposing the game’s UDP port to native clients, depending on gameServers.exposure (LoadBalancer, NodePort, a host port or nothing)

The server’s status says when it’s ready, which is once its pod runs the current spec, and has its external address once it’s reachable. When the pod is stuck rather than starting, e.g. because its image can’t be pulled or its content fails to download, the status has a message saying why, which the web app shows.

The controller also runs a Quake 3 master server. Game servers send it heartbeats, and it answers server browsers’ queries with every ready server’s external address.

Game servers

Each game server’s pod runs the q3 command from the game server image:

• q3 fetch, an init container, puts the server’s content in place: it downloads pk3s and zip archives, and links the files from content images, which are mounted as image volumes.
• q3 run adds the Quake 3 demo’s files, turns the configuration into a server.cfg and runs ioq3ded. When the pod stops, it warns players and counts down before shutting the server down.

q3 run also serves, on port 8080, the server’s content and info for browser clients, its maps’ levelshots, its Prometheus metrics, and a gRPC health check for the pod’s probes.

When the configuration changes, q3 run changes the running server from its console where it can: a different map, game type, limits or bots reload the map, the way a map rotation does, and players stay connected. Only a different mod restarts the server, and that replaces its pod anyway.

Gateway

The gateway serves the web app, which lists the servers, creates them from the presets in gateway.presets, and runs the browser client.

The Quake 3 protocol is UDP, which browsers can’t send. The browser client’s traffic goes to the gateway instead, which relays it to the server’s UDP port. It uses WebTransport when it can, whose unreliable datagrams behave like UDP, and falls back to a WebSocket, where one lost packet holds up everything behind it. Since the servers only ever see UDP, they’re plain ioquake3 servers that native clients can join too.

Changing a server

A server’s owner can change it from the web app’s server list, or from the game’s pause menu while playing: its map, game type, bots and limits, or which game it runs. The change is saved in the QuakeServer, but that takes a minute to reach the server: the controller renders the Secret, and Kubernetes updates the file in the pod. So the gateway also sends the server the same configuration right away:

The gateway renders the configuration with the controller’s code and authorizes it with the server’s rcon password, which it reads from the server’s Secret. When the Secret’s copy arrives, q3 run finds nothing new and ignores it.

Switching to another game changes the server’s content, so its pod is replaced. Players’ pages notice when the server comes back running the other game and rejoin it. The browser client also rejoins when the server changes to a map whose files it didn’t download, unless it reads files as it needs them.

Idle servers

Servers created from the web app are deleted after nobody has played on them for gateway.idleTimeout, 30 minutes by default. That includes servers that never start. Kubernetes deletes their pods and Services with them.

Levelshots

Servers play a rotation of maps, moving to the next one after each match. The web app shows a picture of each map, its levelshot, which comes from the map’s pak, with its title and the game types it’s made for, from its arena file. For maps of games no server is running, the gateway runs a pod for each preset that assembles the preset’s content like a game server’s pod, with q3 levelshots serving thumbnails of the maps’ levelshots and what their arena files say about them. The gateway keeps them in a ConfigMap, quake-kube-levelshots-<preset>, and only makes them again when the preset’s content or maps change.

Browser client

The play page downloads the server’s content through the gateway, the same files the server has, so it plays the server’s mod and maps. It keeps them in the browser’s cache, keyed by checksum, so they’re only downloaded once and shared between servers. Browsers with JSPI read the paks as the game needs them instead, so joining doesn’t wait for a whole mod to download. Mods have to ship QVMs to run in the browser, since it can’t load native game libraries.

Quake 3 demo EULA

The game servers use the files from the Quake 3 demo, whose End-User License Agreement must be agreed to before they’re distributed. q3 run won’t start without --agree-eula, which the controller passes, so installing QuakeKube means agreeing to it. Browser clients show it to players before downloading the files.

Configuration

QuakeServer

Each server is a QuakeServer. Every field is optional:

apiVersion: quake.kube/v1
kind: QuakeServer
metadata:
  name: quake
spec:
  # Defaults for maps that don't set their own.
  fragLimit: 25
  timeLimit: 15m
  bot:
    minPlayers: 4
    noChat: true
  game:
    # Game type of maps that don't set their own: FreeForAll, Tournament,
    # SinglePlayer, TeamDeathmatch or CaptureTheFlag, or a mod's g_gametype
    # number.
    type: FreeForAll
    motd: "Welcome to Quake Kube"
    password: "letmein"
    forceRespawn: false
    # Idle players are kicked after this long.
    inactivity: 10m
    quadFactor: 3
    # Seconds.
    weaponRespawn: 3
    singlePlayerSkill: 2
  server:
    # Defaults to the QuakeServer's name.
    hostname: "Quake Kube"
    maxClients: 12
  # The map rotation, played in order.
  maps:
    - name: q3dm7
    - name: q3dm17
      fragLimit: 30
      timeLimit: 20m
    - name: q3tourney2
      type: Tournament
  # Console commands run once the first map has loaded.
  commands:
    - addbot sarge 2
  # Cvars set before the first map loads.
  cvars:
    sv_fps: "30"

Without maps, the server plays q3dm7 and q3dm17. The Quake 3 demo only has q3dm1, q3dm7, q3dm17 and q3tourney2. Others need content (see Add Mods And Maps), which is also where game.fsGame, game.native and content are covered.

A map’s fragLimit, timeLimit and captureLimit apply from that map on. Mods number their game types differently, so captureLimit is set for any type, such as Capture The Flag:

maps:
  - name: q3wctf3
    type: CaptureTheFlag
    captureLimit: 8

Changes to the spec are applied to the running server, which restarts with the new configuration. Changing the content or the mod replaces the server’s pod.

The chart

The chart’s values.yaml documents its settings, including:

• gateway.presets: the games the web app offers (see Add Mods And Maps)
• gateway.imagePresets: whether to offer the presets with content images, which need Kubernetes 1.35
• gateway.auth: how the gateway knows who people are, for server owners and admins (see Authentication)
• gateway.namespace: where the web app creates servers, the release namespace by default. The gateway reads Secrets there, for servers’ rcon passwords, so changes made from the web app apply right away, and runs the pods that make thumbnails of each game’s maps, which it keeps in ConfigMaps. Use a namespace of its own if the release namespace has other Secrets.
• gateway.maxServers: how many servers the web app can create, or 0 to turn creating servers off
• gateway.maxServersPerUser: how many each person can have, with authentication
• gateway.idleTimeout: how long a server created from the web app can go without players before it’s deleted (30m by default), or empty to keep servers until someone deletes them
• gameServers.exposure: how native clients reach game servers, and the port range servers get their external ports from when they share an address
• master.service: how native clients reach the master server

Add bots

The simplest way to add bots is to set a minimum number of players. The server adds bots up to that number and removes them as people join:

spec:
  bot:
    minPlayers: 8
  game:
    singlePlayerSkill: 2

singlePlayerSkill sets the skill of the bots it adds, from 1 to 5 (2 by default). The web app’s Bots field sets minPlayers for servers it creates.

Bots can also be added individually with the addbot command, which takes the bot’s name and skill level, using the commands section:

spec:
  commands:
    - addbot crash 1
    - addbot sarge 2

The Quake 3 demo only has some of the game’s bots. Mods like Urban Terror don’t have bots at all, so their presets hide the web app’s Bots field.

Setting a password

To require a password to join a server:

spec:
  game:
    password: "letmein"

The web app’s Password field sets it for servers it creates. The browser client asks for it when a server needs one.

Remote administration

The controller generates an rcon password for each server and keeps it in the server’s Secret, which has the same name as the server:

kubectl get secret <server> -o jsonpath='{.data.rconPassword}' | base64 -d

A native client can then administer the server from its console with \rcon <password> <command>.

Add mods and maps

A QuakeServer’s content lists where its files come from, in addition to the Quake 3 demo. They’re put in place when the server’s pod starts, before the game server runs.

Content images

An entry with image names an OCI image whose top-level directories are game directories, e.g. q3ut4/. The image is mounted in the pod as an image volume and its files are linked into the content, so nothing is downloaded once the node has the image, and servers on the same node share it. Image volumes need Kubernetes 1.35, or 1.33 with the ImageVolume feature gate.

content:
  - image: ghcr.io/chrisrx/quake-kube-urban-terror:4.3.4

These images are published for x86_64 and aarch64:

Native game code

Browsers need a mod’s game code as QVMs, and so do servers normally. Rocket Arena 3 ships QVMs for the client, but its server code is only a 32-bit x86 library. With game.native, the server runs a 32-bit build of the game server that loads it:

spec:
  game:
    fsGame: arena
    native: true

Such servers only run on x86-64 nodes, which the controller selects. The library is closed source native code from 2004, and runs inside the server’s pod with an executable stack, which it needs. CPMA’s Clan Arena is the same game mode with no native code, and the Rocket Arena preset plays RA3’s maps with it.

They’re built from content/ by the flake. To build one yourself and push it to your own registry:

nix build .#urban-terror-image
docker load < result
docker tag quake-kube-urban-terror:latest registry.example.com/quake-kube-urban-terror:4.3.4
docker push registry.example.com/quake-kube-urban-terror:4.3.4

A content image is any image with game directories at its top, so one with other mods or maps can be built however you like, e.g. from a Dockerfile:

FROM scratch
COPY q3ut4 /q3ut4

Downloads

Each other entry has a url to a pk3 or a zip archive, downloaded each time the server’s pod starts:

• A pk3 goes in the server’s game directory, game.fsGame or baseq3, unless dir says otherwise.
• An archive is extracted at the top of the content, where each game has its own directory (baseq3/, cpma/, …), so a mod’s release that contains cpma/z-cpma-pak153.pk3 lands where the game looks for it. Use dir to extract it somewhere else, and stripComponents to remove leading directories from its paths. Files left outside a directory are skipped.

Setting sha256 makes the server refuse a download that doesn’t match.

Changing content replaces the server’s pod.

Mods

game.fsGame picks the mod the server runs, and cvars sets anything that has to be in place before the first map loads:

apiVersion: quake.kube/v1
kind: QuakeServer
metadata:
  name: cpma
spec:
  game:
    fsGame: cpma
  cvars:
    mode_start: FFA
    # Otherwise CPMA picks each map's mode from its own list.
    server_useMapModes: "0"
  content:
    - url: https://cdn.playmorepromode.com/files/cpma/cpma-1.53-nomaps.zip
      sha256: edfffa0c1a0375ba46a5b42257a168fb15086712245733526ab2d9ccdd821ca0
    - url: https://cdn.playmorepromode.com/files/cpma-mappack-full.zip
      dir: cpma
  maps:
    - name: cpm22

Mods number their own game types, so type also takes a g_gametype number, e.g. 7 for Capture The Flag in Urban Terror. The names (FreeForAll, CaptureTheFlag, …) are the standard Quake 3 game types.

Browser clients download the server’s content, so the mod has to ship QVMs: the browser can’t load native game libraries. CPMA and Urban Terror both do.

Loading in the browser

Browsers with WebAssembly’s JSPI, such as Chrome and Edge, load files from paks as the game reads them, rather than downloading whole paks. A map in Urban Terror reads around 150 MB of its 1.5 GB. If the content has a profile, which lists the files the game reads on each map, the play page fetches those in a few large batches before the game starts, and the game only waits for anything the profile missed. Browsers keep what they fetch in their cache, so joining again, or joining another server with the same content, fetches only what’s new.

Other browsers download paks before the game starts and hold them in memory while playing, so a large mod like Urban Terror needs a desktop browser there. They only download a map’s own pak when the server plays that map, or when the profile says the map reads from it, and fetch the paks of the other maps the server rotates through in the background after joining. A map’s pak is one named after every map in it, like map_cpm22.pk3. Paks that bundle maps under other names, like Urban Terror’s, often carry shared content too, so browsers always download them. The play page says how much joining downloads, and browsers keep the files in their cache.

Profiles

A content image’s profile is <game>/quake-kube-profile.json, e.g. cpma/quake-kube-profile.json. The CPMA and Urban Terror images have one, generated by playing each of their preset’s maps in the dev cluster (see Development). A server merges the profiles of all its content, so a map pack can bring its own.

Presets

The web app creates servers from the presets in the chart’s gateway.presets. Each one is a game: its game types, its maps, whether it has bots, and the start of each server’s spec, such as a mod’s game.fsGame and content. A game type sets type, cvars or both, since mods choose game types differently:

gateway:
  presets:
    - name: urban-terror
      title: Urban Terror 4.3
      bots: false
      gameTypes:
        - {name: Capture The Flag, type: 7}
        - {name: Team Deathmatch, type: 3}
      maps: [ut4_turnpike, ut4_casa]
      spec:
        game: {fsGame: q3ut4}
        content:
          - image: ghcr.io/chrisrx/quake-kube-urban-terror:4.3.4
    - name: cpma
      title: CPMA
      gameTypes:
        - {name: Free For All, cvars: {mode_start: FFA}}
        - {name: Duel, cvars: {mode_start: 1v1}}
      maps: [cpm22, cpm4a]
      spec:
        game: {fsGame: cpma}
        cvars: {server_useMapModes: "0"}
        content:
          - image: ghcr.io/chrisrx/quake-kube-cpma:1.53

These are in the chart’s default presets, with more of their game types and maps, alongside the Quake 3 demo. Setting gateway.presets replaces them.

Servers play a rotation of the preset’s maps, picked in the web app. The picker offers the maps made for the chosen game type first, from each map’s arena file (e.g. scripts/q3dm17.arena). It knows Quake 3’s game types; for a mod’s, set arena to the type its maps’ arena files use, e.g. ctf or team. Presets can also suggest lists of maps to pick from with mapLists:

gateway:
  presets:
    - name: openarena-ctf
      title: OpenArena CTF
      gameTypes:
        - {name: Capture The Flag, type: 4, arena: ctf}
        - {name: One Flag CTF, type: 5, arena: oneflag}
      maps: [oasago2, ps37ctf2, oa_ctf4ish]
      mapLists:
        - name: Classics
          maps: [oasago2, ps37ctf2]

People can save their own lists in their browser too, and share a rotation with a link from the create dialog.

Presets with content images need image volumes, so on clusters older than Kubernetes 1.35 the chart leaves them out. Set gateway.imagePresets to true on 1.33 or 1.34 with the ImageVolume feature gate, or to false to never offer them. A preset with mapsWithoutImages is kept without its images instead, offering only those maps, like the Quake 3 preset, which offers just the demo’s maps then.

Maps for Quake 3 also play in OpenArena, which runs on top of Quake 3’s baseq3, so the demo’s maps and the Quake 3 community maps image are in the OpenArena preset too. Native OpenArena clients don’t have the demo’s files, though browsers download them.

Content that needs the full game

Some mods use files from the full game, which the demo doesn’t have and which can’t be freely distributed. Urban Terror is a standalone game, so it doesn’t need any. CPMA’s client won’t start without the full game’s default player model, mynx, so the CPMA content image builds a stand-in from OpenArena’s “major” model (GPLv2, see content/cpma/openarena-mynx.sh).

Maps often use textures from the full game too. OpenArena has same-named replacements for many of them, so content images include a pk3 of the ones their maps need, built by content/openarena-standins.sh from a list that hack/standins.py writes:

python3 -I hack/standins.py content/cpma/standins \
  --base <demo paks> <the mod's paks> \
  --source <OpenArena's baseoa paks> \
  --maps <map pk3s>

It reports what’s still missing. The maps in the presets miss at most a few textures, and only Rocket Arena 3’s maps that are complete, or close to it, are included. Missing textures and sounds don’t stop the game, but show up as blank surfaces and silence.

If you own Quake 3, adding your pak0.pk3 to content with dir: baseq3 uses it instead of the demo’s, though it can’t be redistributed.

Authentication

By default, anyone who can reach the web app can create servers and change or delete the ones created there. To know who people are, put a proxy that signs them in in front of the gateway and tell the gateway how to read who they are, with gateway.auth.mode:

• none: the default. Servers have no owners.
• iap: Google Cloud’s Identity-Aware Proxy. The gateway checks the assertion IAP signs for each request.
• header: another proxy that signs people in, such as oauth2-proxy, and names them in a header.

The gateway doesn’t sign people in itself, so it works with whatever your cluster already uses.

Owners and admins

With authentication on, a server created from the web app belongs to whoever created it, which is recorded in its quake.kube/owner annotation. Only they can change or delete it from the web app, or change it from the game’s pause menu while playing, along with admins.

Admins are whoever Kubernetes RBAC lets delete QuakeServers in the web app’s namespace, which the gateway asks the API server about. So anyone who could already delete servers with kubectl can change and delete them from the web app too, and admins can be granted in the same way as other access to the cluster. The chart has a quake-kube-admin ClusterRole for this, and binds it to the users and groups in gateway.auth.admins:

gateway:
  auth:
    admins:
      users: [[email protected]]
      groups: [quake-admins]

Servers created before authentication was turned on have no owner, so only admins can change or delete them.

gateway.maxServersPerUser limits how many servers each person can have at once, alongside gateway.maxServers for everyone together. Admins have the same limit.

Google Cloud’s Identity-Aware Proxy

On GKE, IAP signs people in with their Google accounts before they reach the web app. Who can reach it is managed in IAM, by granting the IAP-secured Web App User role, e.g. to a Google group.

IAP works with GKE’s load balancers, through Gateway API or a GKE Ingress. The chart creates GKE’s settings for the gateway’s Service with gateway.gke: a GCPBackendPolicy for an HTTPRoute, and a BackendConfig for an Ingress. With Gateway API:

gateway:
  auth:
    mode: iap
    iap:
      # The project's number, not its ID.
      projectNumber: "123456789012"
  httpRoute:
    enabled: true
    parentRefs:
      - name: external
        namespace: infra
    hostnames: [quake.example.com]
  gke:
    enabled: true
    iap:
      enabled: true

or with a GKE Ingress, replace httpRoute with:

gateway:
  ingress:
    enabled: true
    className: gce
    host: quake.example.com

IAP signs its assertions for the load balancer’s backend service, which GKE creates and names, so the gateway accepts assertions for any backend service in the project. Set gateway.auth.iap.audience to the backend service’s audience instead, /projects/<number>/global/backendServices/<id>, to only accept its own.

Without an OAuth client, IAP uses a Google-managed one, which only lets in people in your organization. To use your own, set gateway.gke.iap.clientID and gateway.gke.iap.clientSecretName.

gateway.gke also keeps GKE’s load balancers from closing browser clients’ WebSockets after 30 seconds, with timeoutSec, so turn it on with GKE’s load balancers even without IAP.

The gateway’s own Service isn’t behind IAP, so the gateway turns away requests without a valid assertion, e.g. from inside the cluster.

Other proxies

With header, the gateway reads the user’s email address from a header, and their groups from another, if the proxy sends them. With oauth2-proxy’s defaults:

gateway:
  auth:
    mode: header
    header:
      user: X-Forwarded-Email
      groups: X-Forwarded-Groups

The gateway trusts these headers, so it must only be reachable through the proxy, e.g. with a NetworkPolicy that only lets the proxy reach it. Groups from the proxy are what RBAC checks admins’ groups against.

WebTransport

WebTransport sessions reach the gateway over UDP, past the proxy, so the play page gives browsers a ticket for the server they’re joining, which the gateway checks. A ticket works for an hour, and browsers that reconnect later use the WebSocket, which goes through the proxy.

Each gateway replica signs tickets with its own key unless gateway.auth.ticketSecretName names a Secret with a shared key, so set it with more than one replica:

kubectl --namespace quake-kube create secret generic quake-kube-tickets \
  --from-literal=key="$(head -c 32 /dev/urandom | base64)"

Native clients

Native Quake 3 clients connect to game servers directly, past the proxy. To keep people from joining, give servers a password (see Setting A Password), or turn off their external addresses with gameServers.exposure.type: None, which leaves them to browser clients.

Development

Layout

• controllers/: the Rust binary for the controller (including the master server) and the gateway, the web app’s static files, and the browser client’s build (upstream ioquake3 with Emscripten, see controllers/client/)
• cmd/q3, internal/: the Go q3 command that runs in each game server’s pod (q3 fetch and q3 run)
• chart/: the Helm chart. chart/files/crds is generated from the Rust types with cargo run -- crdgen.
• content/: content images for mods (see Add Mods And Maps)
• nix/: how the flake builds everything, including the images
• hack/: the local cluster’s setup

Images

Every push to main publishes the images and the chart (see Publishing).

The flake builds every image, cross compiling from the build machine for x86_64 and aarch64. The ones for the build machine’s platform are packages:

nix build .#controller-image   # also server-image, urban-terror-image, cpma-image
docker load < result

and every platform’s are in images, e.g. .#images.aarch64.server. The manifests push them to ghcr.io as multi-arch images, which the Push Image and Push Content Images workflows run:

GITHUB_TOKEN=... VERSION=v1.0.0 nix run --impure .#manifests.controller

A flake only sees the files git tracks, so new files are left out until they’re added. nix/dev.nix builds the same packages from the working tree, which is what Tilt uses:

nix build --impure -f nix/dev.nix controller-image

Local cluster

Development uses a k3d cluster that mirrors a typical cloud setup: Traefik for ingress, MetalLB for LoadBalancer Services and cert-manager with a local CA. It requires k3d 5.8 or newer, which the flake’s dev shell provides.

task cluster

This creates the k3d-quake-kube context and a registry at localhost:5005, and prints the Traefik address. LoadBalancer addresses come from 172.28.0.100-172.28.0.199 and are reachable directly from the host on Linux. Set K3D_SUBNET to a different /24 if 172.28.0.0/24 is already in use.

With podman, the rootful socket (/run/podman/podman.sock) is used automatically, since rootless podman networks aren’t reachable from the host.

task cluster-delete removes the cluster and registry.

Tilt

Tilt builds the controller and game server images with nix, deploys the chart and the example server in controllers/example.yaml, and rebuilds as files change:

task dev

This creates the cluster if needed. The web app is at https://quake.172-28-0-100.sslip.io through Traefik.

task ui opens it. Tilt’s UI also links to it.

Its certificate comes from cert-manager’s local-ca issuer. For browsers to trust it, which WebTransport needs, run this once per machine:

task ui:trust

It creates a local CA with mkcert and adds it to Chrome’s and Firefox’s trust stores (restart them afterward). The dev cluster then signs with that CA, from task cluster or the next task ui, so the web app is trusted, including after recreating the cluster. On NixOS, mkcert only changes the browsers’ trust stores; the system’s are declared in its configuration, e.g. security.pki.certificateFiles.

Without mkcert’s CA, each cluster has a CA of its own. task ui writes it to hack/k3d/local-ca.crt when it changes and says how to import it into a browser, which a new cluster needs again. task ui:ca writes it on its own.

task ui:forward opens the web app through a port-forward instead, with the current kubectl context, which also works for a cluster installed with the chart, e.g. task ui:forward NAMESPACE=games PORT=9090. Without HTTPS, browsers use a WebSocket rather than WebTransport, and don’t cache the game’s files.

The dev cluster’s presets, in hack/k3d/presets.yaml, are the chart’s with content images from the working tree. Tilt builds them and pushes them to the cluster’s registry. The first build downloads the mods, 1.4 GB for Urban Terror, which nix keeps for later builds. Servers running mods are also in controllers/example-cpma.yaml and controllers/example-urt.yaml.

Profiles

Content images’ profiles say which files the browser client reads on each map (see Add Mods And Maps). Regenerate one after changing a mod’s content or its preset’s maps, with the dev cluster running:

task content:profile PRESET=urban-terror CONTENT=urban-terror

This starts a server for each of the preset’s maps in hack/k3d/presets.yaml, joins it in headless Chromium with the engine logging the files it opens, and writes content/<mod>/profile.json, which the content image includes. Files read on at least half the maps are fetched for every map.

Publishing

There are no tagged releases. Every push to main pushes the server and controller images tagged main and latest, then publishes the chart to oci://ghcr.io/chrisrx/quake-kube with the version in chart/Chart.yaml, replacing the published chart with that version. The chart deploys the latest images. CI adds the commit to the published chart’s annotations, which its pods carry, so helm upgrade restarts them and they pull the newest images. task chart:push publishes the chart by hand.

Content images are versioned separately, by the mod’s version in content/<mod>/default.nix, and published when content/ changes.

Tasks

Common tasks are in Taskfile.yml, for Task, which the flake’s dev shell has. task lists them:

task test   # Go and Rust tests
task lint   # golangci-lint, clippy, rustfmt and helm lint
task crds   # regenerate the chart's CRDs after changing the controller's types
task docs   # build this book
task docs:site   # and the browser preview at /play, as published
task docs:serve  # build those and serve them at http://127.0.0.1:3000/, press space to open the preview

Credits

• ioquake/ioq3 - The community supported version of Quake 3 used for the dedicated server and the browser client. It is licensed under the GPLv2.
• inolen/quakejs - The really awesome QuakeJS project, which was the original browser client and showed it could be done.
• begleysm/quakejs - Information in the README.md (very helpful) was used as a guide for setting up the original QuakeJS client.
• OpenArena - Its “major” player model and voice stand in for CPMA’s default player model, which is only in the full game, and its textures and models stand in for the full game’s in maps and mods. Its game and the OpenArena Community Mappack are what the browser preview runs. It is licensed under the GPLv2.
• Missing textures pack - Gábor Udvari’s replacements for missing Quake 3 textures, made for OpenArena. A few stand in for the full game’s in community maps. It is licensed under the GPLv2.
• q3a2oa textures, version 4 - Jack Thompson’s placeholders for Quake 3 textures, each a block of its average color. They stand in where nothing better has been found. It is licensed under the GPLv2.
• The rest of the community maps’ stand-ins were made for QuakeKube: some generated from scratch, like the lava’s bubbles in ChiropteraTA, and one adding rust to OpenArena’s dirty_pewter_big. Their scripts are in content/quake3-maps/standins/generated, and how each stand-in was chosen is recorded in content/quake3-maps/standins/decisions.jsonl.
• CPMA - Challenge ProMode Arena, by the CPMA team, which the browser preview offers.
• Urban Terror - By FrozenSand, which the browser preview offers.
• Ultra Freeze Tag - By Neil “haste” Toronto, built on Darrell Bircsak’s Freeze Tag.
• DeFRaG - The trick jumping and racing mod, by the DeFRaG team.
• joz3d.net - Useful information about configuration values.

Community maps

The Quake 3 community maps come from lvlworld, and their readmes allow distributing them for free, unaltered. They’re shipped as they were released, with their readmes, in the ghcr.io/chrisrx/quake-kube-quake3-maps image. Bundling them there, rather than downloading them from lvlworld when servers start, keeps QuakeKube from putting load on a site that has kept these maps available for decades: game servers get them from the image, and players’ browsers from the game server.

Content removal

Most of the maps and mods QuakeKube ships were made years ago by people who can’t be reached anymore, and their licenses are often loosely worded. They’re included because their readmes allow distributing them, with every effort to respect their authors’ wishes. If you made something QuakeKube ships and want it removed, or its credit changed, open a content removal request, or use the link next to a map above, and it will be taken out.