Engineering onboardinggiga-swamp@mainSnapshot 2026-09-14

Production architecture

Where our apps run, which clouds and regions they use, how requests reach them, and where the data lives. This is the big picture, not a runbook. Every piece is a swamp model or workflow in giga-swamp.

Architecture

hover a component to trace its connections

Web and telemetry traffic uses Cloudflare only for DNS, then connects straight to the DigitalOcean load balancer. CLI downloads are the only requests Cloudflare proxies: its edge runs the artifacts-edge Worker, which pulls binaries from Spaces. The internal ops platform runs on a separate cluster in Google Cloud.

request data read / write DigitalOcean Google Cloud Cloudflare SaaS

Clouds & regions

Almost everything runs in DigitalOcean's San Francisco region, SFO3. We moved off AWS in May 2026. Google Cloud hosts only the internal ops platform.

ProviderRegionWhat lives there
DigitalOceansfo3The swamp-club-prod Kubernetes cluster (DOKS), ClickHouse and Prometheus droplets, Spaces buckets, the container registry DOCR swamp, and the prod VPC 10.10.0.0/16 with its NAT gateway.
DigitalOceannyc3The forgejo-runner DOKS cluster: CI runners that autoscale from 1 to 12 nodes.
Google Cloudus-west1-aThe GKE platform cluster in project swamp-club-ops: swamp serve, workers, and the dashboard, plus Artifact Registry and the GCS datastore.
CloudflareglobalDNS for swamp.club, swamp-club.com, swamp-club.ai, and systeminit.com, the artifacts-edge Worker, and DNS-01 challenges for TLS certificates.
SaaS—MongoDB Atlas (primary database), Axiom (logs and traces), Resend (email), Anthropic (LLM), Stripe, Attio, Mixpanel, GitHub, Discord, and 1Password.

Main apps

Everything customer-facing runs on swamp-club-prod. Workloads land on dedicated node pools: swamp-club-16g for the core apps and Traefik, and corp-apps for internal tools and monitoring. All images come from DOCR, pinned by digest.

AppWhat it doesRuns onScale
swamp-club-apins swamp-clubThe swamp.club product: web UI, API, sign-in (GitHub, Google, Microsoft, Discord), and the admin proxy to corp apps.DOKS prodHPA 2–6
telemetry-apins telemetryTakes in swamp CLI telemetry events and GitHub webhooks.DOKS prodHPA 2–10
Traefikns traefikIngress controller behind the DO load balancer. Redirects HTTP to HTTPS.DOKS prodHPA 4–10
discord-bot+ discord-summarizerBackground workers. The bot writes Discord activity to Mongo. The summarizer posts Claude-written digests. Neither has a public route.DOKS prod1 each
Corp appsInternal tools: dossier, biz-model, fundraising, and the pitch/partner/value decks. None has its own Ingress. Admins reach them through swamp-club's proxy.DOKS prod1 each
GrafanaDashboards over ClickHouse and Prometheus, plus alerting.DOKS prod1
swamp serveops.swamp-club.comOur internal ops platform. Runs swamp workflows from Forgejo, Grafana, and Axiom webhooks. Also serves the swamp dashboard.GKE platform1 + 5 workers

Public routes

swamp-club.com is the canonical host. The .club names are proxied by Cloudflare and redirect to it. TLS certificates are issued by cert-manager in each cluster, using Let's Encrypt with DNS-01 through Cloudflare.

HostnamePathLands on
swamp-club.comCloudflare DNS-only → DO load balancer → Traefikswamp-club-api
swamp.clubCloudflare proxy, 301 redirect→ swamp-club.com
telemetry.swamp-club.comCloudflare DNS-only → DO load balancer → Traefiktelemetry-api
telemetry.swamp.clubCloudflare proxy, 301 redirect→ telemetry.swamp-club.com
artifacts.swamp-club.comCloudflare Worker artifacts-edgeSpaces bucket artifacts-swamp-club
artifacts.systeminit.com/swamp/*Same Worker, kept for older CLIsSame bucket
ops.swamp-club.comGCP load balancer → Traefik on GKEswamp serve, /hooks/*, /dashboard

Data & storage

We have two databases with different jobs, and object storage holds the long-term record.

MongoDB Atlas

  • Operational data for swamp-club, telemetry, and the Discord bot
  • IP allowlist maintained via the grant-mongo-ip and audit-mongo-ips workflows

ClickHouse

  • Analytics and scoring
  • 3 replicated servers and 3 Keepers on droplets, reachable only from the cluster
  • Apps connect to clickhouse.clickhouse.svc, which points at the droplets
  • Nightly backups go to Spaces

Spaces · sfo3

  • events-archive-swamp-club: the event lake and system of record
  • artifacts-swamp-club: CLI binaries
  • swamp-club-extensions: the extension registry
  • clickhouse-production-backups, dossier-swamp-club
  • giga-swamp: the infra repo's swamp datastore

Shipping changes

Infrastructure is defined and changed with swamp. Model and workflow definitions are in git (giga-swamp). Runtime state is in the Spaces datastore.

  1. Merge app code
    A merge in the swamp-club repo starts a release in GitHub Actions.
  2. Build & push
    A release workflow such as release-swamp-club-api-k8s builds the image and pushes it to DOCR.
  3. Roll out
    The workflow rolls the Deployment using a narrowly scoped release-deployer account, then waits for the rollout to finish.
  4. Change structure
    Changes to resources, env, probes, or ingress are model YAML edits, deployed via workflow. Dry-run first.
House ruleNever change live workloads with kubectl edit/patch/scale. The next swamp deploy overwrites the change and field ownership conflicts follow. Go through the swamp model instead.

Observability

Traces & logs

  • Axiom is our OpenTelemetry and log aggregator
  • Apps and swamp serve send OTLP traces and logs to dataset swamp_club_prod
  • Axiom monitors raise alerts from those queries

Metrics & dashboards

  • A Prometheus agent in the cluster remote-writes to a Prometheus droplet in the VPC
  • Grafana, a corp app reached via the admin proxy, shows dashboards over Prometheus and ClickHouse and holds alert rules

Alert triage

  • Grafana and Axiom each have their own signed endpoint on ops.swamp-club.com: /hooks/grafana-alerts, and /hooks/axiom-alerts, which goes through axiom-webhook-relay first
  • The ops-signal-ingest workflow dedupes each alert, triages it with Claude, and posts it to Discord #ops

Names that aren't prod

The repo keeps models for retired or rollback-only infrastructure. Don't mistake them for live systems.

DO App Platform
Before DOKS, swamp-club, telemetry, and discord-bot ran here. Traffic has since moved to DOKS.
platform-cluster
The DOKS cluster that previously hosted swamp serve, kept only as a rollback path since the move to GKE in August 2026.
platform-orchestrator
The original swamp serve droplet, deleted 7 Aug 2026.
swamp-club-staging
A staging cluster, torn down July 2026. Its models remain so it can be rebuilt.
clickhouse-v2
The single-pod, in-cluster ClickHouse that the droplet cluster replaced. It's kept for rollback.
AWS
A frozen snapshot of S3 buckets from before the May 2026 cutover. Nothing new is written there.
Two swamp-servesPlatform swamp serve is ops.swamp-club.com on GKE. The app-side swamp-serve is a WebSocket companion to swamp-club-api, built into DOCR.

Review notes · remove before sharing

This page was put together from repo models, manuals, and a live DNS check on 14 Sep 2026. These items are unconfirmed or have conflicting sources:

  • Whether the App Platform apps have actually been deleted, or are only scaled to 0.
  • Whether the app-side swamp-serve still runs in the swamp-club namespace on DOKS.
  • Whether idle clickhouse-v2 and the empty clickhouse node pool are still running and billed.
  • The Atlas cluster's region and tier aren't recorded in the repo, and the allowlist model still has an old App Platform egress IP.
  • Which platform datastore is live: GCS or Spaces swamp-platform-datastore. The GKE notes point to GCS.
  • The Forgejo instance and which repos forgejo-runner builds.
  • The Athena read path is still referenced in telemetry env but is broken.