SlotSense: A Multi-Tenant, AI-First Sports Booking Platform

Why this article exists

I built SlotSense as a working answer to a specific question: can one engineer, working with disciplined AI collaboration, ship a production-grade multi-tenant SaaS with the same care a senior engineering team would bring — ADRs before code, verified accessibility, a formal SLO, a real DR runbook, honest treatment of tradeoffs — and prove it in the open?

Seventeen phases and forty-three Architecture Decision Records later, this article is that proof, written for the reader who wants to see how the pieces fit: the product economics that shaped scope, the architectural depth that keeps a multi-tenant platform safe, the non-functional discipline that separates “production-functional” from “production-ready,” and the engineering process that let it all happen inside one person’s calendar.

An earlier version of this writeup closed the day Phase 9 (the AI booking agent) shipped. Since then the platform has gained a full billing pipeline, DPDP-compliant entity lifecycle management, voice I/O for the agent, a portfolio-quality UI with verified accessibility, and a deep pass at production readiness. This version reflects the repository as of Phase 17.

The problem

In Indian residential communities, booking shared sports facilities — tennis courts, badminton courts, the cricket nets, the swimming pool — is usually handled through WhatsApp groups, paper rosters, or rudimentary booking apps. Coordination friction is real: residents double-book by accident, courts sit empty because the booking system was offline, admins spend evenings reconciling who-owes-what.

Most communities outsource facility management to third-party operators. These operators typically take 50–75% of booking revenue. The previous-generation vendor in this space charged ~75% of booking fees; current vendors have compressed to around 50%. Their genuine operational contribution is modest — a few support people, maintenance coordination — and the real value they capture is the booking platform itself, which most communities can’t build independently.

This is the gap SlotSense closes. If a community has access to a multi-tenant booking platform purpose-built for residential sports facilities, the case for outsourcing management collapses. Communities can run facility operations through their existing maintenance and admin staff, and the 50–75% revenue cut goes back to either the community treasury or to lower booking fees for residents. The economics matter: for a community with even modest facility revenue (say, ₹50,000/month in booking fees), the gap represents ₹3–4.5 lakh per year in retained value.

SlotSense is built to make that retention possible.

What SlotSense is

SlotSense is a multi-tenant SaaS platform for sports facility booking in residential communities, designed for one-platform-per-country deployment serving thousands of communities, each with thousands of residents. Four architectural commitments make it different from typical booking systems:

• Rigorous tenant isolation — five independent layers, every one of which has to fail for a cross-tenant leak.
• An AI assistant residents can talk to — in text and, since Phase 16, in voice — with hard safety properties that prevent the LLM from ever directly mutating state.
• Zero-key-management security — every secret-bearing pattern deliberately replaced with something that doesn’t require a secret.
• Production-ready operational posture — a formal 99% availability SLO, four alert policies, a six-layer DR runbook at 4-hour RTO/RPO, Terraform-codified infrastructure, and a cost budget with graduated thresholds.

The product mindset — what shaped scope

Before the architecture, three product decisions defined the shape of everything that followed.

One-platform-per-country deployment. SlotSense is designed to run as a single logical deployment per country, hosting many communities as tenants. This is the shape that makes the vendor-fee math work: the platform economics amortize across all tenants, so a single community never has to pay the true cost of building this in-house. It also matches how the data-sovereignty story needs to look — an Indian deployment stays in Indian regions, keeping DPDP compliance straightforward. When SlotSense expands to a new country, it is literally a new deployment, not a new tenant configuration on the same substrate.

Postpaid, admin-configured billing — never a payment processor. When I built the billing pipeline (Phase 15), the tempting scope creep was to become a payment processor. I deliberately didn’t. SlotSense generates immutable invoices — deterministic IDs so re-runs are idempotent, integer paise so there are no floating-point money bugs, household-level rollups derived at generation time from actual bookings — and the entire question of payment status is external. The community handles collection through whatever channels they already use; SlotSense provides the source-of-truth line-items. This kept the regulatory surface area tiny and the product boundary crisp. It also meant I could ship it in a single phase.

Product retention over feature breadth. Every phase’s scope was sized to close a specific vendor-fee gap or unlock a specific tenant capability, not to check a competitive-feature box. That’s why voice I/O (Phase 16) shipped before push notifications; residents can already talk to the assistant in text, but a resident on a WhatsApp-and-voice diet gets a completely different experience from a talk-to-book assistant than from a fifth notification-preferences pane. The roadmap is ordered by user value against a specific persona, not by feature list.

Requirements — where the discipline starts

Every architectural decision in SlotSense traces back to something in docs/REQUIREMENTS.md. The document is the canonical scope of the project, reconciled against actual state at each phase closure. It is not a wish list — it captures what is committed, distinguishes it from what is deferred, and every deferral cites the ADR that documents the deferral rationale.

The requirements are split three ways:

Functional. Booking, cancellation, facility catalog, per-tenant schedules, household-based user provisioning, admin surfaces, notification pipelines, the AI booking assistant, voice I/O, billing and invoicing, DPDP-compliant entity erasure.

Non-functional (product quality attributes). Tenant isolation without exception, availability at a formalized 99% monthly SLO (ADR-0041), disaster recovery at 4-hour RTO/RPO (ADR-0038), zero long-lived credentials outside Google-managed services (ADR-0018), all infrastructure state in Terraform, DPDP compliance for personal data, accessibility at WCAG 2.1 AA on every user-facing page, a cost baseline of ≤₹5K/month in dev and ≤₹2K/tenant in production (ADR-0005, ADR-0042).

Operational. ADRs before code. Live verification on every phase, not just tests. Slice-level CHANGELOG discipline. Phase retrospectives. Branch protection on main. No secret-bearing files anywhere in the repository, tree, or history.

The point of writing requirements this way — with the non-functional attributes given the same rigor as the functional ones — is that PQ attributes are the ones that decide whether the product survives production. A booking flow with 100% feature coverage but no backup strategy is not a shippable product. A billing pipeline without idempotency will corrupt itself on the first retry. Naming these things up front, and holding each phase to them, is what keeps the platform from drifting into “works on my machine.”

Architecture — depth in defense

Five-layer tenant isolation (ADR-0004)

The platform serves many communities (tenants) from one deployment. The risk that makes this hard isn’t performance — it’s the catastrophic outcome where one tenant ever sees another’s data. I architected five independent layers of isolation that all have to fail simultaneously for a leak:

1. Deny-all Firestore rules. The Firestore Security Rules deny everything by default. Read paths are enumerated explicitly and are always keyed on tenant. Even a compromised backend cannot read data outside its tenant path — the database refuses.
2. Repository pattern that requires TenantContext at construction. Every Firestore query in the backend goes through a repository object whose constructor demands a TenantContext. There is no code path that can query Firestore without a resolved tenant.
3. JWT-vs-subdomain cross-check middleware. Every request’s JWT tenant_id claim is checked against the subdomain the request came in on. Mismatches are rejected before any handler sees the request. This closes the class of attack where a valid token for tenant A is replayed against tenant B’s subdomain.
4. Automated cross-tenant tests. The test suite has explicit negative tests that attempt cross-tenant reads and writes. They must fail. A regression that opens a cross-tenant path breaks CI loudly.
5. CI static-analysis gates. A CI job forbids specific patterns that could bypass the repository (raw Firestore client calls, for instance). This catches the human-error case where someone tries to write “just one quick query” outside the safe abstraction.

The cost of this depth is real — every layer adds verification overhead and code complexity. But the failure mode it protects against is severe enough that depth is the only honest answer. Single-layer isolation is how every multi-tenant SaaS data breach starts.

AI agent safety — three ADRs that matter (0023, 0024, 0026)

The platform’s AI booking assistant runs on Vertex AI Gemini 1.5 Pro with function calling. Residents can say “book my usual tennis slot tomorrow at 7 PM” or “cancel my badminton on Friday.” Every mutation the agent proposes passes through three fail-closed guards.

The propose-confirm-execute gate (ADR-0023). The LLM never directly mutates state. It proposes an action via a structured tool call; the backend validates it deterministically (does the facility exist? is the slot bookable? does the resident have quota left?); a single-use pending action is written to Redis with a 5-minute TTL; the user sees a confirmation card; only on an explicit user confirmation does the backend execute the mutation — with the originally-stored parameters, verbatim, ignoring anything the LLM might hallucinate on the confirmation turn. The pending-action key is consumed on read (GET + DEL), so replay can’t double-book.

The output guard (ADR-0024). A second Vertex call validates that every entity reference in the natural-language reply actually exists for the current tenant. If the model’s reply mentions “Court 3” but the tenant doesn’t have a Court 3, the reply is failed closed rather than shown. This defends against the specific failure mode where a model hallucinates a plausible entity that doesn’t exist — the class of error that would otherwise slip past a human reader.

Deterministic Python guards (ADR-0026). Anything the LLM is demonstrably unreliable at — temporal reasoning (“tomorrow” resolved to the tenant’s timezone, not UTC), quota arithmetic, disambiguation matching between similar-looking cancel requests — is handled in deterministic Python. The LLM’s job is intent extraction and parameter proposal; the accountable logic runs in code.

These three ADRs collectively encode a principle I care about deeply: an LLM is a structured-input parser, not a mutation engine. Everything about the agent’s design flows from that. When the Phase 15 billing work exposed a Gemini tool-selection reliability bug — the model would occasionally choose the wrong invoice-lookup tool on ambiguous queries — the fix was not to prompt-engineer harder. It was to add a deterministic pre-Vertex router in front of the read-only invoice tools, extending the same pattern. The pattern generalizes.

Voice I/O — extension without core changes (ADRs 0036, 0037)

Phase 16 added voice input and output to the AI booking assistant. The architectural constraint I set for myself: zero changes to the agent core. The existing text agent, with its propose-confirm-execute gate and its deterministic guards, had to remain the entire mutation path. Voice had to happen at the edges.

The design (ADR-0036) does exactly that. Google Cloud Speech-to-Text (Chirp_2 at asia-southeast1) transcribes the resident’s speech to text at the edge; the resulting text enters the same /agent/query path the text UI uses; the response comes back as text; and Google Cloud Text-to-Speech (Chirp3-HD Kore in en-IN at the global endpoint) synthesizes the response for playback. Nothing about the agent’s safety properties changed — the LLM still doesn’t mutate state, the pending actions still expire in 5 minutes, the output guard still runs.

There are two implementation lessons worth surfacing:

Real-world model availability is a real-world hazard. ADR-0036 originally specified chirp_3 STT with 9-language auto-detect. During Phase 16 rollout, chirp_3 was withdrawn from the API mid-implementation. ADR-0037 superseded ADR-0036 §D3 with a per-tenant candidate-language set on chirp_2, and carried a documented residency exception into the Phase 16 DPDP self-assessment. The lesson: an ADR is not permanent, and GA-revocation of a managed API is a scenario your architecture has to be able to handle without a rewrite. In this case it took an ADR revision and a small code delta, because the boundary was clean.

Deterministic confirm/deny extended cleanly. The propose-confirm-execute gate already required an explicit user confirmation. Voice’s confirm/deny guard is just another instance of the deterministic-Python pattern (ADR-0026): “yes”/“confirm”/“do it” and “no”/“cancel”/“never mind” are matched by code, not by another LLM turn. The gate stays reliable, and voice replies feel snappy because the confirmation loop doesn’t need a model round-trip.

Booking-model v2 — real-world schedules (ADR-0030)

The initial booking model assumed a facility had a single open/close window per day (say, 6 AM to 10 PM). Real operational patterns don’t work that way — a community pool may be open 6–9 AM, closed for maintenance and midday sun 9 AM–4 PM, then open 4–10 PM. ADR-0030 replaced the single-range model with multi-range per-day-of-week schedules. Availability computation still runs deterministically at query time (no pre-generated slot documents to keep in sync), but the schedule model now matches how communities actually run their facilities.

This kind of change is the value of computed availability: the schema change is local to the schedule document, and the query-time availability function absorbs the new range shape without a data migration. The abstraction was the right one from the beginning (ADR-0010); v2 is what happens when reality tightens the shape.

Billing & invoicing (ADR-0035)

Phase 15 shipped the full billing pipeline in one phase. Notable decisions:

• Optional flat-rate per-facility pricing, in integer paise. Floating-point money is a class of bug I refuse to ship. All monetary values are integers. Formatting for display happens at the render layer.
• Postpaid monthly invoicing via Cloud Scheduler. Fixed billing day per tenant, admin-configurable time only. This is dramatically simpler than a prepaid or hybrid model, and it matches how community treasuries actually collect.
• Immutable create-only invoices with deterministic IDs. An invoice document never changes after creation. The ID is a deterministic function of (tenant, household, billing period) so a scheduler retry or an admin’s manual recovery trigger cannot create a duplicate. Idempotency without a dedup table.
• Household-level billing, resolved directly from bookings. The invoice line-items are derived from the booking log at generation time, with flat_number and resident_name denormalized into the invoice snapshot. The invoice remains readable and accurate even if the household composition changes later.
• Payment status is external. SlotSense never tracks whether an invoice was paid. That is the community treasurer’s system’s job. This is the product-mindset boundary I refused to blur.
• Keyless GCS export via self-impersonated signed URLs. Tenant admins can download month-end invoice summaries as CSV. The GCS export is generated with the Cloud Run service account self-impersonating a short-lived signing identity, so no signing key ever exists as a persistent artifact. The export is summary-level only — the invoice document in Firestore remains the system of record.
• Read-only agent invoice tools, deterministic pre-Vertex routing. Residents can ask the agent “what’s my invoice for June?” The tool the agent uses is read-only; the routing to it is deterministic Python in front of Vertex, so tool-selection is not a Vertex judgment call for these queries. This was the direct fix for a live-reproduced Gemini reliability bug.

Five bugs in this phase were caught by live verification, not the test suite. That is a datapoint I take seriously: tests are necessary but not sufficient at the seams where deterministic code meets a probabilistic model and a stateful external scheduler. Phases now close on live verification, not on green CI alone.

Entity lifecycle & DPDP-compliant erasure (ADR-0034, ADR-0017)

Phase 13 redesigned entity deletion and deactivation. The initial design (ADR-0017) had a three-stage lifecycle (ACTIVE → INACTIVE → PURGED) with a 90-day PURGE script. In implementation, a real production incident during the phase drove a key design reversal: for on-demand erasure, a direct independent Delete action beats a staged lifecycle for both usability and DPDP compliance.

ADR-0034 supersedes the on-demand slice of ADR-0017. Residents, tenant-admins, and tenants themselves can Delete directly. The action is independent (not sequenced through a Deactivate stage). Audit trail is preserved. Invoice history is excluded from the erasure via a specific carve-out (invoices are the treasury’s system-of-record record and are retained on a separate schedule). The facility lifecycle is Delete-only — no Deactivate or PURGED stage — because a facility that stops being used just goes away.

DPDP (India’s Digital Personal Data Protection Act) is not an afterthought here. The erasure model was designed to satisfy the regulation as a first-order requirement, and the invoice-exclusion carve-out is explicit and documented.

Non-functional posture — the PQ attributes made concrete

The functional architecture is the visible part. The non-functional posture is what makes the platform survivable in production. Phase 17 is the phase dedicated to formalizing every one of these attributes.

Security (ADRs 0018, 0022–0027, 0032, 0043)

Zero-key-management, as a property of the whole system. Every secret-bearing pattern in SlotSense was deliberately replaced with something that doesn’t need a secret:

• CI/CD authenticates to GCP via Workload Identity Federation. No service account keys are downloaded. GitHub Actions exchanges its OIDC token for short-lived GCP credentials on every workflow run.
• Service-to-service auth uses OIDC. Cloud Tasks calling worker endpoints signs with its own service account identity; the worker validates the OIDC token. No shared secrets between components.
• Firebase ID tokens for user auth. Short-lived (1h), refreshed by the client SDK. Refresh tokens never touch the backend.
• Cloud KMS for any signing material. Managed rotation, no key material in code.

The principle: every secret that doesn’t exist is a secret that can’t leak.

AI agent safety as a security property. ADRs 0022–0027 collectively document the guardrails discussed above. From a security lens, the key property is that the LLM cannot mutate state directly — so an adversarial prompt cannot cause a state change. The worst case is a rejected proposal.

Cloud Armor WAF (ADR-0032, ADR-0043). Edge and API WAF policies started in log-only preview mode to gather real traffic data before enforcement. Phase 17’s PR-5c moved the API-policy SQLi/XSS rules into enforce, with a documented voice-path exemption (VOICE-INPUT-VALIDATION) tracked as an accepted residual for the next phase. The voice endpoint accepts arbitrary audio, which triggers false positives on the injection rules; the appropriate defense is input validation at the STT/TTS boundary, not at the WAF layer.

Security hardening split by blast radius (ADR-0043). Security hardening was deliberately split into three PRs by blast radius:

• PR-5a: HTTP security headers, CI secret scanning, container registry cleanup, credential rotation policy — low risk, wide surface.
• PR-5b: WIF least-privilege (tightened GitHub Actions’ GCP IAM to the minimum necessary), Cloud Armor edge-policy enforce mode.
• PR-5c: Cloud Armor API-policy enforce mode, with the documented voice-path exemption.

Splitting this way meant a rollback in one PR wouldn’t undo the others.

Accepted residuals, documented (ADR-0039). CMEK for Firestore, VPC

• NAT for Cloud Run egress, admin MFA, and third-party penetration testing are all deferred — not silently, but explicitly, as an ADR with dated revisit triggers (tenant count threshold, first paying customer, first regulated-industry tenant). Documenting deferrals as first-class decisions is what separates “we haven’t gotten to it” from “we made a call, here are the tripwires that reopen the call.”

Availability & SLO (ADR-0041)

The platform has a formalized 99% monthly availability SLO for the API surface. Phase 17’s PR-3 codified the operational posture behind that number:

• Cloud Run maxScale raised from 2 to 10. Two instances was fine for early testing; ten gives the tail latency headroom to survive a brief burst without cold-start dominating p95.
• TCP → HTTP startup probe, plus a liveness probe. The TCP probe couldn’t distinguish “port open” from “app healthy.” The HTTP probe hits /readyz, which validates the process is actually ready to serve traffic. A separate liveness probe restarts an instance that’s in a bad state without ever having been ready.
• “SlotSense Ops” dashboard, Terraform-managed. Not a hand-clicked dashboard that drifts. Every widget is code.
• The Redis-tier decision, documented (ADR-0041). Memorystore Redis is on the BASIC tier. STANDARD_HA would give automatic failover but doubles the cost and doesn’t remove the single-tenant lock behavior we actually need to worry about. The residual is documented with revisit triggers.

The SLO itself is currently formalized at the document level; the Monitoring SLO API resources are gated behind a SLO-LOAD-TEST item, so they get created after a real load-test run rather than being hand-calibrated. This is the discipline: don’t ship an SLO alert whose threshold is a guess.

Disaster recovery (ADR-0038)

The DR runbook is a six-layer restoration procedure at 4-hour RTO and 4-hour RPO, documented as an ADR and rehearsed in a timed drill.

The six layers, each independently restorable:

1. Firestore — Point-in-Time Recovery enabled, delete protection on, daily backup schedule. RPO is bounded by the PITR window.
2. Secret Manager — replicated in Google-managed regions; the secret material is regeneratable from Terraform on a full rebuild.
3. Terraform state and modules — every piece of GCP infrastructure is in code. A rebuild is terraform apply, not “let me remember what I clicked.”
4. GCS — object versioning and lifecycle rules.
5. Container images — Artifact Registry with retention; the images are also reproducible from the repository tags.
6. Firebase Auth — export/import procedure documented, user directory recoverable from the last successful export.

The critical enabler for layer 3 was Phase 17’s PR-1a and PR-1b, which codified the six service accounts, IAM bindings, Cloud Run services, Redis instance, and Artifact Registry that had previously been imperative (the oldest HIGH item in the project backlog). Import blocks were used over CLI-based imports — a lesson banked into the engineering protocol after the fact.

Observability & alerting (ADR-0040)

• Two notification channels — email and console-owned SMS. Two so a single-channel outage never silences the alerting.
• Four alert policies — 5xx rate, p95 latency, uptime failure, backup failure. Each is bounded to a real user-visible symptom, not an internal metric that would go off without any user impact.
• Two uptime checks — one at the edge (through the LB), one at the service path (bypassing the CDN). If the two disagree, the disagreement is itself a signal.
• Error Reporting enabled — errors are surfaced through Google-native tooling rather than a bespoke email pipeline.
• Voice/agent turn-counter log-based metrics — because voice and agent throughput are the two most-recent surfaces, they get their own log-based counters so any regression shows up on the dashboard.

Everything is Terraform-created. No console-clicked resources. Drift between prod configuration and repo state is the top-level failure mode Phase 17 spent effort preventing.

Cost guardrails (ADR-0005, ADR-0042)

The project has had a cost baseline since day one (ADR-0005): dev at ≤₹5K/month, production target ≤₹2K per tenant. Phase 17 formalized the enforcement (ADR-0042):

• One Terraform-managed google_billing_budget for the sport-slot-dev project.
• Five graduated alert thresholds — 50%, 80%, 100%, 120% actual and 100% forecasted. Alert-only. Routed to the ADR-0040 channels.
• No automated billing-disable or service-cap. Automated shutoffs are explicitly rejected — the failure mode of accidentally turning the platform off for a legitimate traffic spike is worse than the failure mode of a temporary cost overrun.

Cost is a first-class NFR, not a monthly billing surprise.

Performance — the Redis choice, revisited

The original decision (ADR-0009) was Redis-backed distributed locks for booking contention. The choice of Redis over a Firestore transaction was deliberate: Firestore transactions have read-modify-write limits that don’t compose well with the multi-step booking flow (quota check, availability check, write, audit), and Redis gives sub-millisecond lock acquisition that’s effectively imperceptible to users while bulletproof under contention.

Two years and many phases later, Redis is now doing three jobs: distributed locking, the pending-action store for the AI agent (ADR-0025), and the cancel-disambiguation cache (ADR-0027). The single-instance BASIC tier is honest about the availability tradeoff (ADR-0041). If a booking-latency SLO or a real HA requirement ever demands STANDARD_HA, the tier upgrade is a Terraform-only change; the application code doesn’t need to move.

Usability — the visible product

PWA-first delivery — one codebase, every device

SlotSense is delivered as a Progressive Web App. The same React/TypeScript codebase serves the desktop web experience and installs as a native-feeling app on iOS and Android home screens — with offline detection, app-shell architecture, full-viewport mobile layout (100dvh, keyboard-aware), and 44pt+ tap targets throughout.

This was a deliberate alternative to native iOS/Android development:

• One codebase, three platforms. No separate Swift / Kotlin codebases; no separate test pipelines; no separate review-and-release cycles. Updates land everywhere on the next page load.
• No app-store gatekeeping. Critical for a B2B SaaS where each tenant rolls out independently and updates need to be deployable on demand — not on Apple’s or Google’s review timeline.
• Free distribution. No developer-program fees, no certificates, no provisioning profiles.
• Genuine mobile UX. Service workers for offline behavior, install prompts for home-screen placement, full-screen standalone display. Modern PWA capabilities close most of the historical gap with native apps.

The chat UI for the AI assistant was specifically designed mobile-first — 100dvh instead of 100vh for correct iOS keyboard behavior, flex-anchored input bar that tracks the on-screen keyboard, 44pt minimum tap targets on every interactive element. A resident pulling out their phone to book a court works exactly as well as a desktop user.

For B2B SaaS targeting residential communities — where the user base is largely on mobile and the tenant rollout cadence is independent of app-store timelines — PWA is structurally the right delivery model.

Design system, accessibility, and co-branding (ADR-0028, ADR-0029)

Phase 10 was a dedicated pass at UI quality. Two ADRs came out of it.

Frontend design system and theming (ADR-0028). The frontend adopted Tailwind v4, shadcn/ui, and Radix primitives, with Inter as the typeface and light/dark mode as a first-class feature (dark mode’s FOUC was fixed during the phase — a specific production bug caught before it ever mattered). 238 frontend tests across 37 files cover the visible surface. The design system replaced the earlier hand-rolled CSS approach; the migration was staged and every migrated screen was verified.

PWA co-branding hierarchy (ADR-0029). Each tenant has its own logo, color palette, and name, applied via CSS variables at runtime — the tenant brand is the primary identity everywhere in the header. The platform brand (“powered by SlotSense”) is a secondary footer presence. This is an explicit product decision: SlotSense is the platform a community chooses; the community’s identity is what its residents see.

Verified accessibility. Phase 10 ran 28 automated axe-core scans (14 pages × light/dark modes) with zero serious or critical violations. Keyboard navigation was verified explicitly. The ConfirmDialog (the booking-confirmation component central to the agent flow) has a verified focus trap. Slot availability states are never color-only — they always have text labels alongside the Radix state indicators.

Voice I/O (Phase 16) is also an accessibility win. A resident who finds touch input hard, or who prefers Telugu or Hindi to English text, can just talk to the assistant. That was always in the medium-term vision; Phase 16 is what shipped it.

Per-tenant branding in the real world

Per-tenant branding is easy to get nearly right and hard to get completely right. The nearly-right version leaks the platform brand in the wrong places (email templates, error pages, browser tab titles). SlotSense treats these as first-class — every email template is tenant-brandable, every browser tab title uses the tenant name, and the service worker cache is keyed per tenant so a stale asset for tenant A never appears on tenant B’s subdomain. Phase 8b’s wildcard-subdomain routing was the enabler for making all of this work at *.slotsense.chandraailabs.com.

Process — how a solo engineer keeps a production platform honest

The three-agent protocol

I worked with Claude (Anthropic’s AI assistant) as a technical partner using a Three-Agent Protocol, now on its v3.8 revision after seventeen phases of production use:

• Strategist (Claude in conversation) — architectural design, tradeoff analysis, code review, prompt-writing for the Worker, ADR drafting.
• Worker (Claude Code in the CLI) — disciplined execution: opens PRs, runs tests, follows specifications precisely, never invents scope.
• Coordinator (me) — final review, merge authority, deployment, credential handling, and every architectural decision that matters. Every CLI operation that touches credentials runs from my terminal, never the Worker’s.

The protocol has been revised thirteen times based on real failure patterns caught in real work — most recently v3.8, which banked four new failure patterns from Phase 17 and made Terraform import blocks the default over CLI imports (the shared-interface ripple pattern from v3.6 is why “passing tests” is no longer accepted as a clean type-check signal; a phase-9 lesson about the Worker’s narration being unreliable is why every completion claim is independently verified).

The protocol is not “AI built my project.” It is a specific division of labor that keeps the human in the loop where judgment matters, keeps the AI in the loop where consistent execution matters, and holds both sides to the same rigor: no unverified completion claims, no ADR-less code, no secret-bearing artifacts ever committed.

This division let me ship 43 ADRs, 143+ merged PRs, and 17 phases across the project while maintaining the quality bar I’d expect from a senior engineering team. The CHANGELOG.md is 240 KB of slice-level history; every material change is traceable.

Live verification, always

Every phase closes on live verification, not just on green CI. Phase 15 (billing) alone caught five production-only bugs that the test suite missed — race conditions in the Cloud Scheduler retry path, signed-URL expiry surprises, and the Gemini tool-selection reliability issue that became the deterministic-router change. Phase 16 (voice) caught six stacked production failures during live testing (a missing Firebase key in CI, a Pydantic environment-prefix naming issue, an IAM permission denial, and three others). Phase 8b (networking) caught seven — a Certificate Manager classic-cert limitation on wildcards, a Cloud Armor WAF naming issue, a CDN cache mode gotcha, a GCS bare-path XML fallback.

That is thirty-plus real bugs, each caught before an actual user hit it, only because “phase-complete” doesn’t count until live verification passes. The tests are necessary but they are not the definition of done.

ADRs before code

Every significant architectural decision is captured as an ADR before implementation begins. Forty-three so far. The ADR captures context, options, decision, rationale, consequences — the same shape a mature engineering organization uses. Superseded ADRs stay in the tree; the supersession is documented as a link. The history is the point: someone reading the repository in a year should be able to reconstruct not just what the code does, but why it looks this way and what alternatives were rejected.

Some ADRs supersede others because reality reshaped a design:

• ADR-0037 supersedes ADR-0036 §D3 after Google withdrew chirp_3 STT during Phase 16.
• ADR-0034 supersedes ADR-0017 for on-demand erasure after a Phase 13 production incident.

Both are honest documentation of the fact that a real system’s design evolves, and the discipline is not to pretend otherwise.

The retrospectives

Every completed phase has a retrospective in docs/retrospectives/. The Phase 9 retrospective in particular seeded eleven protocol-level lessons that fed the next revision of the three-agent playbook. The Phase 10 retrospective covered the density saga (a UI-density debate that went several rounds), deploy/cache lessons, and the a11y audit methodology. The Phase 8b retrospective walked through each of the seven networking issues caught before harm. These are not marketing documents. They are engineering artifacts for future me.

Tech stack

The deliberate cost-optimization throughout: scale-to-zero on the backend, no managed compute for batch work, free tiers exploited where they exist (Firebase Auth, Firebase Hosting, Cloud Tasks baseline), Gemini Pro chosen for the correctness-sensitive agent path, PWA delivery avoiding native-app development cost entirely. The platform is engineered to remain viable at low tenant counts and scale economically to thousands.

What I learned

A few things I went into the project assuming and came out the other side reconsidering.

Stateless agent designs work harder than they look. I assumed I’d need conversation history for the agent to feel natural. I deliberately chose a mostly-single-turn architecture instead to keep the safety properties simple. The cost is some conversational awkwardness (“which date?” “tomorrow.” “tomorrow what?”). The benefit is no class of replay attacks, no token-window management, no cross-session state to leak. Voice made the single-turn choice even easier to justify — a voice user tolerates crisp confirmation more than a text user tolerates prompting. The right tradeoff for v1 was clear in retrospect; it wasn’t obvious going in.

Defense in depth has compound costs. Each isolation layer is small in isolation but the testing matrix grows multiplicatively. Five layers of tenant isolation means five places to keep in sync when adding a new feature, five paths to test, five different debug locations when something goes wrong. The discipline is worth it for the security properties — but I now have a real sense of why mature SaaS products often compromise on one or two layers. Documenting the depth as a first-class NFR is what keeps the temptation to shortcut at bay.

The “right” abstraction often emerges from refactoring. Several major architectural improvements (the services/ layer that decouples routers from business logic, the agent’s tool dispatcher pattern, the policy service, the pre-Vertex deterministic router for read-only tools) came from extracting them from cluttered code rather than designing them upfront. I now distrust elegant up-front designs that haven’t survived contact with real implementation.

LLM tool routing reliability has a ceiling. Even with careful prompt engineering, current LLM tool-routing accuracy on common queries hovers around 80–90% — there’s no prompt that gets you to 100%. Production AI integrations have to design around this with graceful fallbacks, retry affordances, and structured-input parsing that doesn’t rely on the LLM “getting it right.” This shaped how I built the agent’s confirm gate and, later, the deterministic pre-Vertex router for the read-only invoice tools. The pattern generalizes.

Live verification catches what tests can’t. Thirty-plus real bugs across three phases (Phase 8b, Phase 15, Phase 16) were caught by live verification, not the test suite. Test suites are the necessary foundation; live verification is the honest ceiling. A phase is not done until a real request has traveled the real production path.

“Deferred” is a first-class engineering artifact. ADR-0039 formalizes CMEK, VPC+NAT, admin MFA, and pen testing as deferred with dated revisit triggers. This turns out to be more useful than shipping them all half-well before the platform is ready — and infinitely more useful than pretending they aren’t gaps. Naming the deferral as a decision, with a trigger to revisit, keeps the backlog honest.

A managed API can vanish under you. chirp_3 STT was withdrawn from the Vertex API during Phase 16 implementation. ADR-0037 handled it in a superseding decision. The lesson is architectural: build the integration with a clean boundary, so the fallback is a small delta and an ADR, not a rewrite.

The three-agent protocol scales with the project. v3.8 is not v1.0 with more rules; it is v1.0 with thirteen banks of banked failure patterns. The protocol works because it is willing to change. That mirrors the ADR discipline: designs are not immutable, and neither is the way you build.

Status and roadmap

Now — Phase 17 (Production Readiness). Backup and disaster recovery foundations are in place (PR-1a, PR-1b). Terraform codification of the service accounts and IAM bindings closed the project’s oldest HIGH backlog item. Observability (PR-2), SLO/Redis (PR-3), cost guardrails (PR-4), and security hardening (PR-5a–c) are the remaining substream; several ADRs are already accepted and the code work is in flight. A timed DR drill closes PR-1.

Next — Phase 18 (Production Launch Gate). The gate consolidates the outstanding operational and DR verifications, then flips the platform to its first real tenant. Binary Authorization is scheduled to land here.

After — Phase 19 (Voice-ML). Extending the voice path with per-tenant model tuning and richer disambiguation. The clean boundary Phase 16 preserved is what makes Phase 19 an extension rather than a redesign.

Phase 8’s original “Binary Auth + supply-chain security” slot never shipped in that slot; that scope is what Phase 17 (with ADR-0043) and Phase 18 (Binary Authorization) are now closing, deliberately.

Code

https://github.com/chandranakkalakunta/slot-sense — open source for the architecture, ADRs, and process artifacts. Every ADR, every phase retrospective, every runbook is in the repository. The live deployment is private to my own dev tenant.

Portfolio site: https://chandraailabs.com Contact: Chandra AI Labs — see the site.

Written July 2026, aligned to the repository at Phase 17. Supersedes the June 2026 SlotSense writeup (current through Phase 9).