1. Business & Users
- What exact business outcome are we driving?
- Who is the user? (Internal employee, developer, external customer)
- Autonomy: Advisory (recommends), Decision-Supporting (prepares), or Autonomous (executes writes)?
Viewing Playbook — 0–60 min structure
Opening script (0–2 min)
I’ll structure this system design into three explicit phases: Scope & Discovery: I will define the business problem, user personas, scale math, and data security boundaries. MVP Architecture: I will map out the leanest core loop using a single-agent pattern with human-in-the-loop controls to establish immediate value. Production & Scale: I will scale the design into an enterprise system—detailing multi-agent decomposition, isolated data layers, security guardrails, resilience, observability, and evaluation metrics.
Phase 1 — Scope & Discovery
Drive the interview. Cover all five dimensions before drawing boxes.
2. Data Landscape
- Where is the source of truth? (On-prem EHR/ERP, Cloud DBs, SaaS APIs)
- Data types: structured, unstructured (PDFs/docs), or streaming?
- Sensitivity: PII, PHI, PCI, or internal IP?
- Residency: GDPR, HIPAA, sovereign cloud boundaries?
3. Scale Math
- Registered users vs Daily Active Users (DAU)
- Average RPS = (DAU × Requests/Day) / 86,400; Peak ≈ 10× average
- LLM bottlenecks: average turns/session and token footprint (in/out)
4. Non-Functionals
- Latency: end-to-end (e.g. p95 < 30s) vs first-token TTFT (< 1.5s)
- Availability: SLO target (e.g. 99.9% uptime)
5. Success Metrics
- Business: task completion rate, AHT reduction, cost per completed task
- AI quality: groundedness, citation accuracy, tool-selection precision
Phase 2 — MVP Core Loop
Defend single-agent first. Split only when a hard boundary is crossed.
Lean MVP: API Gateway → Agent Runtime (Cloud Run) → Gemini plan → tools → grounded response, with HITL on any write that changes real-world state. Session state stays outside the compute layer.
Multi-agent decision framework
Security / Permission Boundary
Sub-tasks need different IAM levels, tenant isolation, or data residency domains.
Domain Complexity Boundary
System-prompt bloat and tool count (>10 tools) degrade reasoning accuracy.
Latency / Execution Boundary
Independent sub-tasks can run in parallel on specialized smaller models.
Phase 3A — Isolated Data & RAG
Never lump everything into “the database.” Separate by lifecycle and access pattern.
| Layer | Purpose | GCP stack |
|---|---|---|
| Session State | Active interaction state, step logs, short-term turn history | Memorystore (Redis) / Firestore |
| Long-Term Memory | Identity-scoped preferences and cross-session facts | Memory Bank / Firestore |
| Enterprise RAG | Unstructured knowledge — policies, runbooks, protocols | Vertex AI Search / Vertex Vector Search |
| Operational DB | Authoritative transactional SoR — billing, accounts, inventory | Spanner / AlloyDB / Cloud SQL |
Dual-path RAG — batch ingest
- Pull docs from source repositories
- Document AI parse → DLP redact
- Section-aware chunking (not blind token cuts)
- Vertex embeddings → Vector Search / Vertex AI Search
Dual-path RAG — online retrieve
- Query with tenant + ACL filters
- Hybrid / semantic retrieval + rerank
- Pass chunks as grounded context only
- Validate citations against retrieved IDs
Phase 3B — Five Guardrail Checkpoints
Security is a vertical cross-cut — not a bolt-on.
Checkpoint 1 · User Input
- Authentication & token validation (Apigee / Firebase Auth / Identity Platform)
- Prompt-injection detection (Model Armor)
- Input PII/PHI/PCI redaction (Sensitive Data Protection / DLP)
Checkpoint 2 · Retrieval
- User-level ACL filtering at query time
- Multi-tenant isolation via tenant-ID metadata checks
Checkpoint 3 · Tool Execution
- Least-privilege service accounts with identity delegation
- Tool schema validation & input sanitization
- Action tiering: READ → auto; LOW-RISK WRITE → policy + confirm; HIGH-RISK WRITE → human approval + immutable audit
Checkpoint 4 · Model Output
- Groundedness verification against RAG / tool context
- Citation validation (sources must exist in retrieved context)
- Output leakage prevention (Model Armor for PII/secrets)
Checkpoint 5 · Post-Action
- Idempotency keys to block duplicate tool calls
- Immutable audit logging (Cloud Logging → BigQuery)
Phase 3C — Scale & Resilience
Enterprise failure modes, not just happy-path boxes.
Stateless compute
Agent runtime on Cloud Run or GKE Autopilot — horizontal scale on concurrency/CPU.
Async tool queueing
Cloud Tasks / Pub/Sub for tools >5s so HTTP stays non-blocking.
Circuit breakers
Trip on downstream error/latency; degrade to cached or RAG-only responses with explicit gaps.
Backoff + jitter
All LLM and tool calls retry with exponential backoff to absorb 429s.
Rate limits & quotas
Per-tenant limits via Apigee to protect shared LLM quota.
Semantic cache
Exact/similar query matching via Vector Search to skip LLM for frequent static asks.
Phase 3D — Observability & Dual-Track Eval
Log trajectories, not just inputs and outputs.
Trajectory stack
- Cloud Trace — runtime, LLM steps, vector search, tool calls
- Cloud Logging — session_id, turn_id, tokens, tools_called, policy_decision, guardrail_flags
- Cloud Monitoring — token throughput, cost/session, tool failure rate, TTFT
Dual-track evaluation
- Offline — Golden Dataset; groundedness, task completion, safety flags
- Online — shadow mode → sampled human review → prompt/router updates
- Gate — only promote when both tracks meet thresholds
Closing summary (min 58–60)
To summarize: We designed an enterprise agentic system that balances speed to market with enterprise compliance. We started with a clean, single-agent MVP on Cloud Run to validate core business metrics. To scale to production, we split responsibilities across specialized sub-agents, isolated our storage into explicit Session, Memory, RAG, and Operational layers, and placed a 5-point security guardrail using Model Armor and Sensitive Data Protection. Finally, we protected system reliability using asynchronous queues and circuit breakers, while maintaining full visibility into the agent's trajectory using Cloud Trace and dual-track evaluation datasets.