Skip to content

Observability Implementation Plan

Field Value
Status Draft
Author Backend / Platform
Created 2026-04-30
Last updated 2026-05-01
Target start 2026-05-01 (Stage A)
Scope apps/backend (FastAPI), all LLM call paths, downstream services
Topology App stays on GCP Cloud Run. Observability infra on Azure. Cross-cloud accepted — see §3.
Related docs/plans/RFC-001-kwilo-memory.md (memory service will share this stack)

1. Why this exists

Kwilo's backend has 500+ FastAPI endpoints, 107+ DB tables, and three LLM providers (Gemini primary, Claude + GPT-4o fallbacks) with no centralised view of:

  1. Where errors come from. Today error attribution requires SSH'ing into Cloud Run logs and grepping. There is no error-rate-by-endpoint dashboard, no alerting, no MTTR signal.
  2. What is slow. No latency distribution by endpoint, no DB-query timing, no LLM call timing. Performance regressions ship undetected.
  3. LLM cost and behaviour. No per-tenant token tracking, no per-model cost split, no prompt versioning, no eval harness. Today's cost figures come from each provider's billing console weeks later.

This plan introduces a single observability stack on Azure that delivers logs, traces, metrics, and LLM-specific telemetry together, with a vendor-neutral instrumentation layer (OpenTelemetry) so backend choices stay reversible.

2. Goals and non-goals

Goals

  1. P95 latency, error rate, and request volume per endpoint, available within 60 seconds of a request.
  2. Distributed traces from FastAPI handler → SQLAlchemy → asyncpg → downstream LLM call, in one timeline.
  3. Token usage, cost, prompt, and completion captured for every LLM call, queryable per tenant, per model, per endpoint.
  4. Alerts on error-rate and latency burns aligned to declared SLOs.
  5. Production cutover with no app-code lock-in to a vendor SDK.

Non-goals

  • Not a frontend RUM solution (out of scope for v1; revisit after backend stable).
  • Not replacing Cloudflare's edge logging.
  • Not a SIEM or security-event store. CloudFlare Access + GCP Audit logs handle that.
  • Not a full APM rebuild — we lean on Application Insights for general APM and add Langfuse only for LLM-specific work.

3. Architecture

Cross-cloud topology: FastAPI backend stays on GCP Cloud Run (no migration). Observability infra runs on Azure. To keep app-path latency in-region, the OTel Collector and LiteLLM proxy run on GCP Cloud Run alongside the backend. Only the Collector-to-Azure exporter and LiteLLM-to-Langfuse callbacks cross the cloud boundary, and both are async + batched.

┌─ GCP asia-south1 ──────────────────────────┐    ┌─ Azure centralindia ───────────┐
│                                            │    │                                │
│  ┌────────────────┐    ┌────────────────┐  │    │  ┌──────────────────────────┐  │
│  │  FastAPI app   │───▶│ LiteLLM proxy  │──┼────┼──▶ Gemini / Claude / GPT-4o │  │
│  │ (Cloud Run)    │    │ (Cloud Run)    │  │    │   (provider APIs)            │  │
│  │  + OTel SDK    │    │ + OTel exporter│  │    │  └──────────────────────────┘  │
│  └───────┬────────┘    └────────┬───────┘  │    │                                │
│          │ OTLP gRPC             │ OTLP    │    │  ┌─────────────────┐           │
│          └──────────┬────────────┘         │    │  │ App Insights    │◀──┐       │
│                     ▼                      │    │  │ + Log Analytics │   │       │
│           ┌─────────────────────┐          │    │  └─────────────────┘   │       │
│           │  OTel Collector     │          │    │                        │       │
│           │  (Cloud Run svc)    │──────────┼────┼──┐                     │       │
│           └─────────────────────┘   OTLP   │    │  │ traces/logs/metrics │       │
│                     │                      │    │  └─────────────────────┘       │
│                     │ Langfuse OTLP        │    │                                │
│                     │ (LLM spans)          │    │  ┌─────────────────────────┐   │
│                     └──────────────────────┼────┼──▶ Langfuse Web + Worker   │   │
│                                            │    │   (Container Apps)         │   │
│                                            │    │   ├ Postgres Flex          │   │
│                                            │    │   ├ ClickHouse Cloud       │   │
│                                            │    │   ├ Redis Cache            │   │
│                                            │    │   └ Blob Storage (S3)      │   │
└────────────────────────────────────────────┘    │  └─────────────────────────┘   │
                                                  └────────────────────────────────┘

Three layers: 1. Instrumentation — OpenTelemetry SDK in app code (vendor-neutral). 2. LLM telemetry — LiteLLM proxy emits OTel + Langfuse natively, replacing direct provider SDK calls. 3. Backend — Application Insights for general APM, Langfuse self-hosted for LLM-specific.

Why Collector + LiteLLM stay on GCP: keeps the synchronous path (app → LiteLLM, app → Collector OTLP receiver) in-region. Only the Collector's outbound batched export crosses the cloud boundary, where async + retry buffers absorb any cross-cloud hiccups.

4. Decisions to lock first (Phase 0)

Decision Default Owner
Backend compute GCP Cloud Run, asia-south1 (no migration) Platform
Collector + LiteLLM compute GCP Cloud Run, asia-south1 (in-region with backend) Platform
Azure region centralindia (closest to app, DPDP) Platform
Azure compute for Langfuse Azure Container Apps Platform
ClickHouse ClickHouse Cloud on Azure (managed, ~$100/mo dev tier) Platform
Object store for Langfuse Azure Blob Storage (S3-compat mode) Platform
Log retention 30 days hot, 90 days archive Backend
Trace sampling 100% errors + slow requests, 10% baseline Backend
PII redaction At OTel Collector, not in app Backend + DPO
Langfuse access gate Cloudflare Access, @kwilo.ai only Platform

Sign-off required before Phase 3.

5. Phases

Stage A — Local OTel proof (1 day, no Azure required)

  • Add OTel deps to apps/backend/pyproject.toml.
  • New module apps/backend/src/observability/{tracing.py,logging.py,redaction.py}.
  • setup_observability(app, engine) called from main.py, gated on OTEL_ENABLED env var.
  • docker-compose.observability.yml adds Jaeger + OTel Collector.
  • Verify a /health request produces a trace in Jaeger UI.

Exit criteria: any backend dev can docker compose up jaeger otel-collector, run the API, hit an endpoint, and see a trace.

Stage B — Application Insights (single shared resource for staging + prod)

Decision (2026-05-01): ONE Azure setup serves both environments. Traces are tagged with deployment.environment (already wired in Stage A) and filtered in App Insights / Log Analytics queries. Halves the bill, simpler IaC, single dashboard.

  • Terraform module in MySetu-AI/kwiloai_infra observability/ (PR #1):
  • Resource Group kwilo-observability-shared-rg (region centralindia)
  • Log Analytics workspace kwilo-observability-shared-logs (PerGB2018, 30-day retention, 5 GB/day cap)
  • Application Insights kwilo-observability-shared-ai (workspace-based)
  • Monitor Action Group kwilo-observability-shared-alerts (placeholder)
  • Backend code: new OTEL_BACKEND=azure_monitor mode in tracing.py using the Azure Monitor distro (already in pyproject.toml from Stage A). OTLP/Jaeger path retained for local dev.
  • Connection string lives in GCP Secret Manager (vidyanet-applicationinsights-connection-string) and is injected into Cloud Run via --set-secrets — backend stays on GCP.
  • Cloud Run env vars per env:
  • staging: OTEL_ENABLED=true, OTEL_BACKEND=azure_monitor, OTEL_TRACES_SAMPLER_ARG=1.0
  • prod: same but OTEL_TRACES_SAMPLER_ARG=0.1 (sample 10%)
  • Direct backend → App Insights for now (no Collector). Collector lands in Stage C, which adds redaction + the Langfuse fan-out path.

Exit criteria: real staging traffic produces traces in App Insights within ~60s; deployment.environment=staging filter narrows to staging-only; logs show trace_id correlated.

Stage C — OTel Collector on GCP Cloud Run (2 days)

Backend stays on Cloud Run. We deploy a dedicated Collector service in the same region to receive OTLP from the backend and fan out to App Insights.

  • New Cloud Run service kwilo-otel-collector in asia-south1 (same project as backend).
  • Image: otel/opentelemetry-collector-contrib:latest (pinned >=0.112).
  • Config (apps/backend/otel-collector-config.yaml) mounted via GCP Secret Manager.
  • Min instances = 1 (avoid cold-start dropping spans), max = 5.
  • Backend's OTEL_EXPORTER_OTLP_ENDPOINT switched from local Jaeger to the Collector's internal URL (Cloud Run private ingress + service-to-service IAM).
  • Egress: Azure App Insights connection string in GCP Secret Manager; Collector's azuremonitor exporter pulls it.

Exit criteria: staging traffic emits traces from Cloud Run backend → Collector (Cloud Run) → App Insights (Azure). End-to-end latency budget for export <5s P95.

Stage D — LiteLLM (GCP) + Langfuse (Azure) (4 days)

  • LiteLLM on GCP Cloud Run (kwilo-litellm, asia-south1) with config covering Gemini, Claude, GPT-4o. Co-located with backend for low latency.
  • Langfuse infrastructure on Azure provisioned:
  • Container Apps for langfuse-web + langfuse-worker (2 replicas each)
  • Azure Database for PostgreSQL Flex (B2s) for Langfuse state
  • ClickHouse Cloud (dev tier) for traces
  • Azure Cache for Redis (Basic C1) for queue
  • Azure Blob Storage container langfuse-events
  • DNS: langfuse.kwilo.ai behind Cloudflare Access.
  • Migrate first LLM endpoint behind LiteLLM (start with low-risk path, e.g. content generation).
  • Validate token tracking + cost in Langfuse UI for one endpoint.
  • Ramp remaining LLM call sites over 1 week — small PRs per call site, each independently revertable.

Exit criteria: 100% of LLM calls flow through LiteLLM, 100% appear in Langfuse with cost + tokens.

Stage E — Dashboards, SLOs, alerts (2 days)

  • App Insights workbooks:
  • Golden signals dashboard (RPS, error rate, P95 latency per endpoint)
  • DB health (slow queries, pool waits, asyncpg errors)
  • Failed requests with linked traces
  • Langfuse saved views:
  • Cost by model / by virtual key (org) / by endpoint
  • P95 LLM latency per provider
  • Failed generations + retry chains
  • Alert rules → Action Group → Slack #alerts-platform:
  • Error rate >2% for 5min on /api/v1/*
  • P95 latency >2s sustained 10min
  • DB pool waits P95 >100ms
  • Daily ingestion >80% of cap
  • SLO targets declared:
Surface SLO Burn-rate alert
/api/v1/auth/login 99.9% success, P95 <500ms 14.4× over 1h
Assessment submit 99.95% success, P95 <2s 14.4× over 1h
LLM tutor reply 99% success, P95 <8s 6× over 6h

Exit criteria: oncall has dashboards bookmarked, alerts route to Slack, runbook links from each alert.

6. Cost estimate (steady state)

Item Cloud Monthly USD
Application Insights ingestion (~50 GB/mo) Azure 115
Log Analytics archive (90d) Azure 20
Langfuse: Container Apps (Web×2 + Worker×2) Azure 80
Langfuse: Azure Postgres Flex B2s Azure 35
Langfuse: Azure Cache Redis Basic C1 Azure 20
ClickHouse Cloud (dev tier) Azure 100
Azure Blob Storage (lifecycle) Azure 10
LiteLLM: Cloud Run (min 1 instance) GCP 25
OTel Collector: Cloud Run (min 1 instance) GCP 20
GCP egress (telemetry to Azure, ~50 GB/mo) GCP 6
Total ~430

For comparison: Datadog APM + LLM Observability at this scale = $1500–3000/mo. Pure OSS-in-GCP variant (SigNoz + Langfuse on GCE) ≈ $200/mo — kept as escape hatch if cross-cloud ops becomes painful.

7. Risks and mitigations

Risk Mitigation
LiteLLM proxy outage takes down all LLM features 2 replicas + health check; app-side fallback to direct SDK call for first 30 days
Langfuse self-host operational burden Use ClickHouse Cloud (managed); upgrade quarterly only
App Insights cost surprise at high traffic Daily cap + 80% alert; tail-based sampling at Collector
PII leaks (prompts contain student data) Collector redacts gen_ai.prompt before App Insights pipeline; Langfuse gated by Cloudflare Access
Cross-cloud egress and Azure-side outage Collector buffers spans on disk; async batch export with retries. Azure outage delays telemetry, doesn't drop it. App-path latency unaffected (Collector sits in GCP).
Two-cloud operational complexity Documented runbooks for each cloud; secrets in respective managers (GCP Secret Manager + Azure Key Vault); single Terraform repo with two providers
Sampling drops important slow traces Tail-based sampling (sample by latency/error) at Collector
DPDP: prompts are student PII, must not leave India Pin all Azure components to centralindia; verify ClickHouse Cloud region in India; verify provider APIs from GCP asia-south1 honour data residency

8. Open questions

Closed: - ~~Is the FastAPI backend migrating to Azure?~~ → No, stays on GCP Cloud Run. (2026-05-01)

Must close before Stage C: 1. DPDP classification of prompts: are they student PII? If yes, all Langfuse infra must be in India region. 2. Who owns Langfuse ops post-launch — backend team or future SRE? 3. Existing fallback logic in the backend for Gemini → Claude → GPT — does LiteLLM's fallback config preserve the exact semantics? Needs a behaviour-equivalence test. 4. Per-tenant cost tracking: should each Org get its own LiteLLM virtual key? Implications for key management at signup. 5. Cross-cloud auth: how does GCP Collector authenticate to Azure App Insights and Azure Langfuse? Connection string in GCP Secret Manager works but rotation strategy needs design.

9. Rollout sequencing

Week Stage Risk
1 A — local OTel proof Low
2 B — App Insights staging Low
3–4 C — Container Apps backend cutover Medium (gated on Azure migration)
5–6 D — LiteLLM ramp + Langfuse Medium
7 E — SLOs, dashboards, alerts Low
8 Tune sampling, redaction, on-call runbooks Low

Each stage is independently revertable: disable OTel SDK with one env var, point Collector at nop exporter, route around LiteLLM at the app layer.

10. Changelog

  • 2026-05-01: Stage B implemented. Backend gains OTEL_BACKEND switch (otlp | azure_monitor); Azure Monitor distro path lazy-loaded so local dev cost is zero. Single shared Azure resource for staging + prod (decision flipped from per-env resources). Terraform module landed in MySetu-AI/kwiloai_infra PR #1; CI/CD wiring in MySetu-AI/kwiloai_webapp PR #723 (draft until secrets bootstrap).
  • 2026-05-01: Stage A.2 merged (#720). LLM gateway emits gen_ai.* semconv spans; streaming methods carry token counts.
  • 2026-05-01: Stage A merged (#719). OTel SDK + auto-instrumentation in apps/backend, Jaeger compose for local dev.
  • 2026-05-01: Topology decided — backend stays on GCP Cloud Run; observability infra on Azure. Collector + LiteLLM colocated in GCP asia-south1; App Insights + Langfuse in Azure centralindia. Cost estimate revised (~$430/mo). Stage C rewritten as Cloud Run Collector deployment, not Container Apps backend cutover.
  • 2026-04-30: Initial draft.