Skip to content

B2C Study Plan — Textbook Grounding (deterministic, no vector search)

Status: shipping (2026-07-05) · webapp #1253 · chatbot kwilo-chatbot#67 Owner: Kantharaju Supersedes: the search_ncert (Qdrant top-k) grounding inside study-plan-chapter, and the study-plan-syllabus Qdrant-scroll runner (kept as fallback during rollout).

Why this exists

Who asked for it

Self-initiated (Kantharaju, 2026-07-05) — flagged honestly: no user quote behind it. The failure mode is structural, not speculative: chapter content was grounded by an optional vector-search tool returning top-k fragments, so generated chapters drift from the actual textbook (invented section names, examples that don't match the book a learner holds).

What user pain does it solve

A Class-11 learner opens their generated Biology plan next to their NCERT textbook and the topics don't line up with the book's own sections — the plan feels "AI-generic", not "my syllabus". Grounding on the real chapter makes the plan use the textbook's terminology, ordering, and examples.

Cost of not doing it

Study Plan is a Pro artifact; plans that visibly diverge from the learner's textbook undercut the willingness to pay for exactly the segment (exam-driven Indian learners) the pilot targets.

Validation

Guess → structural. Vector top-k retrieval mathematically cannot return "the whole chapter, in order"; a keyed fetch can. Counterfactual: if grounded chapters ship and users don't notice, the packing budget was wasted effort — but the syllabus-derivation SQL swap pays for itself in latency/cost regardless.

Mental model

At plan-creation and chapter-writing time the exact chapter is already known — there is no fuzzy query. Retrieval is therefore a primary-key lookup, not a similarity search:

  • textbook_chunk_extraction (DB knowledge_base_status, shared prod Cloud SQL instance; written by the harvesting pipeline) is the catalog: one row per chapter, (board, class_level, subject) indexed, 824 completed chapters (NCERT 1–12).
  • Each row points at a ChunkSet JSON in Azure Blob (ncertbooks-chunks): ordered chunks with topic_label (the textbook's own sections), full text, page ranges, and figure captions.

Two-level grounding (why "many topics" never means "too much data")

  • Layout drafting (breadth): sees the structure of everything selected — titles, section lists, page/figure counts (~1KB per topic). Even a 60-topic full-syllabus selection is a ~40KB pack. Full text never reaches layout.
  • Chapter writing (depth): one unit per LLM call; that call alone downloads its ChunkSet and gets a packed, capped context. N units = N isolated queued calls.

Architecture decision: webapp owns the data plane (Option A)

The webapp backend queries the catalog, downloads the ChunkSet, packs the context (60k-char cap, user-selected topics packed first, figure manifest with captions only), and passes it as an optional textbook_context param to the chatbot's study-plan-chapter runner. The chatbot stays a pure LLM runtime.

Why not chatbot-owned fetch (rejected): the webapp needs the catalog anyway (UI dropdowns) and blob access anyway (future figure rehost to GCS) — chatbot-side fetch would add a Postgres connection + Azure credentials to the chatbot and still leave the webapp reading the same table. Revisit only when a second agent (tutor / lesson-plan) wants deterministic chapter fetch; the packing function is pure and lifts cleanly.

Config: TEXTBOOK_KB_DATABASE_URL (points at knowledge_base_status in BOTH envs — cross-env reference data; staging has no copy) and AZURE_KB_BLOB_CONNECTION_STRING (account kwilojeeharvesta7fde7). Both empty ⇒ feature off, legacy paths run.

What changed

Surface Before After
Syllabus chapter list chatbot runner scrolls ≤4,000 Qdrant points per subject one indexed SQL query (runner = fallback)
UI subject/chapter dropdowns static/derived GET /textbooks/subjects + /textbooks/chapters — only completed corpus rows are offered; UI copy says "topics", never "chapters"
Chapter content grounding optional search_ncert top-k fragments full packed chapter as textbook_context, authoritative-source prompt; search_ncert demoted to prerequisites-only
Grounding failure n/a best-effort: log + hydrate ungrounded — never fails a chapter

Deploy order: chatbot first (extra="forbid" params). Webapp omits the param when grounding is empty, so ungrounded flows never depend on chatbot rollout.

Non-goals

  • Rendering textbook figures in chapter content (manifest is packed; rehost Azure→GCS is a follow-up — raw Azure URLs must never reach users).
  • Topic-level (sub-chapter) UI filtering — needs a topics_summary column in the harvesting pipeline first.
  • Grounding AI Tutor / lesson plans on this path (possible later; triggers the Option-B revisit above).
  • Any change to plan scheduling, quotas, or the weekly planner.

Rejected alternatives

  • Keep vector search but raise top-k — still fragment-based, still cross-chapter bleed, still embedding cost per call.
  • Chatbot-owned corpus fetch — see architecture decision.
  • Mirror ChunkSets to GCS to avoid Azure creds in webapp — considered; deferred. One connection-string secret is cheaper than a sync pipeline; revisit if Azure egress/latency bites.
  • Send textbook text via additional_context — it lands in the reference-document search tool, not the prompt; grounding must be unconditional, not tool-invoked.

Follow-ups (tracked)

  1. Frontend: create-page dropdowns consume /textbooks/* (with/after webapp #1250; copy says "topics").
  2. TOC-pack for goal-based skeleton layout (blocked on #1250 selection contract).
  3. Figure rehost Azure→GCS + [fig:N] rendering.
  4. Cloud Tasks migration for durable hydration (replaces in-process BackgroundTasks; also gives concurrency control for large plans).
  5. Harvesting pipeline: topics_summary JSONB on textbook_chunk_extraction.
  • Study Plan spec (features/b2c-study-plan-spec.md) — weekly planner; shares the "no weightage outside exams" constraint
  • Study Plan gate (features/b2c-study-plan-gate.md)
  • Harvesting pipeline: MySetu-AI/kwiloai_ncert_knowledge_harvesting (src/db/kb_models.py, src/pipeline/chunk_schema.py)