Qdrant Knowledge Base And Memory Architecture

Goal

Kwilo should use Qdrant as the single semantic retrieval layer for:

• Platform exam knowledge: KCET, NEET, JEE Main, JEE Advanced previous-year papers.
• Platform academic knowledge: exam syllabi, official syllabus documents, textbooks, reference material.
• Institution knowledge bases: school, college, training-provider, and enterprise documents.
• Trainer/faculty knowledge bases: private or shared uploaded notes, papers, lesson material.
• Learner and trainer memory: long-term user facts, preferences, mastery evidence, misconceptions, teaching patterns, and retrieval context.

Postgres remains the transactional source of truth for users, permissions, document metadata, ingestion state, structured exam/question records, attempts, analytics snapshots, consent, audit logs, and memory profile tables. Qdrant owns semantic vectors, retrievable text payloads, and metadata filters for semantic search. We should not keep pgvector as a parallel semantic store after migration.

Current State

VidyaNet Backend

The current knowledge base flow is split across:

• apps/backend/src/services/knowledge_base.py
• Uploads documents.
• Tracks TeacherKnowledgeDocument.
• Calls doc-intelligence ingestion.
• Deletes/retries by deleting rows from curriculum_embeddings.
• apps/backend/src/services/rag_service.py
• Generates embeddings with Vertex AI.
• Searches Postgres curriculum_embeddings using pgvector.
• Supports filters such as org_unit_id, subject, class_level, semester, program, branch, document_type.
• apps/backend/src/models/user_memory.py
• Defines UserLearningProfile, ConversationInsight, and MemoryEmbedding.
• MemoryEmbedding stores JSON embeddings in Postgres and is not wired into hot-path retrieval yet.
• apps/backend/src/services/memory_service.py
• Builds profile/context prompt memory.
• Semantic memory retrieval is currently TODO.

kwiloai_memory

The local ../kwiloai_memory package is already designed for the target direction:

• Library-first package consumed by apps/backend, not a separate HTTP service.
• Caller supplies AsyncSession, AsyncQdrantClient, collection name, and embedding adapter.
• retrieve() composes mastery, prerequisite gaps, profile, and Qdrant fact search.
• Qdrant fact search is tenant/user/role filtered by:
• tenant_id
• user_id
• role
• tier2_mode
• status=active
• current=true
• Memory roles are richer than current backend user roles: student, teacher, parent, external_educator, school_admin, org_admin, principal, org_unit_admin.

This means the backend should integrate kwiloai_memory rather than creating a second custom memory implementation.

Architecture Decision

Use Qdrant as the only semantic/vector database.

Use multiple Qdrant collections, not one overloaded collection:

Collection	Purpose
kwilo_knowledge_chunks_v1	Platform, institution, trainer/faculty, syllabus, textbook, and document chunks.
kwilo_exam_questions_v1	Previous-year questions, answer keys, solutions, and topic metadata for KCET, NEET, JEE Main, JEE Advanced.
kwilo_user_memory_v1	User-specific memory facts and semantic summaries.
kwilo_memory_staging / kwilo_memory_prod	Existing kwiloai_memory package collections. These should either be migrated into kwilo_user_memory_v1 or kept as aliases until all consumers move.

Separate collections keep payload schemas stable, reduce accidental data exposure, and let us tune indexes differently for documents, exam questions, and user memory.

Collection Schemas

kwilo_knowledge_chunks_v1

One Qdrant point per retrievable chunk.

Point ID:

kb:{document_id}:{chunk_index}

Payload:

{
  "schema_version": "v1",
  "corpus": "knowledge",
  "tenant_type": "platform|institution|trainer|learner",
  "org_unit_id": "uuid|null",
  "owner_user_id": "uuid|null",
  "visibility": "platform|institution|private|class|course",
  "source_type": "textbook|syllabus|notes|question_paper|reference|web",
  "document_id": "string",
  "chunk_index": 12,
  "content": "retrievable chunk text",
  "title": "document title",
  "filename": "source.pdf",
  "subject": "Physics",
  "board": "CBSE|NCERT|Karnataka|KEA|NTA|JEE",
  "class_level": 12,
  "academic_level_id": "uuid|null",
  "semester": null,
  "program": null,
  "branch": null,
  "chapter": "Electrostatics",
  "topic": "Capacitance",
  "subtopic": "Parallel Plate Capacitor",
  "page_number": 42,
  "source_url": "https://...",
  "rights_status": "licensed|official_public|permission_obtained|restricted_reference|user_uploaded",
  "created_at": "iso8601",
  "updated_at": "iso8601"
}

Required payload indexes:

• corpus
• tenant_type
• org_unit_id
• owner_user_id
• visibility
• source_type
• document_id
• subject
• board
• class_level
• academic_level_id
• semester
• program
• branch
• chapter
• topic
• rights_status

kwilo_exam_questions_v1

One Qdrant point per question. Long passages or solutions can be additional points linked by question_id.

Point ID:

exam:{exam}:{year}:{paper_id}:{question_number}

Payload:

{
  "schema_version": "v1",
  "corpus": "exam_question",
  "exam": "KCET|NEET|JEE_MAIN|JEE_ADVANCED",
  "year": 2024,
  "session": "April|May|Paper 1|Paper 2|null",
  "shift": "1|2|null",
  "paper_id": "string",
  "question_id": "string",
  "question_number": 42,
  "language": "en",
  "subject": "Physics",
  "chapter": "Electrostatics",
  "topic": "Capacitance",
  "subtopic": "Energy Stored In Capacitor",
  "question_type": "single_correct|multiple_correct|integer|numerical|assertion_reason|matrix_match",
  "difficulty": 0.62,
  "marks": 4,
  "negative_marks": 1,
  "has_diagram": true,
  "answer_available": true,
  "solution_available": true,
  "content": "question + options + normalized formula text",
  "source_url": "https://...",
  "rights_status": "licensed|official_public|permission_obtained|restricted_reference",
  "review_status": "raw|machine_tagged|human_reviewed",
  "topic_confidence": 0.91,
  "created_at": "iso8601",
  "updated_at": "iso8601"
}

Required payload indexes:

• exam
• year
• session
• shift
• paper_id
• question_id
• subject
• chapter
• topic
• subtopic
• question_type
• difficulty
• rights_status
• review_status

kwilo_user_memory_v1

The payload should align with kwiloai_memory.services.facts.search_facts.

Point ID:

memory:{fact_id}

Payload:

{
  "schema_version": "v1",
  "corpus": "user_memory",
  "tenant_id": "uuid",
  "org_unit_id": "uuid|null",
  "user_id": "uuid",
  "role": "student|teacher|parent|external_educator|school_admin|org_admin|principal|org_unit_admin",
  "tier2_mode": "learner|educator",
  "fact_id": "uuid",
  "fact_type": "misconception|preference|goal|achievement|common_question|teaching_pattern|semantic_summary|other",
  "content": "memory content",
  "concept_id": "uuid|null",
  "subject": "Physics",
  "topic": "Newton's Laws",
  "confidence": 0.86,
  "status": "active|quarantined|deleted",
  "current": true,
  "source_conversation_id": "uuid|null",
  "source_message_ids": ["uuid"],
  "created_at": "iso8601",
  "updated_at": "iso8601"
}

Required payload indexes:

• tenant_id
• org_unit_id
• user_id
• role
• tier2_mode
• fact_type
• subject
• topic
• status
• current

Retrieval Gateway

Add a backend service layer, tentatively SemanticRetrievalService, that hides collection details from product features.

The service should expose:

• search_knowledge(query, filters, user_context)
• search_exam_questions(query, filters, user_context)
• retrieve_user_memory(query, user_context)
• retrieve_learning_context(query, user_context, intent)
• generate_question_bank_blueprint(filters, constraints)
• generate_mock_test_blueprint(exam, subject, target_profile)

For AI Tutor, AI Teacher, mock-test generation, and question-bank generation, retrieval should run in parallel:

1. Retrieve user memory with kwiloai_memory.retrieve() or a compatible wrapper.
2. Retrieve institution/trainer/platform knowledge chunks.
3. Retrieve previous-year exam questions when the intent is exam-prep or assessment generation.
4. Merge, dedupe, and rerank.
5. Build a prompt context with source citations and rights flags.

Authorization And Visibility

Qdrant filters are mandatory. Never run unfiltered semantic search.

Rules:

• Platform content: visible to all eligible users.
• Institution content: visible only when org_unit_id matches.
• Trainer private content: visible only to owner_user_id, unless explicitly shared.
• Class/course content: visible only to users enrolled in that class/course or trainers assigned to it.
• User memory: visible only to that user and permitted supervisory roles, according to memory consent and safety policy.
• Restricted exam content: usable for internal analytics but not displayed verbatim unless rights allow it.

All frontend calls go through the backend. Qdrant must not be queried directly from browser clients.

Migration From pgvector To Qdrant

Knowledge Base Migration

Source:

• Postgres curriculum_embeddings
• Postgres teacher_knowledge_documents

Steps:

1. Add Qdrant client configuration:
2. QDRANT_URL
3. QDRANT_API_KEY
4. QDRANT_KNOWLEDGE_COLLECTION
5. QDRANT_EXAM_COLLECTION
6. QDRANT_USER_MEMORY_COLLECTION
7. VECTOR_BACKEND=qdrant|pgvector|dual
8. Create Qdrant collection bootstrap script.
9. Add dual-write in doc-intelligence ingestion:
10. Keep current Postgres writes temporarily.
11. Upsert each generated chunk to kwilo_knowledge_chunks_v1.
12. Build backfill script:
13. Read curriculum_embeddings in batches.
14. Join with teacher_knowledge_documents by document_id.
15. Upsert Qdrant points with deterministic IDs.
16. Store migrated counts by document_id.
17. Add parity validation:
18. Compare chunk counts per document.
19. Run top-k retrieval comparison on a seed query set.
20. Validate delete/retry removes stale Qdrant points.
21. Switch reads:
22. Feature flag backend retrieval to Qdrant.
23. Keep pgvector fallback for one release.
24. Stop pgvector writes.
25. Archive/drop curriculum_embeddings only after production signoff and backups.

User Memory Migration

Source:

• Current backend memory_embeddings
• Current backend conversation_insights
• Current backend user_learning_profiles
• Future/parallel kwiloai_memory tables

Steps:

1. Install and configure kwiloai-memory package in apps/backend.
2. Run kwiloai_memory Alembic migrations against the backend memory database or a dedicated memory database.
3. Map backend user roles to MemoryRole.
4. Add Qdrant client factory using kwiloai_memory.adapters.qdrant._parse_qdrant_url.
5. Backfill memory_embeddings to kwilo_user_memory_v1:
6. Point ID: memory:{memory_embedding_id}
7. Payload fields: tenant_id, user_id, role, tier2_mode, fact_type, content, subject, topic, status, current.
8. Convert or link ConversationInsight rows into Qdrant facts where appropriate.
9. Keep UserLearningProfile in Postgres because it is structured profile state, not a semantic search index.
10. Wire MemoryService.get_enriched_context() to call kwiloai_memory.retrieve() for relevant facts.
11. Validate:
12. No cross-tenant memory leakage.
13. No cross-user memory leakage.
14. Memory retrieval latency remains acceptable on chat hot path.
15. Retire legacy MemoryEmbedding writes after Qdrant path is stable.

Previous-Year Exam Data Pipeline

Source Acquisition

Priority order:

1. Official sources:
2. NTA for NEET and JEE Main.
3. JEE Advanced official archives.
4. KEA for KCET.
5. Licensed publisher/coaching archives for missing historical years.
6. Third-party mirrors only as discovery signals, not as final trusted sources.

Each paper must have:

• Source URL.
• Local object-storage URI.
• SHA256 checksum.
• Rights status.
• Paper metadata: exam, year, session, shift, subject, language.
• Parse confidence.
• Review status.

Ingestion

1. Download paper and answer key.
2. Store original PDF/images in object storage.
3. Extract page layout and OCR through doc-intelligence.
4. Segment questions, options, passages, diagrams, formulas.
5. Align answer key and marks.
6. Tag syllabus nodes:
7. subject
8. chapter
9. topic
10. subtopic
11. difficulty
12. Human-review low-confidence questions and high-value papers.
13. Store structured records in Postgres.
14. Upsert semantic points to kwilo_exam_questions_v1.
15. Generate analytics snapshots.

Analytics

Use Postgres/materialized analytics tables for aggregate calculations.

Qdrant is used to retrieve and cluster semantically related questions; Postgres is used to compute and serve:

• Topic weightage by exam/year/subject.
• Chapter frequency by year range.
• Marks distribution.
• Difficulty trend.
• Repeated concept clusters.
• Student weak-topic history.
• Trainer-generated paper coverage.

Analytics jobs should periodically read from structured question tables and, where useful, enrich from Qdrant similarity clusters.

Product Features Enabled

Trainers / Faculty

• Search institutional/trainer KB semantically.
• Search previous-year questions by exam, subject, chapter, topic, difficulty, year range.
• Generate question banks.
• Generate question papers from a blueprint.
• Generate answer keys and solutions.
• Compare generated paper coverage against past exam weightage.
• Get “what to teach next” recommendations from learner gaps.

Learners

• Personalized doubt answering grounded in:
• user memory,
• institution/trainer content,
• platform syllabus/textbooks,
• previous-year questions.
• Topic-wise practice.
• Mock tests based on historical weightage.
• Revision plans based on weak topics, due reviews, and target exam date.
• Similar-question recommendations.

Admins

• Ingestion dashboard.
• Source/rights dashboard.
• Review queue for OCR/topic tagging.
• Tenant-level KB usage analytics.
• Erasure and consent audit for user memory.

Dedicated PR Plan

Each PR must be based on staging.

PR 1: Documentation And Architecture

Scope:

• Add this architecture plan.
• Record collection schemas, migration plan, and phased implementation.

Validation:

• Docs-only review.

PR 2: Qdrant Foundation In Backend

Scope:

• Add qdrant-client dependency if not already present.
• Add backend config for Qdrant URL/API key/collections.
• Add QdrantClientProvider.
• Add collection bootstrap/check command.
• Add unit tests with mocked Qdrant client.

No product behavior change.

PR 3: Knowledge Base Dual-Write

Scope:

• Extend doc-intelligence/backend ingestion contract to upsert chunks into Qdrant.
• Keep pgvector writes for rollback.
• Update delete/retry to delete Qdrant points by document_id.
• Add migration/backfill script for curriculum_embeddings.

Validation:

• Backfill dry run.
• Count parity by document.
• Search smoke tests.

PR 4: Qdrant Read Path For Knowledge Retrieval

Scope:

• Add Qdrant-backed RAGService implementation behind VECTOR_BACKEND.
• Update AI Tutor/AI Teacher retrieval to use Qdrant path.
• Keep pgvector fallback.

Validation:

• Query parity tests against a curated seed set.
• Tenant visibility tests.

PR 5: kwiloai_memory Integration

Scope:

• Add package dependency from GAR.
• Add memory config and Qdrant client wiring.
• Run or document kwiloai_memory migrations.
• Map backend roles to MemoryRole.
• Call kwiloai_memory.retrieve() from AI Tutor memory enrichment.

Validation:

• Unit tests for role mapping and context construction.
• Mocked retrieval tests.

PR 6: User Memory Backfill And Cutover

Scope:

• Backfill legacy memory_embeddings into Qdrant.
• Convert eligible ConversationInsight rows into Qdrant facts.
• Stop writing semantic memory to Postgres.
• Keep structured UserLearningProfile in Postgres.

Validation:

• Cross-tenant isolation tests.
• Cross-user isolation tests.
• Backfill count checks.

PR 7: Exam KB Core Schema And Ingestion

Scope:

• Add structured exam paper/question tables.
• Add source manifest format.
• Add ingestion job model.
• Add Qdrant upsert for question points.
• Ingest initial sample set.

Validation:

• Question count and answer-key coverage checks.
• Topic tagging confidence report.

PR 8: Exam KB Retrieval And Analytics

Scope:

• Search API for exam questions.
• Topic-weightage analytics endpoints.
• Faculty question-bank backend service.
• Mock-test blueprint generation.

Validation:

• Weightage correctness tests against known sample data.
• Retrieval filter tests.

PR 9: UI Surfaces

Scope:

• Faculty question-bank and paper-generation UI.
• Learner practice/mock-test entry points.
• Knowledge source citations.
• Admin ingestion/review dashboard.

Validation:

• Playwright flows for trainer and learner.

PR 10: Decommission pgvector Semantic Store

Scope:

• Stop pgvector writes.
• Remove pgvector retrieval code.
• Archive/drop curriculum_embeddings and memory_embeddings only after backup and production signoff.

Validation:

• Production parity report.
• Backup verification.

Operational Requirements

• Qdrant must not be directly queried by browser clients.
• Qdrant API key must remain mandatory.
• Prefer private networking or ingress restrictions for Qdrant.
• Schedule Qdrant snapshots and restore drills.
• Emit metrics:
• search latency,
• no-result rate,
• Qdrant errors,
• ingestion throughput,
• backfill progress,
• ACL filter rejection counts.
• Add audit events for:
• KB ingestion,
• memory fact creation,
• memory fact retrieval,
• deletion/erasure,
• exam source import.

Open Questions

1. Confirm whether “BSEAG” means B.Sc Agriculture entrance preparation under KCET/UGCET or a separate exam.
2. Decide whether Qdrant should stay publicly reachable at qdrant.kwilo.ai or be moved behind private ingress/VPN.
3. Decide if platform exam KB exact question display requires explicit licenses for each source.
4. Decide whether user memory Qdrant collection should reuse existing kwilo_memory_prod/staging collection names or migrate to kwilo_user_memory_v1.
5. Decide the canonical embedding model and dimension. kwiloai_memory currently assumes 768-dimensional vectors.