LLM-Based Assignment Evaluation System¶
Architecture Design Document¶
Version: 1.0 Date: January 2025 Status: Proposal Author: VidyaNet Engineering
Table of Contents¶
- Executive Summary
- Current State Analysis
- Proposed Solution Overview
- System Architecture
- OCR Pipeline for Handwritten Assignments
- LLM Evaluation Engine
- File Processing Pipeline
- Rubric System
- Integration with Existing System
- Data Flow Diagrams
- API Design
- Database Schema
- Security & Privacy
- Performance Considerations
- Implementation Phases
- Cost Analysis
- Risks & Mitigations
1. Executive Summary¶
1.1 Problem Statement¶
VidyaNet currently supports 30+ assignment types including essays, projects, case studies, research papers, and lab practicals. However, the auto-grading system only works for objective question types (MCQ, True/False, Fill-in-the-blank). All subjective assignments require manual teacher grading, which:
- Creates significant workload for teachers
- Delays feedback to students
- Results in inconsistent grading across different teachers
- Doesn't scale for large class sizes
Additionally, many Indian schools still use paper-based assignments where students submit handwritten answers. Currently, there's no way to evaluate these digitally.
1.2 Proposed Solution¶
Implement an LLM-based evaluation system that can:
- Evaluate all assignment types using AI (essays, short answers, long answers, projects)
- Process handwritten submissions through OCR (Optical Character Recognition)
- Parse uploaded documents (PDF, DOCX, images)
- Provide structured feedback with strengths, improvements, and suggested marks
- Support customizable rubrics for consistent evaluation criteria
- Assist teachers rather than replace them (human-in-the-loop)
1.3 Key Benefits¶
| Stakeholder | Benefit |
|---|---|
| Teachers | 70-80% reduction in grading time, consistent evaluation criteria |
| Students | Faster feedback (minutes vs days), detailed improvement suggestions |
| Schools | Scalable assessment, quality assurance across teachers |
| Platform | Premium feature for subscription tiers, competitive differentiation |
2. Current State Analysis¶
2.1 Existing Assignment Types¶
VidyaNet supports assignments across three organization categories:
K-12 Schools: - Homework, Quiz, Worksheet, Classwork, Project, Essay, Test
Higher Education: - Assignment, Quiz, Case Study, Lab Practical, Viva Voce, Seminar, Research Paper, Group Project, Peer Review, Coding Competition
Corporate: - Knowledge Check, Assessment, Role Play, Compliance Training, Practical Exercise
2.2 Existing Question Types¶
| Question Type | Auto-Gradable | Current Approach |
|---|---|---|
| MCQ (Single) | Yes | Exact match comparison |
| MCQ (Multiple) | Yes | Set comparison |
| True/False | Yes | Boolean comparison |
| Fill in Blank | Yes | Case-insensitive string match |
| Match | Yes | Pair comparison |
| Short Answer | No | Manual grading required |
| Long Answer | No | Manual grading required |
| File Upload | No | Manual review required |
2.3 Current Submission Flow¶
Student System Teacher
| | |
|-- Start Assignment ----->| |
| | |
|-- Answer Questions ----->| |
| (text/file upload) | |
| | |
|-- Submit -------------->| |
| |-- Store Submission ------>|
| | |
| | [If objective only] |
| |-- Auto-grade MCQ/TF ----->|
| | |
| | [If subjective] |
| | |
| |<-- Manual Review ---------|
| |<-- Enter Marks ----------|
| |<-- Write Feedback --------|
| | |
|<-- Receive Grade --------| |
|<-- View Feedback --------| |
2.4 Current Limitations¶
- No AI evaluation for subjective questions
- No OCR capability for handwritten submissions
- No rubric system for standardized evaluation
- No plagiarism detection for text submissions
- Limited file parsing (storage only, no content extraction)
- No feedback generation assistance for teachers
3. Proposed Solution Overview¶
3.1 High-Level Architecture¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ LLM EVALUATION SYSTEM │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ │
│ │ SUBMISSION │ │
│ │ INPUT │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ INPUT PROCESSOR │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ Text │ │ Image │ │ PDF │ │ DOCX │ │ │
│ │ │ Answer │ │ File │ │ File │ │ File │ │ │
│ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
│ │ │ │ │ │ │ │
│ │ │ ┌────▼─────┐ ┌────▼─────┐ │ │ │
│ │ │ │ OCR │ │ PDF │ │ │ │
│ │ │ │ Engine │ │ Parser │ │ │ │
│ │ │ └────┬─────┘ └────┬─────┘ │ │ │
│ │ │ │ │ │ │ │
│ │ └──────────────┴──────────────┴──────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ EXTRACTED TEXT │ │ │
│ │ └────────┬────────┘ │ │
│ └─────────────────────────────┼───────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ EVALUATION ENGINE │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ RUBRIC │ │ LLM │ │ PLAGIARISM │ │ │
│ │ │ MATCHER │───▶│ GATEWAY │───▶│ CHECKER │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ EVALUATION │ │ │
│ │ │ RESULT │ │ │
│ │ └────────┬────────┘ │ │
│ └─────────────────────────────┼───────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ TEACHER REVIEW │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ VIEW AI │ │ ADJUST │ │ ACCEPT/ │ │ │
│ │ │ EVALUATION │───▶│ MARKS │───▶│ OVERRIDE │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
3.2 Core Components¶
| Component | Purpose | Technology |
|---|---|---|
| Input Processor | Normalize all input types to text | Custom service |
| OCR Engine | Extract text from images/scanned docs | Google Vision + Gemini |
| PDF Parser | Extract text from PDF documents | PyMuPDF + OCR fallback |
| DOCX Parser | Extract text from Word documents | python-docx |
| Rubric Matcher | Apply evaluation criteria | Database lookup |
| LLM Gateway | AI-powered evaluation | Gemini 2.0 Flash (primary) |
| Plagiarism Checker | Detect copied content | Future: Turnitin/Copyleaks |
| Teacher Review UI | Human-in-the-loop interface | React components |
4. System Architecture¶
4.1 Component Diagram¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ FRONTEND (React) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Submission │ │ Grading │ │ Rubric │ │ OCR │ │
│ │ View │ │ Panel │ │ Editor │ │ Preview │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
└─────────┼────────────────┼────────────────┼────────────────┼────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API LAYER (FastAPI) │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ /api/v1/evaluations │ │
│ │ POST /submissions/{id}/evaluate - Trigger AI evaluation │ │
│ │ GET /submissions/{id}/evaluation - Get evaluation results │ │
│ │ POST /submissions/{id}/accept - Accept AI grade │ │
│ │ POST /ocr/extract - Extract text from file │ │
│ │ CRUD /rubrics - Manage evaluation rubrics │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ SERVICE LAYER │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ AIEvaluation │ │ OCRService │ │ FileParser │ │
│ │ Service │ │ │ │ Service │ │
│ └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │ │
│ │ ┌───────────────┴───────────────┐ │ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ LLMGateway │ │ StorageService │ │
│ │ (Existing) │ │ (GCS) │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ EXTERNAL APIS │ │ DATABASE │
│ │ │ │
│ - Gemini 2.0 Flash │ │ - ai_evaluations │
│ - Claude Sonnet 4 │ │ - evaluation_ │
│ - Google Vision │ │ rubrics │
│ - (Future) │ │ - ocr_processing_ │
│ Plagiarism APIs │ │ logs │
└─────────────────────┘ └─────────────────────┘
4.2 Integration Points with Existing System¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ EXISTING VIDYANET SYSTEM │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ assignments │ │ submissions │ │ grades │ │
│ │ (table) │◄────▶│ (table) │◄────▶│ (table) │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ │ │ │ │
│ │ ┌───────────────┼───────────────┐ │ │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ NEW: EVALUATION SYSTEM │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ evaluation │ │ ai_ │ │ ocr_ │ │ │
│ │ │ _rubrics │◄──▶│ evaluations │◄──▶│ processing │ │ │
│ │ │ (table) │ │ (table) │ │ _logs │ │ │
│ │ └─────────────┘ └──────┬──────┘ └─────────────┘ │ │
│ │ │ │ │
│ │ │ (links to) │ │
│ │ ▼ │ │
│ │ ┌─────────────┐ │ │
│ │ │ grades │ (ai_evaluation_id FK) │ │
│ │ │ (table) │ │ │
│ │ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ media_ │ │ users │ │
│ │ attachments │ │ (teachers) │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
5. OCR Pipeline for Handwritten Assignments¶
5.1 Overview¶
The OCR pipeline handles the conversion of handwritten assignments (submitted as photos or scanned documents) into machine-readable text for LLM evaluation.
5.2 Supported Input Formats¶
| Format | Common Use Case | Processing Approach |
|---|---|---|
| JPEG/PNG | Phone camera photos of notebook pages | Direct OCR |
| PDF (Scanned) | Scanned answer sheets | Page extraction → OCR |
| PDF (Digital) | Typed submissions | Direct text extraction |
| HEIC | iPhone photos | Convert to JPEG → OCR |
5.3 OCR Model Selection¶
We will use a multi-model approach based on content complexity:
Primary Model: Google Cloud Vision API¶
Best for: - Printed text - Clear handwriting - Structured forms - Multi-language support (Hindi, Kannada, English)
Capabilities: - Document text detection (dense text) - Handwriting recognition - Language detection - Confidence scores per word - Bounding box information
Limitations: - Struggles with very messy handwriting - Limited understanding of context - Cannot interpret diagrams or equations well
Secondary Model: Gemini 2.0 Flash (Vision)¶
Best for: - Messy/cursive handwriting - Context-aware extraction - Mathematical equations and formulas - Diagrams with labels - Mixed content (text + drawings)
Capabilities: - Multimodal understanding - Can follow instructions for extraction - Understands context and structure - Can transcribe LaTeX for math - Better at inferring unclear characters
Limitations: - Higher latency than Vision API - More expensive per request - May hallucinate content
Fallback: Tesseract OCR (Self-hosted)¶
Best for: - Offline processing - Cost reduction for bulk processing - Simple printed text
When to use: - When external API quotas are exhausted - For non-critical pre-processing - Development/testing environments
5.4 OCR Decision Flow¶
┌─────────────────┐
│ Input Image │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Pre-processing │
│ - Deskew │
│ - Denoise │
│ - Contrast │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Content Type │
│ Detection │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Printed │ │Handwritten│ │ Mixed │
│ Text │ │ Text │ │ Content │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────────┐ ┌──────────┐
│ Google │ │ Quality │ │ Gemini │
│ Vision │ │ Check │ │ Vision │
│ API │ └──────┬───────┘ │ (Full) │
└────┬─────┘ │ └────┬─────┘
│ ┌──────┴──────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ │
│ │ Clear │ │ Messy │ │
│ │Handwriting│ │Handwriting│ │
│ └────┬─────┘ └────┬─────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ │
│ │ Google │ │ Gemini │ │
│ │ Vision │ │ Vision │ │
│ └────┬─────┘ └────┬─────┘ │
│ │ │ │
└────────┴──────┬──────┴─────────┘
│
▼
┌─────────────────┐
│ Confidence │
│ Threshold │
│ Check (>0.7) │
└────────┬────────┘
│
┌───────────┴───────────┐
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ Pass │ │ Fail │
│ (>0.7) │ │ (<0.7) │
└────┬─────┘ └────┬─────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ Return │ │ Flag │
│ Result │ │ for │
│ │ │ Manual │
│ │ │ Review │
└──────────┘ └──────────┘
5.5 Handwriting Quality Assessment¶
Before choosing the OCR model, we assess handwriting quality:
Quality Indicators:
| Factor | Clear | Medium | Messy |
|---|---|---|---|
| Letter spacing | Consistent | Some overlap | Heavy overlap |
| Line alignment | Straight lines | Minor drift | Wavy/crossing |
| Character formation | Standard shapes | Minor variations | Unusual forms |
| Ink consistency | Even pressure | Some variation | Faded/blotchy |
| Paper quality | Clean white | Some marks | Lined/gridded |
Assessment Method: 1. Use Google Vision API for initial pass 2. Check confidence scores per word 3. If average confidence < 0.75, switch to Gemini 4. If Gemini confidence also low, flag for manual review
5.6 Multi-Language Support¶
Indian schools use multiple languages. Our OCR pipeline supports:
| Language | Script | Google Vision | Gemini |
|---|---|---|---|
| English | Latin | Excellent | Excellent |
| Hindi | Devanagari | Good | Excellent |
| Kannada | Kannada | Good | Good |
| Tamil | Tamil | Good | Good |
| Telugu | Telugu | Good | Good |
| Marathi | Devanagari | Good | Excellent |
Language Detection: - Auto-detect from image content - Use school's default language as hint - Support mixed-language content (common in Indian education)
5.7 Special Content Handling¶
Mathematical Equations¶
Input Image: [Photo of handwritten equation: x² + 2x + 1 = 0]
│
▼
┌───────────────────────────────┐
│ Gemini Vision Prompt │
│ │
│ "Extract the mathematical │
│ equation from this image. │
│ Output in LaTeX format." │
└───────────────┬───────────────┘
│
▼
Output: $x^2 + 2x + 1 = 0$
Diagrams and Figures¶
Input Image: [Photo of labeled diagram]
│
▼
┌───────────────────────────────┐
│ Gemini Vision Prompt │
│ │
│ "Describe this diagram. │
│ Extract all labels and │
│ annotations. Describe the │
│ relationships shown." │
└───────────────┬───────────────┘
│
▼
Output: "The diagram shows a plant cell with
the following labeled parts:
- Cell wall (outer boundary)
- Cell membrane (inner boundary)
- Nucleus (center, circular)
- Chloroplasts (green oval shapes)
..."
Tables and Charts¶
Input Image: [Photo of handwritten data table]
│
▼
┌───────────────────────────────┐
│ Gemini Vision Prompt │
│ │
│ "Extract the table data from │
│ this image. Preserve the │
│ row and column structure. │
│ Output as markdown table." │
└───────────────┬───────────────┘
│
▼
Output: | Name | Marks | Grade |
|---------|-------|-------|
| Priya | 85 | A |
| Rahul | 72 | B |
5.8 OCR Processing Pipeline Steps¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ OCR PROCESSING PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ STEP 1: FILE RETRIEVAL │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Download file from Google Cloud Storage │ │
│ │ - Verify file integrity (checksum) │ │
│ │ - Check file size limits (max 20MB) │ │
│ │ - Identify MIME type │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 2: FORMAT CONVERSION │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - HEIC → JPEG (for iPhone photos) │ │
│ │ - PDF → PNG pages (for multi-page documents) │ │
│ │ - Resize if needed (optimal: 1500-3000px width) │ │
│ │ - Convert to RGB color space │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 3: IMAGE PRE-PROCESSING │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Deskew (straighten rotated images) │ │
│ │ - Remove noise (Gaussian blur + threshold) │ │
│ │ - Enhance contrast (adaptive histogram equalization) │ │
│ │ - Binarization for very faint writing │ │
│ │ - Border removal (crop to content area) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 4: CONTENT ANALYSIS │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Detect content type (printed/handwritten/mixed) │ │
│ │ - Identify special elements (equations, diagrams, tables) │ │
│ │ - Estimate language(s) present │ │
│ │ - Assess overall quality score │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 5: MODEL SELECTION │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Choose primary OCR model based on content analysis │ │
│ │ - Prepare model-specific parameters │ │
│ │ - Set timeout and retry configuration │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 6: OCR EXECUTION │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Call selected OCR model (Vision API or Gemini) │ │
│ │ - Handle rate limiting and retries │ │
│ │ - Collect confidence scores │ │
│ │ - Log processing metrics │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 7: POST-PROCESSING │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Clean extracted text (remove artifacts) │ │
│ │ - Normalize whitespace and line breaks │ │
│ │ - Correct common OCR errors (spell-check with domain awareness) │ │
│ │ - Structure content (paragraphs, lists, equations) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 8: QUALITY VALIDATION │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Check overall confidence threshold (>0.7 pass) │ │
│ │ - Identify low-confidence segments │ │
│ │ - Flag for manual review if needed │ │
│ │ - Generate quality report │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ STEP 9: STORAGE & LOGGING │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ - Store extracted text in database │ │
│ │ - Log OCR metrics (processing time, model used, confidence) │ │
│ │ - Cache result for future reference │ │
│ │ - Return result to caller │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
5.9 OCR Confidence Handling¶
| Confidence Level | Range | Action |
|---|---|---|
| High | 0.90 - 1.00 | Direct use, no review needed |
| Good | 0.75 - 0.89 | Use with minor caution flag |
| Medium | 0.60 - 0.74 | Show warning, suggest review |
| Low | 0.40 - 0.59 | Require manual verification |
| Very Low | < 0.40 | Reject, request better image |
6. LLM Evaluation Engine¶
6.1 Model Selection Strategy¶
VidyaNet uses a multi-provider LLM architecture with automatic failover:
┌─────────────────────────────────────────────────────────────────┐
│ LLM GATEWAY (Existing) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Request ──▶ ┌─────────────────┐ │
│ │ Primary Model │ │
│ │ Gemini 2.0 Flash│ ◄── Fast, cost-effective │
│ └────────┬────────┘ │
│ │ │
│ [If fails/rate limited] │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Fallback 1 │ │
│ │ Claude Sonnet 4 │ ◄── Better reasoning │
│ └────────┬────────┘ │
│ │ │
│ [If fails] │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Fallback 2 │ │
│ │ GPT-4o │ ◄── Final fallback │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Response │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
6.2 Evaluation Prompt Design¶
The evaluation prompt is structured to ensure consistent, educational, and fair assessment:
┌─────────────────────────────────────────────────────────────────┐
│ EVALUATION PROMPT STRUCTURE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 1: CONTEXT │ │
│ │ │ │
│ │ - Assignment title and type │ │
│ │ - Subject and topic │ │
│ │ - Grade level / academic level │ │
│ │ - Institution type (K-12, college, etc.) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 2: QUESTION DETAILS │ │
│ │ │ │
│ │ - Question text (full content) │ │
│ │ - Maximum marks for this question │ │
│ │ - Question type (short_answer, long_answer, etc.) │ │
│ │ - Expected length/depth │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 3: REFERENCE MATERIAL (if available) │ │
│ │ │ │
│ │ - Model answer / key points to cover │ │
│ │ - Explanation / marking scheme │ │
│ │ - Related course content (RAG context) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 4: RUBRIC CRITERIA (if provided) │ │
│ │ │ │
│ │ - Content accuracy (weight: X%) │ │
│ │ - Clarity of expression (weight: Y%) │ │
│ │ - Use of examples (weight: Z%) │ │
│ │ - Grammar and spelling (weight: W%) │ │
│ │ - ... (other criteria) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 5: STUDENT'S ANSWER │ │
│ │ │ │
│ │ - Full student response │ │
│ │ - Submission metadata (time spent, etc.) │ │
│ │ - OCR source flag (if from handwriting) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 6: EVALUATION INSTRUCTIONS │ │
│ │ │ │
│ │ - Evaluate fairly based on grade level expectations │ │
│ │ - Award partial credit for partially correct answers │ │
│ │ - Consider effort and understanding, not just keywords │ │
│ │ - If OCR source, be lenient on minor transcription errors│ │
│ │ - Provide constructive, encouraging feedback │ │
│ │ - Be specific about what was done well │ │
│ │ - Be specific about what could be improved │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SECTION 7: OUTPUT FORMAT (JSON) │ │
│ │ │ │
│ │ { │ │
│ │ "suggested_marks": <number>, │ │
│ │ "max_marks": <number>, │ │
│ │ "criteria_scores": {...}, │ │
│ │ "feedback": "<detailed feedback>", │ │
│ │ "strengths": ["...", "..."], │ │
│ │ "improvements": ["...", "..."], │ │
│ │ "confidence": <0-1> │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
6.3 Evaluation by Question Type¶
| Question Type | Evaluation Approach | Key Criteria |
|---|---|---|
| Short Answer | Compare key concepts with model answer | Accuracy, completeness, terminology |
| Long Answer / Essay | Holistic rubric-based evaluation | Content, structure, arguments, evidence |
| Case Study | Problem-solving assessment | Analysis, application, recommendations |
| Lab Practical | Procedure + result evaluation | Methodology, observations, conclusions |
| Research Paper | Academic writing standards | Thesis, literature review, analysis, citations |
| Project | Multi-criteria assessment | Innovation, execution, documentation |
| Coding | Code execution + quality | Correctness, efficiency, style |
6.4 Evaluation by Assignment Type¶
Essays (essay, research_paper): - Introduction clarity and thesis statement - Body paragraph organization - Evidence and examples - Logical flow and transitions - Conclusion and synthesis - Grammar, spelling, vocabulary - Word count adherence
Case Studies (case_study): - Problem identification - Stakeholder analysis - Alternative solutions considered - Recommended solution justification - Implementation considerations - Risk assessment
Lab Practicals (lab_practical): - Hypothesis clarity - Methodology correctness - Data recording accuracy - Observation quality - Analysis and interpretation - Conclusion validity - Safety considerations mentioned
Projects (project, group_project): - Problem statement understanding - Approach and methodology - Implementation quality - Results and deliverables - Documentation completeness - Creativity and innovation
6.5 Evaluation Confidence Scoring¶
The LLM provides a confidence score (0-1) for each evaluation:
| Confidence | Meaning | UI Treatment |
|---|---|---|
| 0.90 - 1.00 | High confidence | Green indicator, suggest direct acceptance |
| 0.75 - 0.89 | Good confidence | No special indicator |
| 0.60 - 0.74 | Moderate confidence | Yellow indicator, review recommended |
| < 0.60 | Low confidence | Red indicator, manual review required |
Factors reducing confidence: - Ambiguous or unclear student answer - No model answer provided for reference - OCR-extracted content with low confidence - Answer length significantly different from expected - Mixed language content - Technical content outside common knowledge
7. File Processing Pipeline¶
7.1 Supported File Types¶
| File Type | Extension | Max Size | Processing Method |
|---|---|---|---|
| PDF (Digital) | 20 MB | PyMuPDF text extraction | |
| PDF (Scanned) | 20 MB | Page → Image → OCR | |
| Word Document | .docx | 10 MB | python-docx parsing |
| Word (Legacy) | .doc | 10 MB | Convert to .docx first |
| Images | .jpg, .png, .heic | 10 MB | Direct OCR |
| Plain Text | .txt | 5 MB | Direct read |
7.2 PDF Processing Flow¶
┌─────────────────────────────────────────────────────────────────┐
│ PDF PROCESSING FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Input: PDF File │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Open with │ │
│ │ PyMuPDF (fitz) │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Attempt Text │ │
│ │ Extraction │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ Yes ┌─────────────────┐ │
│ │ Has Extractable │───────────▶│ Extract Text │ │
│ │ Text? │ │ Directly │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ No │ │
│ ▼ │ │
│ ┌─────────────────┐ │ │
│ │ Scanned PDF │ │ │
│ │ Detection │ │ │
│ └────────┬────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────┐ │ │
│ │ For Each Page: │ │ │
│ │ Convert to PNG │ │ │
│ │ (300 DPI) │ │ │
│ └────────┬────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────┐ │ │
│ │ OCR Each Page │ │ │
│ │ (Vision/Gemini) │ │ │
│ └────────┬────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────┐ │ │
│ │ Combine Page │◄────────────────────┘ │
│ │ Texts │ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Structured │ │
│ │ Output │ │
│ │ - Full text │ │
│ │ - Page count │ │
│ │ - Has images │ │
│ │ - Is scanned │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
7.3 Word Document Processing¶
┌─────────────────────────────────────────────────────────────────┐
│ DOCX PROCESSING FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Input: .docx File │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Open with │ │
│ │ python-docx │ │
│ └────────┬────────┘ │
│ │ │
│ ├──────────────────────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Extract │ │ Extract │ │
│ │ Paragraphs │ │ Tables │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌─────────────────┐ │
│ │ │ Convert to │ │
│ │ │ Markdown Tables │ │
│ │ └────────┬────────┘ │
│ │ │ │
│ └──────────┬─────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Check for │ │
│ │ Embedded Images │ │
│ └────────┬────────┘ │
│ │ │
│ ┌────────────┼────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ No │ │ Has │ │ Has │ │
│ │ Images │ │ Diagrams│ │ Photos │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ │ │ │ │
│ │ ▼ ▼ │
│ │ ┌─────────────────────┐ │
│ │ │ Extract & OCR │ │
│ │ │ Embedded Images │ │
│ │ └──────────┬──────────┘ │
│ │ │ │
│ └────────┬─────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Combined │ │
│ │ Structured Text │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
7.4 Mixed Content Handling¶
For files containing multiple content types:
┌─────────────────────────────────────────────────────────────────┐
│ MIXED CONTENT PROCESSING │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Example: Lab Report PDF with typed text + handwritten notes │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ PAGE ANALYSIS │ │
│ │ │ │
│ │ Page 1: [Typed Introduction] │ │
│ │ Page 2: [Typed Procedure] + [Handwritten Calculations] │ │
│ │ Page 3: [Scanned Data Table] │ │
│ │ Page 4: [Handwritten Conclusion] │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SEGMENT DETECTION │ │
│ │ │ │
│ │ For each page region: │ │
│ │ - Detect if typed or handwritten │ │
│ │ - Identify tables, equations, diagrams │ │
│ │ - Mark segment boundaries │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ PARALLEL PROCESSING │ │
│ │ │ │
│ │ Typed Text ──────▶ Direct Extraction │ │
│ │ Handwritten ─────▶ Gemini OCR │ │
│ │ Tables ──────────▶ Structure Detection + OCR │ │
│ │ Equations ───────▶ Math-aware OCR (LaTeX output) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ CONTENT FUSION │ │
│ │ │ │
│ │ - Merge segments in reading order │ │
│ │ - Preserve structure (headings, lists, tables) │ │
│ │ - Mark OCR confidence per segment │ │
│ │ - Generate unified document │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
8. Rubric System¶
8.1 What is a Rubric?¶
A rubric is a scoring guide that describes criteria for evaluating student work. It ensures: - Consistency - Same criteria applied to all students - Transparency - Students know how they'll be graded - Efficiency - Faster grading with clear criteria - Feedback - Structured feedback based on criteria
8.2 Rubric Structure¶
┌─────────────────────────────────────────────────────────────────┐
│ RUBRIC STRUCTURE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Rubric Name: "Essay Evaluation - Class 10 English" │
│ Max Marks: 20 │
│ Applicable: essay, long_answer │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ CRITERIA 1: Content & Ideas (Weight: 35%) │ │
│ │ │ │
│ │ Level 4 (Excellent) - 9-10 points │ │
│ │ Thorough understanding, insightful analysis, │ │
│ │ original ideas, well-supported arguments │ │
│ │ │ │
│ │ Level 3 (Good) - 7-8 points │ │
│ │ Good understanding, relevant analysis, │ │
│ │ adequate support for arguments │ │
│ │ │ │
│ │ Level 2 (Satisfactory) - 5-6 points │ │
│ │ Basic understanding, some analysis, │ │
│ │ limited support for arguments │ │
│ │ │ │
│ │ Level 1 (Needs Improvement) - 1-4 points │ │
│ │ Limited understanding, minimal analysis, │ │
│ │ lacks supporting evidence │ │
│ │ │ │
│ │ Level 0 (Not Attempted) - 0 points │ │
│ │ No response or completely off-topic │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ CRITERIA 2: Organization & Structure (Weight: 25%) │ │
│ │ │ │
│ │ Level 4: Clear intro, logical body, strong conclusion │ │
│ │ Level 3: Good structure, minor flow issues │ │
│ │ Level 2: Basic structure, some organization problems │ │
│ │ Level 1: Poor organization, hard to follow │ │
│ │ Level 0: No discernible structure │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ CRITERIA 3: Language & Grammar (Weight: 25%) │ │
│ │ │ │
│ │ Level 4: Excellent vocabulary, no errors │ │
│ │ Level 3: Good vocabulary, minor errors │ │
│ │ Level 2: Basic vocabulary, some errors │ │
│ │ Level 1: Limited vocabulary, many errors │ │
│ │ Level 0: Incomprehensible │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ CRITERIA 4: Examples & Evidence (Weight: 15%) │ │
│ │ │ │
│ │ Level 4: Multiple relevant examples, well-integrated │ │
│ │ Level 3: Good examples, mostly relevant │ │
│ │ Level 2: Few examples, somewhat relevant │ │
│ │ Level 1: Minimal or irrelevant examples │ │
│ │ Level 0: No examples provided │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
8.3 Rubric Categories by Assignment Type¶
| Assignment Type | Suggested Rubric Criteria |
|---|---|
| Essay | Content, Structure, Language, Examples, Originality |
| Research Paper | Thesis, Literature Review, Methodology, Analysis, Citations |
| Case Study | Problem Analysis, Solutions, Feasibility, Presentation |
| Lab Practical | Procedure, Observations, Calculations, Conclusions, Safety |
| Project | Planning, Implementation, Innovation, Documentation |
| Short Answer | Accuracy, Completeness, Terminology |
| Coding | Correctness, Efficiency, Style, Documentation |
8.4 Rubric Hierarchy¶
┌─────────────────────────────────────────────────────────────────┐
│ RUBRIC HIERARCHY │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Platform Level (VidyaNet Default) │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ - Generic Essay Rubric │ │
│ │ - Generic Case Study Rubric │ │
│ │ - Generic Lab Report Rubric │ │
│ │ - ... (built-in templates) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Organization Level │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ - BET Organization Essay Standards │ │
│ │ - BET Lab Report Standards │ │
│ │ (Org admins can customize for their schools) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ School Level │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ - BETIS High School Essay Rubric │ │
│ │ - BETIS Science Lab Rubric │ │
│ │ (School admins can customize for their school) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Subject Level │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ - Class 10 English Essay Rubric │ │
│ │ - Class 10 Physics Lab Rubric │ │
│ │ (Teachers can customize for their subject) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Assignment Level │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ - "My Summer Vacation" Essay Rubric │ │
│ │ - "Newton's Laws Lab" Rubric │ │
│ │ (Teachers can create assignment-specific rubrics) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
8.5 Rubric Application in LLM Evaluation¶
When a rubric is provided, the LLM evaluation:
- Receives rubric criteria in the prompt
- Evaluates each criterion independently
- Assigns level/score for each criterion
- Calculates weighted total based on weights
- Provides criterion-specific feedback
Example LLM Output with Rubric:
{
"criteria_scores": {
"content_ideas": {
"level": 3,
"score": 7.5,
"max_score": 10,
"feedback": "Good understanding of the topic. Could include more original insights."
},
"organization": {
"level": 4,
"score": 5,
"max_score": 5,
"feedback": "Excellent structure with clear introduction and conclusion."
},
"language_grammar": {
"level": 3,
"score": 4,
"max_score": 5,
"feedback": "Good vocabulary usage. Minor grammatical errors noted."
}
},
"total_marks": 16.5,
"max_marks": 20,
"overall_feedback": "Well-written essay with clear structure..."
}
9. Integration with Existing System¶
9.1 Submission Flow (Updated)¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ UPDATED SUBMISSION & EVALUATION FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Student System Teacher │
│ │ │ │ │
│ │── Start Assignment ─────▶│ │ │
│ │ │ │ │
│ │── Answer Questions ─────▶│ │ │
│ │ (text / file upload) │ │ │
│ │ │ │ │
│ │── Submit ───────────────▶│ │ │
│ │ │── Store Submission ──────▶│ │
│ │ │ │ │
│ │ │ ┌─────────────────────────────────┐ │
│ │ │ │ EVALUATION ROUTING │ │
│ │ │ │ │ │
│ │ │ │ Objective Questions (MCQ, TF) │ │
│ │ │ │ └── Auto-grade (existing) │ │
│ │ │ │ │ │
│ │ │ │ Subjective Questions │ │
│ │ │ │ └── Queue for AI Eval ─────┼────▶│
│ │ │ │ │ │
│ │ │ │ File Uploads │ │
│ │ │ │ └── Process Files ─────┼────▶│
│ │ │ │ └── OCR if needed ─────┼────▶│
│ │ │ │ └── Queue for AI Eval ─────┼────▶│
│ │ │ └─────────────────────────────────┘ │
│ │ │ │ │
│ │ │ ┌─────────────────────────────────┐ │
│ │ │ │ AI EVALUATION │ │
│ │ │ │ │ │
│ │ │ │ - Load rubric (if applicable) │ │
│ │ │ │ - Build evaluation prompt │ │
│ │ │ │ - Call LLM Gateway │ │
│ │ │ │ - Parse response │ │
│ │ │ │ - Store AI evaluation │ │
│ │ │ └─────────────────────────────────┘ │
│ │ │ │ │
│ │ │── Notify: AI Eval Ready ─▶│ │
│ │ │ │ │
│ │ │ ┌───────┴───────┐ │
│ │ │ │ │ │
│ │ │ ▼ ▼ │
│ │ │ ┌──────────┐ ┌──────────┐ │
│ │ │ │ Review │ │ Accept │ │
│ │ │ │ & Edit │ │ As-Is │ │
│ │ │ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ │ │ └───────┬───────┘ │
│ │ │ │ │
│ │ │◄── Finalize Grade ──────┘ │
│ │ │ │
│ │◄── Receive Grade ────────│ │
│ │◄── View Feedback ────────│ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
9.2 Existing Components Used¶
| Component | Current Use | New Use in Evaluation |
|---|---|---|
| LLMGateway | AI Tutor conversations | Evaluation generation |
| StorageService | Media uploads | File retrieval for parsing |
| MediaAttachment | Store uploaded files | Reference for OCR processing |
| Grade model | Store final grades | Link to AI evaluation |
| Submission model | Store student answers | Input for evaluation |
| Assignment model | Assignment structure | Context for evaluation |
| Question model | Question details | Evaluation criteria |
9.3 New Components Added¶
| Component | Purpose | Integration Point |
|---|---|---|
| AIEvaluationService | Orchestrate evaluation | Called after submission |
| OCRService | Text extraction from images | Used by AIEvaluationService |
| FileParserService | Parse PDF/DOCX | Used by AIEvaluationService |
| RubricService | Manage evaluation criteria | Used by AIEvaluationService |
| AIEvaluation model | Store evaluation results | Linked to Submission |
| EvaluationRubric model | Store rubrics | Linked to Assignment/Subject |
| OCRProcessingLog model | OCR audit trail | Linked to MediaAttachment |
9.4 API Endpoint Integration¶
Existing Endpoints (unchanged):
- POST /assignments - Create assignment
- POST /assignments/{id}/submit - Student submit
- POST /assignments/submissions/{id}/grade - Manual grade (still available)
New Endpoints:
- POST /evaluations/submissions/{id}/evaluate - Trigger AI evaluation
- GET /evaluations/submissions/{id} - Get AI evaluation results
- POST /evaluations/submissions/{id}/accept - Accept AI grade
- POST /evaluations/ocr/extract - Extract text from file
- CRUD /evaluations/rubrics - Manage rubrics
9.5 Teacher Workflow Integration¶
┌─────────────────────────────────────────────────────────────────┐
│ TEACHER GRADING WORKFLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Submissions List View │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Assignment: "Essay on Climate Change" │ │
│ │ Total: 35 submissions | Graded: 12 | Pending: 23 │ │
│ │ │ │
│ │ [Run AI Evaluation for All Pending] ← NEW BUTTON │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────┐ │ │
│ │ │ Student │ Status │ AI Score │ Actions │ │ │
│ │ ├─────────────┼───────────┼──────────┼────────────┤ │ │
│ │ │ Priya S. │ Submitted │ 16/20 │ [Review] │ │ │
│ │ │ Rahul K. │ Submitted │ 14/20 │ [Review] │ │ │
│ │ │ Ananya M. │ Submitted │ 18/20 │ [Accept] │ │ │
│ │ │ Vikram P. │ Submitted │ - │ [Evaluate] │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ Individual Submission View │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Student: Priya Sharma │ │
│ │ Submitted: Jan 15, 2025 at 3:45 PM │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Student Answer │ │ AI Evaluation │ ← NEW PANEL │ │
│ │ │ │ │ │ │ │
│ │ │ [Essay text or │ │ Suggested: 16/20│ │ │
│ │ │ file preview] │ │ │ │ │
│ │ │ │ │ Criteria Scores:│ │ │
│ │ │ If handwritten: │ │ - Content: 7/10 │ │ │
│ │ │ [OCR Preview] │ │ - Structure: 4/5│ │ │
│ │ │ Confidence: 85% │ │ - Language: 4/5 │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ Strengths: │ │ │
│ │ │ │ │ - Good examples │ │ │
│ │ │ │ │ - Clear thesis │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ Improvements: │ │ │
│ │ │ │ │ - More evidence │ │ │
│ │ │ │ │ - Conclusion │ │ │
│ │ └─────────────────┘ └─────────────────┘ │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────┐ │ │
│ │ │ Final Marks: [16] / 20 (can adjust) │ │ │
│ │ │ Additional Feedback: [________________] │ │ │
│ │ │ │ │ │
│ │ │ [Accept AI Grade] [Override & Grade Manually] │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
10. Data Flow Diagrams¶
10.1 Complete Evaluation Data Flow¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ COMPLETE EVALUATION DATA FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ │
│ │ Student │ │
│ │ Submits │ │
│ └────┬─────┘ │
│ │ │
│ │ 1. Submission │
│ │ {answers: {...}, files: [...]} │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ submissions │ │
│ │ (table) │ │
│ └──────────┬───────────┘ │
│ │ │
│ │ 2. Trigger Evaluation │
│ │ (auto or manual) │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ AIEvaluationService │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ STEP 1: Load Context │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │
│ │ │ │ Submission │ │ Assignment │ │ Rubric │ │ │ │
│ │ │ │ + Answers │ │ + Questions │ │ (optional) │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ STEP 2: Process Each Answer │ │ │
│ │ │ │ │ │
│ │ │ For each question: │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────────────────────────────────────┐ │ │ │
│ │ │ │ Text Answer? │ │ │ │
│ │ │ │ └── Direct to LLM │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ File Upload? │ │ │ │
│ │ │ │ ├── Image file? │ │ │ │
│ │ │ │ │ └── OCRService.extract() │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ ├── PDF file? │ │ │ │
│ │ │ │ │ └── FileParser.parse_pdf() │ │ │ │
│ │ │ │ │ └── If scanned: OCRService.extract() │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ ├── DOCX file? │ │ │ │
│ │ │ │ │ └── FileParser.parse_docx() │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ └── Extracted text to LLM │ │ │ │
│ │ │ └─────────────────────────────────────────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ STEP 3: LLM Evaluation │ │ │
│ │ │ │ │ │
│ │ │ Build Prompt: │ │ │
│ │ │ - Context (assignment, subject, grade level) │ │ │
│ │ │ - Question details │ │ │
│ │ │ - Rubric criteria (if available) │ │ │
│ │ │ - Student answer (text) │ │ │
│ │ │ - Evaluation instructions │ │ │
│ │ │ │ │ │
│ │ │ Call LLMGateway: │ │ │
│ │ │ ┌─────────────────────────────────────────────────┐ │ │ │
│ │ │ │ Gemini 2.0 Flash ──▶ Claude Sonnet 4 ──▶ GPT-4o│ │ │ │
│ │ │ │ (primary) (fallback 1) (fallback 2) │ │ │
│ │ │ └─────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │
│ │ │ Parse Response: │ │ │
│ │ │ - suggested_marks │ │ │
│ │ │ - criteria_scores │ │ │
│ │ │ - feedback │ │ │
│ │ │ - strengths, improvements │ │ │
│ │ │ - confidence │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
│ │ │ STEP 4: Aggregate & Store │ │ │
│ │ │ │ │ │
│ │ │ - Combine per-question evaluations │ │ │
│ │ │ - Calculate total suggested marks │ │ │
│ │ │ - Generate overall feedback │ │ │
│ │ │ - Store in ai_evaluations table │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────┬───────────────────────────────────┘ │
│ │ │
│ │ 3. Evaluation Result │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ ai_evaluations │ │
│ │ (table) │ │
│ └──────────┬───────────┘ │
│ │ │
│ │ 4. Teacher Review │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Teacher │ │
│ │ Reviews & Accepts │ │
│ │ (or Overrides) │ │
│ └──────────┬───────────┘ │
│ │ │
│ │ 5. Final Grade │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ grades │ │
│ │ (table) │ │
│ │ (ai_evaluation_id) │ │
│ └──────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
10.2 OCR-Specific Data Flow¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ OCR DATA FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ │
│ │ Student │ │
│ │ Uploads │ │
│ │ Photo │ │
│ └────┬─────┘ │
│ │ │
│ │ 1. Upload to GCS │
│ │ {file: handwritten.jpg} │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ media_attachments │────▶│ Google Cloud │ │
│ │ (table) │ │ Storage (GCS) │ │
│ │ │ │ │ │
│ │ - storage_path │ │ vidyanet-uploads/ │ │
│ │ - mime_type │ │ uploads/{user}/ │ │
│ │ - size_bytes │ │ handwritten.jpg │ │
│ └──────────┬───────────┘ └──────────────────────┘ │
│ │ │ │
│ │ 2. Trigger OCR │ 3. Download │
│ │ │ │
│ ▼ │ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ OCRService │ │
│ │ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ Pre-processing │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐│ │ │
│ │ │ │ Deskew │──▶│ Denoise │──▶│Contrast │──▶│ Resize ││ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘│ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ Content Analysis │ │ │
│ │ │ │ │ │
│ │ │ - Detect: printed vs handwritten │ │ │
│ │ │ - Detect: equations, diagrams, tables │ │ │
│ │ │ - Estimate: language (en, hi, kn, etc.) │ │ │
│ │ │ - Assess: quality score │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ Model Selection │ │ │
│ │ │ │ │ │
│ │ │ Quality Score > 0.75 ──────▶ Google Cloud Vision API │ │ │
│ │ │ │ │ │
│ │ │ Quality Score <= 0.75 ─────▶ Gemini 2.0 Flash Vision │ │ │
│ │ │ OR has equations/diagrams │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ OCR Execution │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │
│ │ │ │ Google Vision API │ │ Gemini Vision │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ - Document detection│ │ - Instruction-based │ │ │ │
│ │ │ │ - Word confidence │ │ - Context-aware │ │ │ │
│ │ │ │ - Language detect │ │ - Math/diagram │ │ │ │
│ │ │ │ - Bounding boxes │ │ - LaTeX output │ │ │ │
│ │ │ └─────────────────────┘ └─────────────────────┘ │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ Post-processing │ │ │
│ │ │ │ │ │
│ │ │ - Clean artifacts │ │ │
│ │ │ - Normalize whitespace │ │ │
│ │ │ - Domain-aware spell check │ │ │
│ │ │ - Structure content │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────┬───────────────────────────────────┘ │
│ │ │
│ │ 4. OCR Result │
│ │ {text, confidence, language} │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ ocr_processing_logs │ │
│ │ (table) │ │
│ │ │ │
│ │ - extracted_text │ │
│ │ - confidence_score │ │
│ │ - language_detected │ │
│ │ - ocr_provider │ │
│ │ - processing_time │ │
│ └──────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
11. API Design¶
11.1 New Endpoints¶
| Endpoint | Method | Description |
|---|---|---|
/evaluations/submissions/{id}/evaluate |
POST | Trigger AI evaluation |
/evaluations/submissions/{id} |
GET | Get evaluation results |
/evaluations/submissions/{id}/accept |
POST | Accept AI grade |
/evaluations/batch |
POST | Batch evaluate multiple submissions |
/evaluations/ocr/extract |
POST | Extract text via OCR |
/evaluations/rubrics |
GET | List available rubrics |
/evaluations/rubrics |
POST | Create new rubric |
/evaluations/rubrics/{id} |
GET | Get rubric details |
/evaluations/rubrics/{id} |
PATCH | Update rubric |
/evaluations/rubrics/{id} |
DELETE | Delete rubric |
11.2 Request/Response Examples¶
Trigger Evaluation:
POST /api/v1/evaluations/submissions/{submission_id}/evaluate
Request:
{
"rubric_id": "uuid" (optional),
"check_plagiarism": false,
"force_reevaluate": false
}
Response:
{
"evaluation_id": "uuid",
"status": "processing" | "completed",
"message": "Evaluation started",
"result": { ... } (if completed)
}
Get Evaluation Results:
GET /api/v1/evaluations/submissions/{submission_id}
Response:
{
"id": "uuid",
"submission_id": "uuid",
"status": "completed",
"total_suggested_marks": 16.5,
"max_marks": 20,
"question_evaluations": {
"question_id_1": {
"suggested_marks": 8.5,
"max_marks": 10,
"criteria_scores": {
"content": 8,
"clarity": 9
},
"feedback": "Good understanding...",
"strengths": ["Clear explanation", "Good examples"],
"improvements": ["More detail needed on..."],
"confidence": 0.85
}
},
"overall_feedback": "Well-written submission...",
"overall_strengths": ["..."],
"overall_improvements": ["..."],
"ocr_used": true,
"ocr_confidence": 0.82,
"plagiarism_score": null,
"llm_provider": "gemini",
"processing_time_ms": 2450,
"created_at": "2025-01-15T10:30:00Z"
}
Accept AI Grade:
POST /api/v1/evaluations/submissions/{submission_id}/accept
Request:
{
"adjusted_question_marks": {
"question_id_1": {
"marks": 9, // Teacher adjusted from 8.5
"feedback": "Additional feedback..."
}
},
"override_total_marks": null, // or specific number
"additional_feedback": "Great improvement from last time!"
}
Response:
{
"grade_id": "uuid",
"submission_id": "uuid",
"marks_obtained": 17.5,
"percentage": 87.5,
"graded_at": "2025-01-15T11:00:00Z"
}
12. Database Schema¶
12.1 New Tables¶
evaluation_rubrics: | Column | Type | Description | |--------|------|-------------| | id | UUID | Primary key | | org_unit_id | UUID | FK to org_units (RLS) | | name | VARCHAR(255) | Rubric name | | description | TEXT | Detailed description | | assignment_types | VARCHAR[] | Applicable assignment types | | question_types | VARCHAR[] | Applicable question types | | subject_id | UUID | FK to subjects (optional) | | criteria | JSONB | Criteria configuration | | max_marks | FLOAT | Maximum marks | | created_by_id | UUID | FK to users | | created_at | TIMESTAMP | Creation time |
ai_evaluations: | Column | Type | Description | |--------|------|-------------| | id | UUID | Primary key | | submission_id | UUID | FK to submissions | | org_unit_id | UUID | FK to org_units (RLS) | | evaluation_type | VARCHAR(50) | full/per_question/file_based | | rubric_id | UUID | FK to evaluation_rubrics | | llm_provider | VARCHAR(50) | gemini/claude/openai | | llm_model | VARCHAR(100) | Model version used | | status | VARCHAR(20) | pending/processing/completed/failed | | question_evaluations | JSONB | Per-question results | | total_suggested_marks | FLOAT | Total suggested marks | | overall_feedback | TEXT | Overall feedback text | | overall_strengths | JSONB | List of strengths | | overall_improvements | JSONB | List of improvements | | ocr_extracted_text | JSONB | OCR results per question | | ocr_confidence | FLOAT | Average OCR confidence | | plagiarism_score | FLOAT | Plagiarism percentage | | plagiarism_sources | JSONB | Matched sources | | processing_time_ms | INTEGER | Processing duration | | token_usage | JSONB | LLM token usage | | error_message | TEXT | Error details if failed | | created_at | TIMESTAMP | Creation time |
ocr_processing_logs: | Column | Type | Description | |--------|------|-------------| | id | UUID | Primary key | | media_attachment_id | UUID | FK to media_attachments | | extracted_text | TEXT | OCR result | | confidence_score | FLOAT | Confidence (0-1) | | language_detected | VARCHAR(10) | Detected language | | ocr_provider | VARCHAR(50) | Provider used | | processing_time_ms | INTEGER | Processing duration | | status | VARCHAR(20) | Status | | error_message | TEXT | Error if failed | | created_at | TIMESTAMP | Creation time |
12.2 Modified Tables¶
grades (add column): | Column | Type | Description | |--------|------|-------------| | ai_evaluation_id | UUID | FK to ai_evaluations (nullable) |
13. Security & Privacy¶
13.1 Data Protection¶
| Concern | Mitigation |
|---|---|
| Student data in LLM | No PII sent to LLM; only academic content |
| File access | RLS policies ensure only authorized access |
| OCR logs | Tied to media_attachment access control |
| Evaluation results | Teachers see all; students see only their own |
| Rubrics | Org-level access control |
13.2 Audit Trail¶
All AI evaluations maintain: - LLM provider and model used - Processing time and token usage - Original evaluation vs teacher adjustments - Timestamp of evaluation and acceptance
13.3 Compliance¶
| Requirement | Implementation |
|---|---|
| DPDP Act | No student PII in external API calls |
| FERPA | Education records protected by RLS |
| Audit | Full audit trail for all evaluations |
14. Performance Considerations¶
14.1 Processing Times¶
| Operation | Expected Time |
|---|---|
| Single text answer evaluation | 2-5 seconds |
| OCR (single image) | 3-8 seconds |
| PDF parsing (10 pages) | 5-15 seconds |
| Full submission (5 questions) | 15-30 seconds |
| Batch evaluation (30 submissions) | 5-10 minutes |
14.2 Scaling Strategy¶
| Load | Strategy |
|---|---|
| Low (< 100/day) | Synchronous processing |
| Medium (100-1000/day) | Background task queue |
| High (> 1000/day) | Dedicated worker pods + caching |
14.3 Caching¶
| Item | Cache Strategy |
|---|---|
| OCR results | Cache for 7 days (same file = same OCR) |
| Rubrics | Cache for 1 hour (rarely change) |
| LLM responses | No caching (unique per submission) |
15. Implementation Phases¶
Phase 1: Foundation (Week 1-2)¶
- [ ] Database schema and migrations
- [ ] Basic AI evaluation service
- [ ] Text answer evaluation with LLM
- [ ] API endpoints for evaluation
Phase 2: OCR Integration (Week 3-4)¶
- [ ] Google Cloud Vision API integration
- [ ] Gemini Vision for complex handwriting
- [ ] Image pre-processing pipeline
- [ ] OCR confidence handling
Phase 3: File Processing (Week 5-6)¶
- [ ] PDF text extraction (PyMuPDF)
- [ ] Scanned PDF → OCR pipeline
- [ ] DOCX parsing
- [ ] Mixed content handling
Phase 4: Rubric System (Week 7-8)¶
- [ ] Rubric data model
- [ ] Rubric CRUD API
- [ ] Rubric-based evaluation prompts
- [ ] Default rubric templates
Phase 5: Frontend Integration (Week 9-10)¶
- [ ] AI evaluation panel in grading view
- [ ] OCR preview component
- [ ] Rubric management UI
- [ ] Batch evaluation UI
Phase 6: Testing & Optimization (Week 11-12)¶
- [ ] Unit tests for evaluation service
- [ ] Integration tests for OCR pipeline
- [ ] Performance optimization
- [ ] Documentation
16. Cost Analysis¶
16.1 Per-Evaluation Costs¶
| Component | Cost Per Unit | Typical Usage | Cost Per Evaluation |
|---|---|---|---|
| Gemini 2.0 Flash | $0.075/1M input, $0.30/1M output | ~2000 tokens | ~$0.001 |
| Google Vision OCR | $1.50/1000 images | 1 image | ~$0.0015 |
| Gemini Vision | $0.075/1M input | ~1000 tokens | ~$0.0001 |
Average cost per evaluation: $0.002 - $0.005
16.2 Monthly Cost Projection¶
| School Size | Submissions/Month | Estimated Cost |
|---|---|---|
| Small (100 students) | 500 | $2.50 |
| Medium (500 students) | 2,500 | $12.50 |
| Large (2000 students) | 10,000 | $50.00 |
17. Risks & Mitigations¶
| Risk | Impact | Mitigation |
|---|---|---|
| LLM hallucination | Wrong grades | Human-in-the-loop review required |
| OCR errors | Misread content | Confidence thresholds, manual review flags |
| API rate limits | Slow processing | Queue-based processing, multi-provider fallback |
| Cost overruns | Budget issues | Usage limits per school, monitoring alerts |
| Bias in evaluation | Unfair grading | Regular prompt tuning, teacher calibration |
| Privacy concerns | Data exposure | No PII in API calls, strict access control |
Appendix A: OCR Model Comparison¶
| Feature | Google Vision API | Gemini Vision | Tesseract |
|---|---|---|---|
| Handwriting quality | Good | Excellent | Fair |
| Printed text | Excellent | Good | Good |
| Math equations | Poor | Good | Poor |
| Diagrams | Poor | Good | Poor |
| Multi-language | Excellent | Good | Good |
| Latency | Fast (1-3s) | Medium (3-8s) | Slow (5-15s) |
| Cost | $1.50/1K | ~$0.10/1K | Free |
| Offline | No | No | Yes |
Appendix B: Evaluation Prompt Template¶
# Assignment Evaluation
## Context
- **Assignment:** {assignment_title}
- **Type:** {assignment_type}
- **Subject:** {subject_name}
- **Grade Level:** {grade_level}
- **Institution:** {institution_type}
## Question
**Question #{question_number} ({max_marks} marks):**
{question_content}
{if model_answer}
**Expected Answer / Key Points:**
{model_answer}
{endif}
{if rubric}
## Evaluation Rubric
{for criterion in rubric.criteria}
- **{criterion.name}** (Weight: {criterion.weight}%)
- Excellent: {criterion.excellent_description}
- Good: {criterion.good_description}
- Satisfactory: {criterion.satisfactory_description}
- Needs Improvement: {criterion.needs_improvement_description}
{endfor}
{endif}
## Student's Answer
{student_answer}
{if ocr_source}
*Note: This answer was extracted from a handwritten submission via OCR
(confidence: {ocr_confidence}%). Please be lenient about minor transcription errors.*
{endif}
## Evaluation Instructions
1. Evaluate the answer fairly based on the question requirements
2. Consider the student's grade level when assessing
3. Award partial credit for partially correct answers
4. Provide specific, constructive feedback
5. Identify 2-3 strengths in the answer
6. Suggest 2-3 areas for improvement
7. Be encouraging while being honest about areas to improve
## Response Format
Respond in JSON format:
{
"suggested_marks": <number 0 to {max_marks}>,
"criteria_scores": {
"<criterion_name>": {
"level": "<excellent|good|satisfactory|needs_improvement>",
"score": <number>,
"feedback": "<specific feedback for this criterion>"
}
},
"feedback": "<detailed overall feedback>",
"strengths": ["<strength 1>", "<strength 2>"],
"improvements": ["<improvement 1>", "<improvement 2>"],
"confidence": <number 0 to 1>
}
Appendix C: Glossary¶
| Term | Definition |
|---|---|
| OCR | Optical Character Recognition - converting images of text to machine-readable text |
| LLM | Large Language Model - AI model for text generation and understanding |
| Rubric | Scoring guide with criteria and levels for evaluating student work |
| RLS | Row-Level Security - database-level access control |
| Confidence Score | Numerical measure (0-1) of how certain the system is about a result |
| Human-in-the-loop | Requiring human review/approval before finalizing AI decisions |
Document Version: 1.0 Last Updated: January 2025