AI Document Summarizer

Metrics are sequenced around what you can learn at each stage. Start with "does it work?", then "is it valuable?", then "how do we improve?"

Tier 1: MVP launch (weeks 1-4): reliability

• >95% summary generation rate: technical reliability baseline
• P95 latency <10s: user-perceived performance
• <1% error rate: system stability
• <$0.10 cost per summary: economic sustainability

Tier 2: Post-MVP (months 2-3): value

• >70% positive feedback (thumbs up): direct quality signal
• >10% feedback rate: if fewer than 5% of users give any feedback, the quality signal isn't meaningful yet. A/B test the feedback UX before trusting the number.
• <10% edit rate: the most important quality proxy. When users replace an AI summary with their own text, that's a direct signal the output isn't meeting expectations.
• Measurable reduction in files opened per search (A/B test): efficiency gain

Tier 3: Optimization (month 3+): continuous improvement

• Automated quality scores (LLM-as-judge): scalable measurement of faithfulness, relevance, and conciseness
• Edit pattern analysis: what users consistently change reveals actionable prompt improvements
• Time saved (user surveys): perceived value and ROI

• Manual summary replacement rate. When users replace an AI summary with their own text, that's a direct quality signal. Thresholds: under 5% is excellent, 5-10% is healthy, 10-15% warrants investigation by file type, and above 20% means users are doing manual work and the feature is failing. Tracking what users change (not just how often) reveals actionable prompt improvements. For example, if users consistently add pricing details the summary missed, the prompt should be updated to extract pricing explicitly.
• Session pattern analysis. If a user opens an attached file and returns to the app within 10 seconds, the file likely wasn't relevant, which may indicate a misleading summary. If they stay away for 2+ minutes, they're probably reading it, meaning the summary led them to the right content.

Quick return rate (file opened then immediately closed) seems intuitive but is ambiguous. A user closing a file in under 10 seconds could mean three things: they were scanning multiple files (good behavior), the summary was misleading (bad quality), or the summary told them everything they needed (good summary). Without additional context, the metric is uninterpretable.

Similarly, "find files faster" only works if you first understand how users discover files (search, browse, or notification). The metric needs to be scoped to a specific discovery pattern before it means anything.

• Regression set: 20 representative documents (short, long, technical, non-technical) evaluated after each prompt change
• Human rubric: scored 1-5 on four dimensions: correctness, coverage, specificity, tone
• Prompt versioning: changes can be A/B tested behind feature flags and rolled back if quality drops

Not all failures are equal. A failed summarization that surfaces a clear error message and retry option is acceptable because the user knows what happened and what to do. A silently wrong summary is not acceptable because it erodes trust without the user knowing.

Rough estimates (prices subject to change with provider pricing updates):

• GPT-4o-mini input: ~$0.15 / 1M tokens
• Avg tokens per summary pair: ~2,000
• Cost per summary: ~$0.0003
• 10K summaries / month: ~$3 / month

OCR and multi-format parsing add cost. Cost-per-summary is tracked as a production metric alongside quality; the goal is to optimize without quality loss.

The POC is synchronous: the user waits while the API processes. Production needs to handle timeouts, retries, and long documents gracefully.

• Async job queue: Supabase (chosen for built-in RLS, auth, and Postgres, giving one service for persistence and access control) creates a job row on upload. An AWS Lambda worker (serverless, scales to zero when idle, no infra to manage) processes jobs and writes results back.
• Summary persistence: summaries are stored in Supabase and served from the database on subsequent views, not regenerated. This eliminates redundant API calls, reduces cost, and makes summaries available instantly across sessions.
• Run telemetry from day 1: every summarization run logs the model, token count, latency, and errors to Supabase. This is non-negotiable. Without it, quality and cost issues are invisible.
• Feature flags: enable or disable summarization without breaking existing upload flows. This is both a launch mechanism and a rollback mechanism.

• Supabase Auth + RLS (row-level security): owner-only access by default. RLS is chosen over application-level permission checks because access control lives in the database layer, not scattered across API routes. A user cannot read another user's summaries, enforced at the query level.
• Status-driven UX: the summary card reflects real job states (queued, processing, succeeded, failed) pulled from the database, not local ephemeral state.
• Integration and E2E tests before expanding scope. The pipeline must be proven reliable before adding more complexity.

• Multi-format support: .txt, .md, .docx in addition to PDF.
• OCR for scanned PDFs: this comes after monitoring exists because OCR adds both cost and new failure modes that need the observability from Phase 1.
• Custom summary replacement + regenerate: users can write their own summary (replacing the AI version) or regenerate a new AI summary. Both are trust mechanisms that give users control. Replacement rate also feeds back into quality measurement.
• Manual and batch summarization: auto-summarization only covers new uploads. Manual triggers let users summarize existing files on demand, and batch triggers let admins backfill critical content sets. Without this, the feature only works going forward and ignores the existing library.
• Coverage metric: files with summaries divided by total files in the system. Tracks adoption over time. Target: above 80% within 6 months through a combination of auto-summarization, manual triggers, and batch backfill.

• Phased rollout via feature flags: start at 5% of users, expand to 25%, 50%, then 100% based on stability metrics. Rollback triggers: error rate exceeds 5%, P95 latency exceeds 10s, cost per summary exceeds $0.50, or user complaints exceed 10 tickets/day. Any trigger kills the feature instantly via feature flag, no code deploy required.
• Vercel for hosting: zero-config Next.js deployment with preview environments per PR.
• Dashboards: per-step latency, cost (tokens), success rate, breakdown by file type and provider.
• Automated alerts: spend spikes, error-rate thresholds, queue backlog, timeout surges, with runbooks for each scenario.
• Optimization based on production data: tune chunking, prompts, and retry strategies using real telemetry. Reduce cost without quality loss.

• Graph database for document relationships: once summaries are persisted, they become a foundation for understanding relationships between documents. A graph database (e.g., Neo4j) can store semantic similarity edges between documents based on their summary embeddings.
• Related document suggestions: when a user attaches a document to a project, the system suggests other documents with semantically similar content. This turns summaries from a passive feature (read and move on) into an active discovery tool.
• Embedding pipeline: summaries are embedded at generation time and stored as vectors. Similarity is computed against the existing corpus, and relationships above a confidence threshold are written as edges in the graph. This runs asynchronously alongside summarization, reusing the same job queue infrastructure from Phase 1.