Recall: AI Knowledge Assistant

Overview

I built Recall to turn a collection of documents and voice notes into a searchable study workspace. The Next.js frontend supports cited answers, document explanations, practice quizzes and an agent that proposes tasks or drafts. Behind it, FastAPI coordinates uploads, asynchronous ingestion, retrieval, streaming and billing on Google Cloud.

Problem

A chat interface is only useful if its answers can be checked. Documents need page references, voice notes need searchable transcripts, and the retrieval layer must keep each user inside their own library.

The backend also has to handle work that outlives a browser request: processing uploads, recovering from duplicate events and pausing an agent until a person approves a write. Those states belong in the database, with explicit transitions and failure paths.

Requirements

• Upload documents and audio directly to storage with scoped, temporary access
• Extract, transcribe and index uploads without blocking the browser
• Retrieve only the authenticated user's content and cite the passages used
• Support document explanations and quizzes alongside strict source-based answers
• Persist agent progress and require approval before creating tasks or saving drafts
• Apply plan-aware limits and grant paid access only after verified payment

Architecture

The Next.js client sends authenticated requests to FastAPI. The API creates a pending file record and signs a browser-to-Cloud-Storage upload. Eventarc invokes a private Cloud Run worker, which claims the file, extracts document text or transcribes audio, then chunks and embeds it with Vertex AI. Chunks, voice-note summaries and ready status commit together in Neon PostgreSQL. Questions combine pgvector similarity search with PostgreSQL full-text search before Gemini streams an answer. Agent steps and pending approvals are persisted separately; signed Cashfree webhooks update paid access.

{ "status": "skipped" }

Engineering decisions

PostgreSQL for both retrieval methods

pgvector and full-text search keep chunks, ownership and retrieval in one database. This avoids a second index to synchronise, while leaving room to evaluate a reranker if paraphrase retrieval remains weak.

Recover abandoned claims with a lease

Acknowledging every duplicate would strand a file after a worker crash. A live claim asks Eventarc to retry; an expired claim can be reclaimed. Attempts are bounded so a repeatedly crashing file eventually fails.

Derive identity outside the model

The API verifies Firebase identity and supplies the user ID to search and agent tools. The model cannot choose another user through tool arguments. File and chat lookups also check ownership.

Persist approval rather than waiting in memory

Human decisions can outlast an HTTP request. The database holds the pending write; an atomic claim prevents repeated approval clicks from executing the same pending action twice. This is not a blanket guarantee of recovery from every mid-tool crash.

Treat payment notifications as repeatable events

The backend verifies the webhook signature and fetches the order from Cashfree. Recording the payment ID and extending access happen in one transaction, so a redelivery cannot grant the same pass twice. A browser redirect never grants access.

Challenges

Results

Qualitative outcomes; benchmark measurements are not published.

• A live Next.js application backed by FastAPI and event-driven processing on Google Cloud.
• Documents and transcribed voice notes share searchable content with source references in streamed answers.
• Document explanations, practice quizzes and voice-note summaries support study workflows around the same library.
• Agent writes require an explicit decision, with pending approvals and tool history stored in PostgreSQL.
• Verified, deduplicated payment events control prepaid access; model usage is checked against each user's plan.

Lessons learned

• A useful AI product needs explicit ownership, state and failure handling around every model call.
• At-least-once events need both duplicate handling and a way to recover abandoned work.
• Source citations make answers inspectable; they do not prove every generated claim is correct.
• Measure retrieval separately from generation, and repeat evaluations before interpreting an answer score.
• Start with one database and a small explicit agent loop; add retrieval infrastructure when evaluation justifies it.

Technologies used

• Python
• FastAPI
• PostgreSQL / pgvector
• Next.js / TypeScript
• Cloud Run
• Eventarc
• Vertex AI (Gemini)
• Firebase Auth
• Cashfree