Multilingual Consent Verification

Prototype · Tata Consultancy Services, prototype · Engineer, design through delivery

Customers read consent statements aloud in Indian languages. Someone has to prove they actually said the right words, at scale, without a human listening to every recording.

My contribution: Designed the verification pipeline end to end, built the alignment and scoring engine, integrated GPU transcription and the LLM fallback, and exposed it as a REST API.
Key challenge: Code-mixed speech. Speakers switch between English and their native language mid-sentence. Transliterating everything into one script before alignment let English loanwords match regardless of how the model wrote them.
Outcome: Deterministic, auditable verdicts for the common case, with the LLM reserved for genuinely ambiguous words instead of every request.

On this page

Overview

A verification service that answers one question with evidence: did this person say the consent statement they were shown? It accepts a recording and the expected text in an Indian language, and returns a verdict, a similarity score, the exact words that did not match, and an explanation a non-technical reviewer or customer can understand.

Problem

Speech-to-text alone does not answer the question. Transcripts of Indian languages come back in mixed scripts: a Telugu sentence may contain Latin-script English loanwords, or the model may emit a romanised form of a native word. A naive string comparison marks a perfectly valid reading as a failure.

An LLM-only judge can approve a recording that skipped a clause and does not provide a reproducible word-level comparison. Reviewers need to inspect the exact mismatch behind a verdict.

Requirements

Support multiple Indian languages and mixed-script speech
Produce a verdict that is reproducible for the same input
Point to the exact words that were missed, added or changed
Keep per-request cost and latency predictable under load
Explain results in language a customer can understand
Run entirely server-side; recordings never leave the service boundary

Architecture

A staged pipeline where every step is deterministic except one. Transcription runs on GPU with Faster Whisper. Both the transcript and the expected text are transliterated into a common script and normalised, so comparison happens in one representation. An alignment step pairs spoken words with expected words, a scorer computes similarity and classifies each gap, and only spans the scorer cannot confidently classify are sent to an LLM. A final step turns the structured result into a readable explanation.

Architecture sketch. Solid arrows are the main path, dashed arrows run only when needed, and the outlined box marks the key engineering decision.

Audio inUpload + expected consent text
TranscribeFaster Whisper on GPU
Transliterate + normaliseUnify mixed scripts before comparison
AlignDeterministic word alignment
ScoreSimilarity + mismatch detection
ResolveLLM judges the ambiguous spanOnly in the uncertain score band
ExplainPlain-language verdict

{
  "verdict": "mismatch",
  "similarity": 0.91,
  "language": "te",
  "issues": [
    { "type": "missing", "expected": "not", "position": 14, "critical": true }
  ],
  "llmResolved": 0,
  "explanation": "The required negation was not heard. Without it, the statement means the opposite."
}

Illustrative Telugu consent mismatch. The missing word and explanation are shown in English translation; the score and position are examples, not production data.

Engineering decisions

Rule-based core, LLM at the edges: Auditable, cheap and fast for the majority of requests; the model adds judgement only where rules are genuinely unsure.
Compare in a single normalised script: Mixed-script output is the norm for Indian-language ASR. Normalising both sides removes an entire class of false rejections.
Faster Whisper on GPU: CTranslate2-based inference gives Whisper-level accuracy with lower latency and memory, which matters when recordings arrive in bursts.
Return evidence, not just a boolean: Reviewers and customers trust a result they can see. Word-level diffs and explanations turn disputes into quick checks.

Challenges

Code-mixed speech

Speakers switch between English and their native language mid-sentence. Transliterating everything into one script before alignment let English loanwords match regardless of how the model wrote them.

Defining "close enough"

Too strict and valid readings fail; too loose and skipped clauses pass. Thresholds were tuned against labelled recordings, with critical words (negations, amounts, names) held to a stricter standard.

Keeping the LLM honest

Constraining its input to a single ambiguous span and asking for a structured answer made its output predictable and testable.

Results

Qualitative outcomes; benchmark measurements are not published.

Verification verdicts that are reproducible and explainable word by word.
In testing, readings that differ only in script or spelling pass, where naive transcript comparison rejected them.
LLM calls are restricted to ambiguous spans.
Readable explanations that let non-engineers understand and act on a failed verification.

Lessons learned

Most AI accuracy problems here were data representation problems. Normalisation beat model tuning.
Put the model where judgement is needed, and surround it with code you can test.
An explanation is part of the product, not a debugging aid.

Technical approach and performance notes

Technical approach

Normalise before you compare

Transliteration and script normalisation put mixed-script transcripts into a common representation before alignment. This separates representation differences from missing or substituted words.

Deterministic alignment

A sequence alignment over normalised tokens maps each expected word to what was spoken, producing explicit match, substitution, insertion and deletion operations. That gives a verdict that can be replayed and explained word by word.

Similarity with thresholds, not a single number

Token-level similarity tolerates pronunciation and transcription noise, while statement-level rules catch what matters, like a skipped negation or a missing clause. Scores fall into accept, reject and uncertain bands.

LLM as a tie-breaker

Only the uncertain band reaches the LLM, with a narrow prompt: here is the expected phrase, here is what was heard, are they the same in meaning and wording? The model never sees the whole decision, which keeps cost bounded and behaviour easier to test.

Performance and scalability

GPU transcription is the dominant cost, so the rest of the pipeline is kept CPU-light and linear in transcript length.
LLM calls are proportional to ambiguity, not traffic; clean recordings never touch the model.
Stateless API workers scale horizontally; GPU workers scale independently of the web tier.
Model weights are loaded once per worker and reused across requests to avoid cold-start penalties.

Technologies used

Python
Faster Whisper
LLMs
GPU inference
REST APIs