1. Introduction

The platform stores thousands of video assets but none have machine-generated transcripts or captions today. This document describes the end-to-end design for automatically generating transcripts and captions, allowing human review, and optionally translating into multiple languages.

2. Problem Statement

Video content has no transcripts or captions today, making it:

• Inaccessible to hearing-impaired learners

• Unusable in low-bandwidth or noisy environments

• Spoken content inside videos cannot be extracted, summarized, or reused; making it impossible to auto-generate descriptions, chapter markers, or study notes from video at scale.

Producing them manually at scale is not feasible.

3. Out of Scope

• Real-time transcription

• Speaker diarization

• YouTube content

• Auto-translation on transcription completion

4. Object Model

Each language's transcript and caption are stored as a linked Asset node on the Content. One asset per language — each has its own lifecycle.

Content ──"transcripts"──► [ Asset(en), Asset(hi), Asset(es) ]

Asset fields table

Storage layout

content/<contentId>/video.mp4          ← source video (existing)
content/<contentId>/assets/<assetId>/transcript.json       ← transcript segments + metadata
content/<contentId>/assets/<assetId>/captions.vtt          ← WebVTT for video player

Transcript JSON Schema

{
  "source": "https://blob.../video.mp4",
  "generatedBy": "faster-whisper:large-v3-turbo",
  "generatedOn": "2026-07-01T09:00:00Z",
  "duration": 378.32,
  "language": "en",
  "segments": [
    { "id": 0, "start": 10.67, "end": 24.67, "text": "Hello children..." }
  ]
}

5. End-to-End Workflow

1. Upload & Trigger: Content Creator uploads video → upload API emits generate-transcript Kafka event

2. Transcribe: Processing job picks event, downloads video, runs transcription, uploads transcript.json + captions.vtt, creates Asset node (status Draft), links to Content

3. Review & Edit: Content Creator opens asset, reads transcript, optionally edits, saves → PATCH /content/v4/transcript/update/:id

4. Approve / Reject: Content Creator approves → asset moves to Live + optional translation triggered. Reject → back to Draft

5. Content Review Gate: Sending content for review blocked unless default-language asset is Live

6. Translation: On approval with generateTranslation: true → translation job creates one Asset per language → auto-approved to Live

6. APIs

Endpoints table

6.1 Read Path

When a content node is read today, related assets return only identifier, name, objectType, relation, description, status. The video player needs artifactUrl, captionsUrl, language, languageCode, isSource to render captions without extra per-asset API calls.

Fix: Extend NodeUtil.serialize() in ontology-engine to return full fields for related assets where primaryCategory = Transcript. Same content read endpoint, richer response.

"transcripts": [
  {
    "identifier": "asset_123",
    "language": ["English"],
    "languageCode": "en",
    "isSource": true,
    "status": "Live",
    "artifactUrl": "https://blob.../transcript.json",
    "captionsUrl": "https://blob.../captions.vtt"
  }
]

6.2 Kafka Topics

POST cases

7. Editorial & Review Flow

Only the source language transcript (isSource=true) can be edited. Machine-translated entries are read-only.

PATCH endpoint flow:

1. Content Creator edits segments in UI, saves

2. PATCH /content/v4/transcript/update/:id called with updated segments

3. System validates isSource=true for that language - returns ERR_CANNOT_EDIT_TRANSLATION if not the source language

4. New transcript.json + captions.vtt uploaded to blob

5. Content node updated with new URLs

6. Updated transcript.json and captions.vtt are saved. No Kafka event is emitted on edit.

8. Implementation Options

Option 1: PyFlink Job

A PyFlink ProcessFunction consumes Kafka events directly and runs faster-whisper transcription inline. Translation via litellm runs in a ThreadPoolExecutor inside the same job.

Pros

• No new technology introduced as it fits directly into existing platform architecture

• Kafka offset committed only after successful checkpoint; event never lost or double-processed if pod crashes mid-transcription

• Backpressure built-in: if transcription slow, Kafka consumer slows automatically, no manual rate limiting needed

Cons

• Transcribing a 1hr video blocks a task slot for 20–45 min, parallelism must be sized accordingly

• Each parallel task loads its own model (~2GB each) → 5 slots = ~10GB RAM

• Checkpoint timeout must be manually tuned to exceed worst-case processing time

Tuning required

env.enable_checkpointing(3600000)          # 1hr — longer than max job
env.get_checkpoint_config().set_checkpoint_timeout(3600000)
# transcription operator parallelism = 1 (one model instance)
# upload/translation operator parallelism = N

Option 2 — Standalone Python Service (+ optional KEDA)

A standalone FastAPI service acts as a Kafka consumer. It polls <env>.media.transcription.request, processes events, and publishes results back to Kafka. KEDA can optionally scale pod count based on Kafka consumer group lag.

Pros

• Each pod runs one model instance; memory is predictable and isolated, no cross-task interference unlike Flink's shared slot model

• Pod crash does not affect other workers; Kubernetes restarts it, Kafka offset was never committed, event is automatically retried with zero custom retry logic

• Same Docker image runs as both HTTP server and Kafka worker; one codebase handles direct API calls and async batch processing without duplication

Cons

• Retry, DLQ, and backpressure must all be hand-rolled; in Flink these are native primitives; here they are custom code that needs to be written, tested, and maintained

• No built-in flow control - if events arrive faster than pods process, consumer lag grows unboundedly until KEDA kicks in; without KEDA there is no automatic relief valve

• No equivalent of Flink Web UI - operator-level throughput, lag, and failure metrics require manual Prometheus instrumentation

KEDA ScaledObject (optional)

triggers:
  - type: kafka
    metadata:
      topic: media.transcription.request
      consumerGroup: caption-generator
      lagThreshold: "5"     # scale up when lag > 5
      offsetResetPolicy: latest

Option 3 — Cloud Transcription API (Groq / OpenAI / Azure)

Instead of running faster-whisper locally, send audio directly to a cloud provider. Already supported via the pluggable TranscriptionProvider interface — config change only, no code change.

Provider comparison

1hr video on Groq = ~20 seconds, costs ~₹9

Pros

• Processing time drops from 20–45 min (CPU) to ~20 seconds (Groq) → same whisper-large-v3-turbo model quality, just on their hardware; eliminates slot blocking and checkpoint tuning entirely

• Zero model management; no GPU provisioning, no ctranslate2 packaging, no memory sizing; infrastructure cost is purely per API call, no idle cost

• Switching providers is a single config line change, no code change required due to pluggable TranscriptionProvider interface already in place

Cons

• Audio data leaves our infrastructure; for government or education content this may violate data residency or privacy requirements

• Cost is unpredictable at scale; $0.111/hr sounds cheap, but 10,000 videos at 1hr average = $1,110 in a single batch run with no cost ceiling unless explicit throttling is added

• Rate limits can stall bulk processing; Groq and OpenAI impose per-minute and per-day audio second limits; a large content upload event can hit these mid-batch and halt processing silently

9. Transcription & Translation Processing

Transcription

• Model: faster-whisper large-v3-turbo, int8 quantization, runs on CPU (GPU optional)

• faster-whisper internally chunks audio using VAD (Voice Activity Detection) — no manual chunking needed

• VAD filter enabled; skips silence, reduces processing time 20-40%

• Audio extracted from video before processing (ffmpeg) — reduces download size 10-20x

• Processing time estimates table:

• Checkpoint interval must be set longer than max processing time when using PyFlink

Translation

• Translation via litellm -> supports Claude, OpenAI, Ollama, and any litellm-compatible model

• Segments sent in batches of 80 (~5 min of audio per batch) to stay within LLM context windows

• Each batch includes last 2 segments of previous batch as context (not translated, for continuity)

• All requested languages translated in parallel (ThreadPoolExecutor)

• Failed language goes to DLQ which does not block other languages

• Failed language will have no Asset node created. DLQ consumer can retry by re-emitting the translation event for that language only. Source asset status is unaffected.

• Re-translation on re-approval; if translation assets already exist for a language, they are overwritten in-place with new files. Asset identifier and relation to content are preserved.

• generatedBy field updated to include translation model, e.g. "faster-whisper:large-v3-turbo + claude-3-5-sonnet"

10. Plugin Architecture

Both transcription and translation are pluggable. Adopters implement the interface and configure the alias.

Transcription providers implement a common interface that accepts an artifact URL and returns a list of segments with id, start, end, and text. Translation providers implement an interface that accepts segments, source language, and target language and returns translated segments. Both are resolved by alias (faster_whisper, groq, litellm) or by fully qualified class name for custom implementations.

# config.yaml
transcription:
  provider: "faster_whisper"     # or "groq", "openai", "azure", or FQCN

translation:
  provider: "litellm"
  model: "claude-3-5-sonnet"     # or any litellm-supported model

11. Prerequisites & Required Platform Changes

These changes are required before the feature can be implemented. Without them, schema validation will fail on day one.