Gnomad Harvester

Why this exists

Agents are useful when they can read a talk, not when they guess from the title. I got tired of pasting YouTube links into a chat and hoping the model had watched the video.

Gnomad Harvester pulls official or auto captions on the machine. Default path is $0. It does not call the YouTube Data API. It does not touch Google Cloud.

Product card: /apps/gnomad-harvester

Architecture

playlist or watch URL
        │
        ▼
 yt-dlp --flat-playlist     (ids + titles)
        │
        ▼
 youtube-transcript-api     (free, public timedtext)
        │ fail
        ▼
 yt-dlp --write-auto-subs   (still local, still $0)
        │
        ▼
 transcripts/ + optional Ollama 3-bullet summary
        │
        ▼
 agent KB markdown

Optional paid fallback: youtube-transcript.io. Third-party credits — not GCP. Skip it unless captions are blocked and you explicitly want to pay.

Recreate it in an afternoon

1. Install the two free tools

# Fedora / Nobara / most Linux
sudo dnf install yt-dlp   # or: pip install yt-dlp

python3 -m venv .venv
source .venv/bin/activate
pip install youtube-transcript-api

Windows/macOS: install yt-dlp from their release page and put it on PATH.

2. One video, no framework

from youtube_transcript_api import YouTubeTranscriptApi

video_id = "dQw4w9WgXcQ"
ft = YouTubeTranscriptApi().fetch(video_id, languages=("en", "en-US", "en-GB"))
text = "\n".join(s.text.strip() for s in ft.snippets if s.text.strip())
print(text[:800])

If YouTube blocks that (bot check, missing track), fall back to auto-subs without downloading the video:

yt-dlp --skip-download --write-auto-subs --sub-langs "en,en-US,en-orig" \
  -o "./%(id)s" "https://www.youtube.com/watch?v=VIDEO_ID"

Convert .vtt to plain text (strip cue numbers and timestamps). That file is what you hand an agent.

3. A playlist

yt-dlp --flat-playlist --dump-single-json \
  "https://www.youtube.com/playlist?list=PLAYLIST_ID" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('entries') or []))"

Loop each id, run step 2, write transcripts/<topic>/<title>__<id>.txt.

4. Optional local summary (still $0)

If Ollama is running:

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Three factual bullets from this transcript:\n\n...",
  "stream": false
}'

Harvester’s --ollama flag does exactly that: an “Alpha Summary” of three bullets, plus a richer KB markdown file for agents.

What this is not

• Not OCR. On-screen slides with no captions need a different pipeline (frames + vision / Whisper on audio).
• Not YouTube Data API v3. That is a billed Google product. Harvester never uses it.
• Not a hosted SaaS. You run it where yt-dlp can reach YouTube.

Studio CLI (ours)

We wrap the pattern as python3 -m gnomad_harvester PLAYLIST_URL --category <slug>. Flags worth copying: --limit / --offset for batches, --incremental so you do not refetch, --json-summary for agents.

Source lives in the Gnomad ORGANIZATION workspace (02_ai_engineering/tools/yt-summarizer/). This page is the public recipe so you can rebuild it without that tree.

Cost table

Path	Cost
Captions via transcript API + yt-dlp	$0
Ollama Alpha Summary	$0 (your GPU)
youtube-transcript.io	paid credits (optional)
YouTube Data API / GCP	not used

Next

Wire the output into whatever you already use for agent memory — we drop markdown next to the transcript and let the swarm read it. If you publish a clone, keep the cost line in the README: captions first, cloud APIs never by default.