Organizing the archive
MethodologyHow the archive was checked, cleaned up and linked back to YouTube: verifying transcript quality, finding and removing duplicate streams, rebuilding the index, and a long investigation into why dozens of streams had no working YouTube link.
46 duplicate streams were found by their content and removed.
Out of 746, so only a handful had to be written again.
Out of 746 streams, each confirmed by matching duration.
Hidden by a tool bug, not deleted, and restored.
Telling Whisper from Parakeet by the timestamps
The whole archive had already been transcribed again once, from OpenAI Whisper to NVIDIA Parakeet, because Whisper looped on long streams and wrote the same short phrase hundreds of times over silence or music. That pass took the archive from about 780 streams and 53,471 passages to 785 streams and 59,280 passages of cleaner text, and the duplicate line rate fell from an average of about 27% to about 8%. This step spot-checked that pass. A file can be replaced on disk without the database knowing, so the stored tag cannot be trusted on its own.
The fingerprint
A transcript's cue timestamps carry a signature of the model that made it. Parakeet's cue times land on a multiple of 80 milliseconds about 93 to 97% of the time. Whisper's are only 20 ms aligned, so they land on a multiple of 80 ms just 21 to 35% of the time. Combined with the duplicate line percentage, the longest run of identical lines, and the 5-word text overlap against the current copy and a Whisper backup, this tells the two models apart from content alone.
What it found
Four DEAD SHPACE transcripts were checked. Three were Parakeet, and one (2019-10-23 (1)) was still the old Whisper version, so it was flagged for transcribing again. Two Shlug Éireann transcripts (11-03 and 11-04) were both confirmed Parakeet and matched their indexed copies exactly. They also overlap heavily: 11-03 looks like the same broadcast as 11-04 minus its first 4.7 minutes, which fed into the duplicate search below.
Comparing what was said, not what the file is called
Many streams had been uploaded or downloaded again from other channels under a different title and date, so file names could not be trusted to catch a duplicate. A scanner, backend/ingest/dupe_scan.py, compares the transcripts instead.
Five-word phrases
Each transcript becomes a set of overlapping 5-word phrases, each tagged with the time it first occurs.
Containment, not overlap
Two streams are a candidate pair when they share a large part of their phrases, scored as shared phrases divided by the smaller stream's own count. A duplicate is often a trimmed or partial copy of a longer original, which a symmetric overlap score would miss.
Stock phrases ignored
Phrases that turn up in an unusually large share of all streams, such as “thank you very much for the”, are left out, so two unrelated streams never link just because the host repeats himself.
Pointing to the original
Each stream is also matched against the YouTube playlist export. When exactly one side of a pair has a real, working video link, it is flagged as the likely original and sorts first.
Read-only
The scanner writes a CSV and an HTML review page for a person to act on. It never deletes anything itself.
Result
46 duplicate streams were identified and removed by hand, taking the archive from 785 streams down to 746.
Rebuilding the index without losing hours of work
A full rebuild throws the database away and derives everything again from the transcripts on disk. But the database also holds about 780 AI-written stream summaries, each of which costs real generation time. Rebuilding naively would discard every one of them.
1. Back up
The database from before the rebuild was backed up.
2. Rebuild without summaries
Every transcript was chunked and embedded again.
3. Copy summaries back
A saved summary was copied back for every stream whose transcript file was byte for byte the one the summary was written against, matched by file size and modification time. A transcript that really had changed correctly lost its now out of date summary, instead of keeping one that described different text.
4. Write only what is missing
The summariser ran only on the few streams left without one.
Result
741 of 746 summaries were kept. The reindex also picked up the new Plex library tags and playlist entries that had been prepared, and confirmed that all 746 streams matched an entry in the Plex catalogue.
No link does not mean deleted
After the rebuild, 93 of the 746 streams had no working YouTube link. The first instinct was to read that as 93 deleted streams. That was wrong for roughly a quarter of them, and finding out why took most of the effort.
How links are matched
A playlist export, produced by a separate yt-dlp based lister tool, is compared against each transcript's title and date. The pairing is then confirmed by checking the video's duration against the transcript's length to within a few seconds. That duration check is what makes a match trustworthy rather than a guess.
A yt-dlp bug that hid live videos
Running the matcher again on a fresh playlist scan seemed to lose 24 working links. The cause was YouTube's SABR streaming rollout hitting a tracked upstream bug in yt-dlp (yt-dlp/yt-dlp#12482), where the tool reports zero playable formats for a video that is completely normal and public. The video then gets no entry in the playlist CSV at all, so the matcher has nothing to compare and the stream looks exactly like a deleted one. Running yt-dlp -v on the affected videos showed the failure: the web_embedded and tv_downgraded clients both reported the video unplayable, then the web client was blocked by the SABR rollout. Several of these videos were opened in a browser and played normally. The 24 links were restored from a backup taken before the rebuild.
Checking dates against Plex
Several high confidence matches sat exactly one day away from the date in the transcript's own filename. Rather than guess which date was right, the Plex library export, an independent catalogue of the same media files, was used as a second source of truth. The transcript 2019-08-29 - Resident Carmody 4_ Willie Harold matched nothing on the 29th, but Plex listed the real file as 2019-08-30. The filename date was simply wrong, so the fix was to rename the transcript instead of forcing a link onto the wrong day's video. All 746 transcript filenames were cross-checked, every mismatch was corrected, and none now disagree with Plex on date.
Verifying by hand
Every stream still unlinked after the automated steps was checked directly in a browser rather than trusted to tool output. It was the slowest part, and it caught mistakes in both directions. A real copyright block names a specific claimant on the video page itself. A SABR failure shows nothing wrong when the page is opened directly, because nothing is.
Absence from the playlist scan does not mean absence from YouTube. It has to be checked directly.
Confirmed blocked by a real copyright claim, checked one by one
| Stream | Blocked by |
|---|---|
| 2019-03-12 - Shenmuesday 12 March | RTÉ |
| 2019-05-30 (2) - Test Drivetime | Fédération Française de Tennis |
| 2019-07-23 (2) - Resident Evil 4 - HERB KENNEDY | Fédération Française de Tennis |
| 2019-07-31 (1) and (2) - RE4 NEVER STOP SPEEDRUNNING | Fédération Française de Tennis |
| 2019-08-02 - Die Hard 2, Part 1 and 2 | Fédération Française de Tennis |
| 2019-10-16 - SPEEDRUN ... APPLAUDING THERE | Fédération Française de Tennis |
| 2019-11-16 - Hitman Part 1 | Fédération Française de Tennis |
| 2020-08-23 - Shlug Lingus, Leaving Limerick 2 | SME |
| 2020-08-24 to 26 - Aer Shlungus (4 streams) | SME |
| 2025-03-01 - New Vegas | Grammys |
Looked unavailable to yt-dlp, but played fine when opened directly
| Stream | yt-dlp said | What was true |
|---|---|---|
| 2019-05-19 - Eastenders The Video Game | Video unavailable | Plays at full length, matching the transcript exactly |
| 2019-12-04 - HITMAN.ie | Video unavailable | Plays at full length |
| 2020-06-15 - BLACK LIVESTREAM MESA | Video unavailable | Plays at full length |
| 2026-02-15 - FALLOUT 3, Part 11 | Video unavailable | Plays, but the audio is muted by a real copyright claim |
The test for “genuinely deleted”
Every stream in this archive is uploaded as Unlisted, and the playlist scan is the only way any of them can be discovered. They do not appear in search, on the channel page or anywhere else. That gives a defensible test:
If a stream does not appear anywhere in a playlist scan that already accounts for the SABR bug, and its transcript content does not match any other transcript already in the archive under a different name or date, then by elimination its video no longer exists to be found.
Why the second condition matters
Several early “missing” streams turned out to be duplicates of a stream already archived correctly under a different title. The duplicate scanner was run again with a much lower containment floor, down to 5% where a genuine duplicate shows 60 to 95%, to rule this out, and it consistently found nothing for the streams in question.
The caveat
The test is only as good as the playlist scan behind it. A scan run before the SABR bug was understood gave false “deleted” verdicts for videos that were fine, which is exactly what happened at the start. The scan behind the final numbers reflects an all night effort to update the extractor and run a full scan again, so the false negative rate should be much lower, but it is not proven to be zero.
Library problems found along the way
Stale deleted tags
Plex tags some streams as Deleted or Missing Stream independently of the YouTube link. Several of these tags were stale, because the video actually plays fine, while others were confirmed correct by the same browser checking. Tags were synced in both directions as each case was settled.
An unresolved duplicate pair
2024-12-16 - GLITCHED Clean Streets (Dead Air Trimmed) is a hand edited cut of the same date's original, with intermissions removed. The plan is to keep the trimmed cut under the original's filename and drop the untrimmed original. This is in progress.
Unrelated files grouped together
Plex was silently grouping unrelated files as multiple versions of one item, such as two different streams that were both titled “Resident Evil 2” on different dates. The cause was Plex's local scanner grouping files that share a title substring and a similar runtime, even with the Personal Media agent and Other Videos library type that should not do this. The fix is giving every stream file its own subfolder, because Plex treats folder boundaries as a far stronger signal than filename similarity. The tool for it is written but has not been run on the live library, pending a full backup.
What was built
| Tool | Purpose |
|---|---|
| backend/ingest/dupe_scan.py | Finds duplicate content across the archive by phrase overlap, and writes a CSV and an HTML review page. |
| Fingerprint and provenance script | Tells Whisper from Parakeet transcripts using cue timestamp alignment, duplicate line rate and text overlap. |
| reorganize_streams_gui.py | A small Tkinter tool that previews, then after confirmation moves each file into its own folder. Safe to run again. This is the Plex grouping fix. |
| Unlinked stream Excel reports | A breakdown of every unlinked stream by category, with candidate matches and confidence, for manual review. |
Where things ended up
| Category | Count |
|---|---|
| Streams in the archive | 746 (down from 785 before removing duplicates) |
| Linked to a verified YouTube video | 660 |
| Confirmed blocked by a real copyright claim, checked in a browser | About 12 |
| Tagged deleted in Plex, consistent with having no link | 39 |
| Assessed as deleted by elimination, pending a final scan check | About 40 |
| Still open for manual investigation | A small remainder |
Still open
One duplicate pair (GLITCHED Clean Streets) is waiting for the original file to be deleted. The Plex folder fix is written but not yet run on the live library. A handful of streams with no strong automated candidate would benefit from a manual look at the channel itself.
No link is not the same as deleted
Treating the two as interchangeable would have wrongly written off dozens of streams that are actually fine. Roughly a quarter of the streams that started this process unlinked turned out to be tooling or metadata problems, such as a buggy extractor, a wrongly dated filename or a stale library tag, rather than anything actually missing. Getting to a trustworthy list meant checking against an independent source (Plex's own export), reproducing failures with verbose tool output instead of trusting a one line error, and opening a browser to look at the actual video for anything the automated tools disagreed about.