Francis Higgins Archive News

Organizing the archive

Methodology

How the archive was checked, cleaned up and linked back to YouTube: verifying transcript quality, finding and removing duplicate streams, rebuilding the index, and a long investigation into why dozens of streams had no working YouTube link.

Streams in the archive
785→746

46 duplicate streams were found by their content and removed.

Summaries kept
741

Out of 746, so only a handful had to be written again.

Linked to a verified video
660

Out of 746 streams, each confirmed by matching duration.

Links wrongly lost
24

Hidden by a tool bug, not deleted, and restored.

Verifying

Telling Whisper from Parakeet by the timestamps

The whole archive had already been transcribed again once, from OpenAI Whisper to NVIDIA Parakeet, because Whisper looped on long streams and wrote the same short phrase hundreds of times over silence or music. That pass took the archive from about 780 streams and 53,471 passages to 785 streams and 59,280 passages of cleaner text, and the duplicate line rate fell from an average of about 27% to about 8%. This step spot-checked that pass. A file can be replaced on disk without the database knowing, so the stored tag cannot be trusted on its own.

The fingerprint

A transcript's cue timestamps carry a signature of the model that made it. Parakeet's cue times land on a multiple of 80 milliseconds about 93 to 97% of the time. Whisper's are only 20 ms aligned, so they land on a multiple of 80 ms just 21 to 35% of the time. Combined with the duplicate line percentage, the longest run of identical lines, and the 5-word text overlap against the current copy and a Whisper backup, this tells the two models apart from content alone.

What it found

Four DEAD SHPACE transcripts were checked. Three were Parakeet, and one (2019-10-23 (1)) was still the old Whisper version, so it was flagged for transcribing again. Two Shlug Éireann transcripts (11-03 and 11-04) were both confirmed Parakeet and matched their indexed copies exactly. They also overlap heavily: 11-03 looks like the same broadcast as 11-04 minus its first 4.7 minutes, which fed into the duplicate search below.

Duplicates

Comparing what was said, not what the file is called

Many streams had been uploaded or downloaded again from other channels under a different title and date, so file names could not be trusted to catch a duplicate. A scanner, backend/ingest/dupe_scan.py, compares the transcripts instead.

Five-word phrases

Each transcript becomes a set of overlapping 5-word phrases, each tagged with the time it first occurs.

Containment, not overlap

Two streams are a candidate pair when they share a large part of their phrases, scored as shared phrases divided by the smaller stream's own count. A duplicate is often a trimmed or partial copy of a longer original, which a symmetric overlap score would miss.

Stock phrases ignored

Phrases that turn up in an unusually large share of all streams, such as “thank you very much for the”, are left out, so two unrelated streams never link just because the host repeats himself.

Pointing to the original

Each stream is also matched against the YouTube playlist export. When exactly one side of a pair has a real, working video link, it is flagged as the likely original and sorts first.

Read-only

The scanner writes a CSV and an HTML review page for a person to act on. It never deletes anything itself.

Result

46 duplicate streams were identified and removed by hand, taking the archive from 785 streams down to 746.

Rebuilding

Rebuilding the index without losing hours of work

A full rebuild throws the database away and derives everything again from the transcripts on disk. But the database also holds about 780 AI-written stream summaries, each of which costs real generation time. Rebuilding naively would discard every one of them.

1. Back up

The database from before the rebuild was backed up.

2. Rebuild without summaries

Every transcript was chunked and embedded again.

3. Copy summaries back

A saved summary was copied back for every stream whose transcript file was byte for byte the one the summary was written against, matched by file size and modification time. A transcript that really had changed correctly lost its now out of date summary, instead of keeping one that described different text.

4. Write only what is missing

The summariser ran only on the few streams left without one.

Result

741 of 746 summaries were kept. The reindex also picked up the new Plex library tags and playlist entries that had been prepared, and confirmed that all 746 streams matched an entry in the Plex catalogue.

YouTube links

No link does not mean deleted

After the rebuild, 93 of the 746 streams had no working YouTube link. The first instinct was to read that as 93 deleted streams. That was wrong for roughly a quarter of them, and finding out why took most of the effort.

How links are matched

A playlist export, produced by a separate yt-dlp based lister tool, is compared against each transcript's title and date. The pairing is then confirmed by checking the video's duration against the transcript's length to within a few seconds. That duration check is what makes a match trustworthy rather than a guess.

A yt-dlp bug that hid live videos

Running the matcher again on a fresh playlist scan seemed to lose 24 working links. The cause was YouTube's SABR streaming rollout hitting a tracked upstream bug in yt-dlp (yt-dlp/yt-dlp#12482), where the tool reports zero playable formats for a video that is completely normal and public. The video then gets no entry in the playlist CSV at all, so the matcher has nothing to compare and the stream looks exactly like a deleted one. Running yt-dlp -v on the affected videos showed the failure: the web_embedded and tv_downgraded clients both reported the video unplayable, then the web client was blocked by the SABR rollout. Several of these videos were opened in a browser and played normally. The 24 links were restored from a backup taken before the rebuild.

Checking dates against Plex

Several high confidence matches sat exactly one day away from the date in the transcript's own filename. Rather than guess which date was right, the Plex library export, an independent catalogue of the same media files, was used as a second source of truth. The transcript 2019-08-29 - Resident Carmody 4_ Willie Harold matched nothing on the 29th, but Plex listed the real file as 2019-08-30. The filename date was simply wrong, so the fix was to rename the transcript instead of forcing a link onto the wrong day's video. All 746 transcript filenames were cross-checked, every mismatch was corrected, and none now disagree with Plex on date.

Verifying by hand

Every stream still unlinked after the automated steps was checked directly in a browser rather than trusted to tool output. It was the slowest part, and it caught mistakes in both directions. A real copyright block names a specific claimant on the video page itself. A SABR failure shows nothing wrong when the page is opened directly, because nothing is.

Absence from the playlist scan does not mean absence from YouTube. It has to be checked directly.

Confirmed blocked by a real copyright claim, checked one by one

StreamBlocked by
2019-03-12 - Shenmuesday 12 MarchRTÉ
2019-05-30 (2) - Test DrivetimeFédération Française de Tennis
2019-07-23 (2) - Resident Evil 4 - HERB KENNEDYFédération Française de Tennis
2019-07-31 (1) and (2) - RE4 NEVER STOP SPEEDRUNNINGFédération Française de Tennis
2019-08-02 - Die Hard 2, Part 1 and 2Fédération Française de Tennis
2019-10-16 - SPEEDRUN ... APPLAUDING THEREFédération Française de Tennis
2019-11-16 - Hitman Part 1Fédération Française de Tennis
2020-08-23 - Shlug Lingus, Leaving Limerick 2SME
2020-08-24 to 26 - Aer Shlungus (4 streams)SME
2025-03-01 - New VegasGrammys

Looked unavailable to yt-dlp, but played fine when opened directly

Streamyt-dlp saidWhat was true
2019-05-19 - Eastenders The Video GameVideo unavailablePlays at full length, matching the transcript exactly
2019-12-04 - HITMAN.ieVideo unavailablePlays at full length
2020-06-15 - BLACK LIVESTREAM MESAVideo unavailablePlays at full length
2026-02-15 - FALLOUT 3, Part 11Video unavailablePlays, but the audio is muted by a real copyright claim

The test for “genuinely deleted”

Every stream in this archive is uploaded as Unlisted, and the playlist scan is the only way any of them can be discovered. They do not appear in search, on the channel page or anywhere else. That gives a defensible test:

If a stream does not appear anywhere in a playlist scan that already accounts for the SABR bug, and its transcript content does not match any other transcript already in the archive under a different name or date, then by elimination its video no longer exists to be found.

Why the second condition matters

Several early “missing” streams turned out to be duplicates of a stream already archived correctly under a different title. The duplicate scanner was run again with a much lower containment floor, down to 5% where a genuine duplicate shows 60 to 95%, to rule this out, and it consistently found nothing for the streams in question.

The caveat

The test is only as good as the playlist scan behind it. A scan run before the SABR bug was understood gave false “deleted” verdicts for videos that were fine, which is exactly what happened at the start. The scan behind the final numbers reflects an all night effort to update the extractor and run a full scan again, so the false negative rate should be much lower, but it is not proven to be zero.

Plex

Library problems found along the way

Stale deleted tags

Plex tags some streams as Deleted or Missing Stream independently of the YouTube link. Several of these tags were stale, because the video actually plays fine, while others were confirmed correct by the same browser checking. Tags were synced in both directions as each case was settled.

An unresolved duplicate pair

2024-12-16 - GLITCHED Clean Streets (Dead Air Trimmed) is a hand edited cut of the same date's original, with intermissions removed. The plan is to keep the trimmed cut under the original's filename and drop the untrimmed original. This is in progress.

Unrelated files grouped together

Plex was silently grouping unrelated files as multiple versions of one item, such as two different streams that were both titled “Resident Evil 2” on different dates. The cause was Plex's local scanner grouping files that share a title substring and a similar runtime, even with the Personal Media agent and Other Videos library type that should not do this. The fix is giving every stream file its own subfolder, because Plex treats folder boundaries as a far stronger signal than filename similarity. The tool for it is written but has not been run on the live library, pending a full backup.

Tools

What was built

ToolPurpose
backend/ingest/dupe_scan.pyFinds duplicate content across the archive by phrase overlap, and writes a CSV and an HTML review page.
Fingerprint and provenance scriptTells Whisper from Parakeet transcripts using cue timestamp alignment, duplicate line rate and text overlap.
reorganize_streams_gui.pyA small Tkinter tool that previews, then after confirmation moves each file into its own folder. Safe to run again. This is the Plex grouping fix.
Unlinked stream Excel reportsA breakdown of every unlinked stream by category, with candidate matches and confidence, for manual review.
Final state

Where things ended up

CategoryCount
Streams in the archive746 (down from 785 before removing duplicates)
Linked to a verified YouTube video660
Confirmed blocked by a real copyright claim, checked in a browserAbout 12
Tagged deleted in Plex, consistent with having no link39
Assessed as deleted by elimination, pending a final scan checkAbout 40
Still open for manual investigationA small remainder

Still open

One duplicate pair (GLITCHED Clean Streets) is waiting for the original file to be deleted. The Plex folder fix is written but not yet run on the live library. A handful of streams with no strong automated candidate would benefit from a manual look at the channel itself.

Takeaway

No link is not the same as deleted

Treating the two as interchangeable would have wrongly written off dozens of streams that are actually fine. Roughly a quarter of the streams that started this process unlinked turned out to be tooling or metadata problems, such as a buggy extractor, a wrongly dated filename or a stale library tag, rather than anything actually missing. Getting to a trustworthy list meant checking against an independent source (Plex's own export), reproducing failures with verbose tool output instead of trusting a one line error, and opening a browser to look at the actual video for anything the automated tools disagreed about.