Methodology

How this brief is built.

1. The corpus

BotwaveBomba starts with a working corpus of English-language headlines gathered from hundreds of outlets across the Western, non-aligned, and adversarial blocs. The corpus is refreshed daily. We do not claim the corpus is complete; we claim it is diverse enough to expose structural coverage gaps in the U.S.-partisan press.

2. Story clustering

Headlines are clustered into stories using:

• Jaccard similarity on normalized headline tokens,
• a bigram phrase gate that forces a shared named-entity or phrase before two headlines can cluster,
• union-find with a max-size cap so one mega-topic does not swallow the day.

3. The scoring function

Each story gets a score from three inputs:

score = non_west_pct × entropy × log₂(source_count + 1)

• non_west_pct — share of sources that are non-aligned or adversarial (not Western). Higher means the story is less covered by the U.S.-partisan axis.
• entropy — Shannon entropy across the three blocs. A story covered evenly by all three blocs scores higher than one dominated by a single bloc.
• log₂(source_count + 1) — more sources raise the score, but with diminishing returns.

4. From score to brief

The daily brief picks the top 3 scored stories, deduplicated by headline signature so you do not get three versions of the same event. Each pick links to its top 5 primary sources, shows the bloc/country spread, and includes a lede excerpt when available.

5. Body excerpts

For the recent-3-day English slice we fetch the article URL, run Mozilla Readability over the HTML, and extract the first ≤600 characters of body text. If Readability cannot parse the page or the body is too short, we fall back to the meta description. Paywalled, bot-blocked, or expired-certificate pages are logged and skipped. The cache is read-only.

6. Ownership and motive

Bias labels only tell you where an outlet sits. The ownership layer tells you who pays for the framing. We join a manually-curated ownership registry to each source domain. Every entry includes the owner, owner type, parent company, motive, and a named evidence URL. No inference — primary sources or nothing.

7. What this is not

• We do not rate “truth.” We rate coverage structure and ownership transparency.
• We do not claim a story is underreported in absolute terms. We claim it is underreported relative to the non-Western share of sources covering it.
• We do not scrape paywalled full articles. Excerpts are lede-length only, rate-limited, and fall back gracefully.

8. Run it yourself

The pipeline is one command:

bun run scripts/run_pipeline.ts --date 2026-07-13

It clusters stories, fetches excerpts, renders the brief, rebuilds the archive index, and refreshes the ownership registry.