ctscout research

Measured properties of Certificate Transparency as an entity-resolution source, from deterministic scripts over public data. Pages regenerate weekly from the live warehouse.

The ctscout research pipeline Certificate Transparency logs feed the certstream collector continuously; it drops domain-validated certificates and keeps organization-validated ones in a warehouse of certificate events, synced daily. The weekly refresh reduces those to organization-to-registrable-domain pairs with the extract scripts and fetches dated snapshots of the GLEIF golden copy, the Wikidata gold set and the ISIN-to-LEI mapping. Ten stats scripts run in order over the pairs and the snapshots, each writing an aggregate JSON; vendor_footprint also resolves live DNS through public resolvers. build_writeup renders the index, the notes and the vendor pages from that JSON; export_product writes the per-LEI product tree and the company pages. Everything goes to the ctscout-research R2 bucket, which the ctscout.dev Worker serves under /research and /lei. collect · continuous CT logs every active shard of five operators certstream collector drops DV; keeps OV and EV warehouse OV/EV certificate events, one row per log; synced daily DV dropped refresh · weekly: extract the pairs, fetch dated snapshots GLEIF golden copy fetch_gleif: every LEI, names, ownership Wikidata gold fetch_wikidata_gold: LEI + official website extract scripts extract_org_apex, extract_certs_by_month: organization × registrable domain ISIN to LEI fetch_gleif_isin: daily mapping measure · stats scripts, in this order matching_difficulty · gleif_join · lei_isin_join · coverage_stats · differs_decomposition · script_matching · vendor_footprint · vendor_series · short_lived_burst · vendor_customers_profile each writes output/<name>.json: aggregates with generated_at, snapshot dates and the PSL version live DNS public resolvers, CNAME chains; vendor_footprint only pairs build build_writeup + build_pages checks the JSON, then renders the index, the three notes and one page per vendor export_product + build_pages --companies per-LEI shards, vendor objects, name index; one company page per LEI output/*.json publish R2 bucket ctscout-research publication_progress: stage timing; publication_guard: unused dates; publication_snapshot: byte checks and backup; pages, data/*.json, vendors/, product/ by wrangler; companies/ by rclone ctscout.dev Worker /research, /research/data, /research/vendors, /research/companies, /lei serves
How a page on this site is made. Certificates are collected continuously; the weekly refresh extracts the pairs and fetches the registries as dated snapshots; the stats scripts write aggregate JSON; two builds render the pages and the product export; everything is uploaded to one bucket the site serves.
build_diagrams.py · stages from scripts/refresh.sh · run 2026-10-25 09:43 UTC

Data behind each note: coverage, matching difficulty, GLEIF join, decomposition, vendor footprint, vendor series, vendor customers profile.