Footnotes to Plato

Pipeline & methods record

The build as actually executed — every decision, assumption, and known limitation

*Companion to ancient-citation-heatmap-handoff.md. Records the build as actually executed: the pipeline, every decision, the assumptions tested, the probes run, and the resulting data quality. Written to be lifted into a methods section. July 2026.*

0. Summary of outcome#

Starting from a 381 MB gzipped JSONL full-text delivery from JSTOR Text Analysis Support (69,177 documents, one requested item withheld by JSTOR), we built a pipeline that extracts candidate ancient-citation strings from article body text, resolves them to specific works of Plato and Aristotle, and reports per-work resolution quality.

Final state on the genuine-citation set (after excluding non-scholarly content):

Crucially, the pipeline was built precision-first: at every stage the default was to decline to resolve rather than guess, and every exclusion diverts data to an auditable file rather than deleting it. The residual queue is genuine, quantified ambiguity, not silent error.

1. The delivered data — what JSTOR actually sent#

Decision point / surprise. The request was expected to return n-grams (the handoff planned around trigram/4-gram token counts, with the "riskiest technical unknown" being whether citation strings survive tokenisation). In fact JSTOR delivered full text, one record per article:

{ "iid": "<id>", "full_text": [ "<page 1 text>", "<page 2 text>", … ],
  "references": [ "<raw citation string>", … ] }

Consequences, all favourable:

Probe used: dumped the keys and structure of the first delivered record before writing any processing code (schema-probe-first discipline).

2. Coverage reconciliation#

Assumption tested: that the delivery matches the request.

This reconciliation is re-run on every extraction pass (coverage_report.txt).

3. The extraction stage (build_citation_db.py)#

Streams the delivery, joins metadata, and emits one row per citation-shaped match to citations.tsv. Design decisions:

3.1 Citation-shape regexes and the three-way corpus tag#

Ancient citations have two systems whose shapes overlap:

The letter partly disambiguates, so extraction tags each match three ways:

Two bugs caught and fixed during testing (both by looking at real cases):

  1. Initial regex required 3 leading digits (\d{3}), silently dropping all sub-100 Stephanus citations (e.g. 80d5, Meno). Widened to \d{2,3}. Rationale: under-capture is worse than over-capture — noise can be filtered downstream using context; missed citations cannot be recovered.
  2. Initial code hard-assigned 327a-type overlaps to Bekker via precedence. Replaced with the mutually-exclusive three-way tag above, so genuinely ambiguous forms are never mislabelled — they are explicitly deferred.

3.2 Context window and reading-order fields#

Each match row carries a ±120-char context window (newline-flattened), plus page_index, char_offset, and a per-article seq counter. The latter three exist to support scope-tracking (see §4.4): they preserve reading order and inter-citation distance in a sort-proof way.

3.3 Metadata join and the two engineering hazards#

3.4 Title join (added later, see §4.4)#

A title column is joined from metadata by iid, sanitised of tabs/newlines. Confirmed populated for journal articles (the null titles observed in probing were JSTOR "contributed_content" community-collection records, not journal articles).

4. The resolution stage (resolve_citations.py)#

Reads citations.tsv, applies exclusion filters, then resolves each surviving candidate through an ordered architecture. Everything is conservative by design: confidence ≥ threshold and no flagged reason ⇒ resolved.tsv; else review_queue.tsv; excluded content ⇒ excluded.tsv.

4.1 Reference tables used#

4.2 Resolution architecture (precedence order)#

For each candidate: compute range candidates (which work(s) contain the page); then resolve by, in order of authority —

  1. Unique range — one candidate ⇒ accept. A mentioned work-name elsewhere in the context does not override a unique range (see §4.3, bug 1).
  2. Range + name — several candidates, context names exactly one ⇒ accept.
  3. Cross-system resolution — an ambiguous number whose candidates span both Bekker and Stephanus, disambiguated by an adjacent name ⇒ accept, tagged cross_system_resolved_by_name.
  4. Scope inheritance — a bare number inherits the last-named work in reading order, confidence decaying with distance, accepted only if the page falls in that work's range.
  5. Title prior — a bare in-range number in an article titled after a work resolves to that work, if the title uniquely intersects the range candidates.
  6. Otherwise ⇒ review queue, tagged with the specific ambiguity.

4.3 Bugs caught by inspecting resolved/queued samples#

The resolution logic was corrected three times, each prompted by reading actual rows rather than aggregate counts:

  1. Mentioned-vs-cited (the biggest recall bug). A unique 4-digit Bekker number (e.g. Politics 1254b) was being queued at low confidence whenever another work was mentioned nearby — e.g. Aristotle discussing Plato's Republic. But a mentioned work is not the cited work; a globally-unique Bekker page is authoritative regardless. Fixed: unique range wins; a non-matching name is recorded (range_unique_name_mentioned) but never blocks resolution. Recovered ~7k outright and cut heavily into ~37k more.
  2. Cross-system mislabelling. 184a/100a-type numbers are ambiguous across pagination systems (Aristotle Bekker vs a Plato Stephanus page), not merely across Bekker treatise boundaries. Reworked so "which system" and "which work within a system" are separate axes; auto-accept only when a name collapses the full cross-system candidate set to one.
  3. Scope threshold + a contaminant. Sub-threshold scope hits were mostly correct (validated by sampling), but included a contaminating pattern (12.25e = Polybius book.section, not Stephanus). Added a book_section_ref filter, then lowered the scope-accept gate — the two together, since lowering alone would have admitted the contaminant.

4.4 Scope-tracking and the title prior#

4.5 Reference-table repairs made during the build#

Boundary line-numbers used (Ross 1931): De Sensu 436a–449a34; De Memoria 449b3–453b10; De Somno 453b11–458a33; De Insomniis 458b–462b12; De Divinatione per Somnum 462b13–464b18; De Longitudine Vitae 464b19–467b9; De Juventute / De Vita et Morte / De Respiratione (one continuous block) 467b10–480b30.

Note carried forward: the added variants are marked verified: false (standard scholarly forms, not yet cross-checked against a concordance), and the remaining Bekker boundaries outside the Parva Naturalia band are still unverified — the highest-value future table work.

4.6 Book-level faceting — derived and verified (this session)#

The four highest-volume works (Republic, Laws, Metaphysics, Nicomachean Ethics) are large enough that a single passage axis is unreadable; the interface will facet them by book (Rep. I–X, Metaphysics Α–Ν, etc.). This required a book value per citation, and surfaced two things:

Boundaries are verified, not heuristic. Exact book-opening loci were looked up against Burnet (Stephanus) and Bekker and inlined as deterministic split points. Each book spans [its opening locus, the next book's opening locus); a citation's (page, section, line) key selects the containing interval. Mid-page openings are handled exactly — Metaphysics little-alpha at 993a30 (so 993a29 → Α, 993a30 → α), NE II at 1103a15 (1103a14 → I, 1103a15 → II). Result: 100.0% of 35,968 faceted-work citations assigned, 0 unassigned.

Three sub-problems worth recording:

This retires a limitation. Book-boundary assignment for these four works was previously flagged as unverified; it is now verified against Burnet/Bekker and no longer needs a caveat. (The other Bekker treatise boundaries — §4.5, §9 — remain unverified; this session verified only the four faceted works' internal book divisions.)

A handful of edge citations (bare opening-page references like NE 1094a, and the genuinely line-ambiguous case of a bare 1103a where a new book opens mid-section) are assigned deterministically to the book owning the top of the cited section, and are rare enough to footnote rather than agonise over.

5. Exclusion filters — what was removed and why#

All exclusions **divert to excluded.tsv with a reason**, never delete. Filters were added one at a time, each justified by inspecting real false positives.

ReasonCountWhat it catchesJustification
excluded_doctype54,409misc + non-content back-matterScope decision — see §5.1
modern_year4,762number ≥ 1463Above Bekker's ceiling (~1462); can only be a publication year
book_section_ref4,22712.25e-type refs (leading N.)Book.chapter reference to another author (e.g. Polybius)
foreign_text1,070Aj. 134a, Od. 2.107a, etc.Adjacent marker names a non-Plato/Aristotle text
dk_fragment190number + DK/D-K/DielsDiels-Kranz Presocratic fragment, not a passage citation

Total excluded: 64,658 (31.4% of raw candidates).

5.1 The doctype decision (the largest single exclusion)#

This was the most consequential methodological call, and it was made on evidence, not assumption. misc doctype was 26.6% of raw candidates — too large to exclude or keep blindly. A dedicated probe (probe_misc.py) found:

Decision: **exclude misc and non-content back-matter; keep research-article, book-review, and discussion.** Framed as a scope decision (indices record locators, not scholarly citations), which also removes the index-page contamination cleanly — replacing an earlier, fragile density/count heuristic that testing showed could not separate real dense footnotes from index pages (both scored ~0.45–0.53 numeric density). A key lesson: the discriminator was doctype, not any within-text signal.

5.2 Reserved for future use#

The dk_fragment exclusions are a ready-made Diels-Kranz dataset for a possible future Presocratics heatmap. excluded.tsv filtered by reason keeps each class recoverable.

6. Data quality — what the resolution rates mean#

A resolution rate is not an accuracy figure. "Phaedrus 61%" means 61% of candidate Phaedrus citations were confidently assigned to Phaedrus; the other 39% are unassigned (queued), not misassigned. The resolved set is clean but incomplete. Two distinct distortions follow, and both were measured (missingness.py):

6.1 Within-work distortion (which passages get attention)#

A citation is unresolved mainly when it is bare and on a page that collides with another work's range. So the risk is that passages on colliding pages are under-represented relative to passages on clean pages. Measured per work as the gap between a page-band's resolved-share and the work's average rate.

This session produced the exhaustive band table (see §6.4 for the method change and the full result; the machine-readable output is collision_bands.json). The earlier "~8 bands" figure was an artifact of a diagnostic that reported only each work's single worst band; re-run to report every band above threshold, the true set is 21 bands across 12 works. Headline shape:

Correction to an earlier characterisation. A prior draft listed Meno 100–109 and Apology 10–19 among the collision bands. The exhaustive re-run shows this is wrong: Meno, Apology, Crito, Charmides, Lysis, Categories and the other low-resolution short dialogues are uniformly hard, not locally collided — their missingness is spread evenly across their whole span, so no single band stands out. Their uncertainty is a cross-work (§6.2) phenomenon (wide floor/ceiling bar), not a within-work one. See §6.4 — this distinction is the single most important empirical finding of the visualisation-design stage.

6.2 Cross-work distortion (relative weight of works) — bound, not point-estimated#

Raw resolved counts are not comparable across works, because resolvability varies (Republic 95%, Meno 26%). Plotting raw counts systematically understates the short, collision-prone dialogues.

Correction on the framing (revised this session). An earlier version of this record proposed dividing resolved count by resolution rate to get an "estimated true count" (est_true). That formula collapses to a no-op: since rate = resolved / (resolved + queued), the quotient resolved / rate is identically resolved + queued — i.e. it just recovers the candidate total. The queue is the missing citations; there is no separate model to fit. The honest object is therefore not a point estimate but a range:

The truth lies between, and the width of the interval is the uncertainty. This is both more correct and more legible than an error bar derived from the rate: the interface should render each work as a floor-to-ceiling bar, where a tight bar means the count is solid and a wide bar means genuine uncertainty about how studied the work is.

The widths are dramatic and carry the finding. Examples (resolved → total): Republic 12,350 → 12,997 (essentially certain); Timaeus 3,977 → 4,198 (tight); but Apology 1,738 → 6,822, Meno 1,575 → 6,156, Crito 637 → 3,261, Categories 122 → 1,389. On raw counts Apology (1,738) ranks below Timaeus (3,977); on the ceiling it outranks it. That inversion is exactly the distortion this section warns about, made visible by the bar width rather than hidden behind a single number.

Note this is the cross-work uncertainty axis (how much a work is studied in total). It is distinct from, and must not be conflated with, the within-work collision-band axis of §6.1 (which passages are under-counted). The two are empirically different populations of works — see §6.4.

6.3 What the heatmap can claim (three products)#

  1. Resolved-count heatmap — "at least this much attention." Rock-solid everywhere; a floor.
  2. Rate-corrected heatmap (est_true) — honest cross-work comparison. Valid for ~45 works; caveated for the localised-gap works and the spuria.
  3. Within-work passage heatmaps — trustworthy for Tier A; footnoted on specific collision-bands for Tier B.

The handoff's original caveat still stands and should be repeated: raw frequency conflates the contested passage, the boilerplate-cited passage, and the translation crux; a blank cell may mean "understudied" or "settled." A raw-vs-normalised toggle remains advisable.

6.4 Two uncertainty axes — the key design finding#

The visualisation-design stage established, on evidence, that uncertainty in this dataset is two structurally different phenomena, and that a single uncertainty treatment would misdescribe both. The exhaustive band re-run (§6.1) partly overturned the prior expectation and made the distinction sharp:

The prediction going in was the opposite (that the low-rate short dialogues would be the band-flagged ones). The data refuted it, and the refutation is the design rationale: shading a uniformly-hard work's whole span would be wrong (it is thin everywhere, not collided), and leaving Laws IX–XII unshaded would be wrong (real localised holes in an otherwise reliable text). The interface must therefore carry both treatments and keep them visually distinct — a floor/ceiling bar per work at the text level, and named band shading per passage within a work.

6.5 Page-range integrity audit (this session)#

To confirm no mis-parsed page numbers were silently inflating any band, every resolved citation was checked against its work's known Stephanus/Bekker span (audit_page_ranges.py). Result: 3 out-of-range citations in 90,502 (0.003%) — all three in De Juventute, and all three attributable to adjacent-work Bekker citations bleeding in from the tightly-packed Parva Naturalia cluster (e.g. a Metaph. 1067a6 marker, or a neighbouring treatise's page). No systematic page-parse leakage anywhere in the corpus. This is a publishable integrity result: it can be stated plainly that a full range audit found out-of-range parsing in 0.003% of citations, all from known adjacent-work collisions.

*(One cosmetic note carried to the build: missingness.py flags a Republic "60–69" band of 39 citations. Since the range audit shows zero Republic citations parse outside 327–621, these are not out-of-range; the "60–69" label is a low-volume binning artifact, not a data-integrity problem. Immaterial to Republic's 95% reliability; note but do not chase.)*

7. Method discipline (worth stating in the writeup)#

Three principles were held throughout, and account for the data quality:

  1. Look before filtering. Every exclusion filter and threshold was set by inspecting real samples (random and targeted), never by assumption. Several plausible-sounding filters (numeric-density for index pages; aggressive citation-count cutoffs) were rejected after measurement showed they would remove genuine data.
  2. Precision over recall; decline over guess. Ambiguous cases are deferred to the review queue, never force-resolved. The 64.1% is a trustworthy floor, not a ceiling reached by relaxing correctness.
  3. Divert, don't delete. Every excluded row is retained with a reason, so every decision is auditable and reversible, and by-products (Diels-Kranz) are preserved for reuse.

8. Pipeline artifacts (file inventory)#

Scripts (in run order):

Visualisation-stage artifacts (this session):

Diagnostics (throwaway/analysis, kept for reproducibility):

Reference tables (some repaired this build):

Key intermediate/outputs:

9. Known limitations & the honest to-do list#

*End of methods record. Visualiser design is now settled (see the handoff memo's "Interface design — settled" section); the build files (resolved_with_books.tsv, collision_bands.json) exist and are verified. Next stage: build the interface (new working session).*