Extraction evaluation
Four extraction methods, scored against a manually annotated gold subset, plus an error-propagation study asking how much extraction mistakes actually move the network conclusions.
Gold subset
10 chapters (1, 2, 17, 36, 45, 54, 77, 92, 111, 129), 310 annotated relations, one annotator (see annotation guidelines).
RQ1 · relation extraction, by method
| Method | Edges found | Strict F1 | Lenient P | Lenient R | Lenient F1 | Pair-level F1 | Recall (explicit) | Recall (implicit) |
|---|---|---|---|---|---|---|---|---|
| Sentence co-occurrence (baseline) | 162 | — | 46.9% | 75.3% | — | 0.578 | — | — |
| Rule-based | 44 | 0.203 | 85.4% | 18.4% | 0.303 | 0.400 | 31.1% | 7.0% |
| Rules + coreference | 85 | 0.321 | 82.9% | 30.5% | 0.446 | 0.558 | 50.0% | 13.0% |
| LLM structured extraction | 169 | 0.255 | 46.2% | 34.7% | 0.396 | 0.589 | 42.2% | 28.0% |
Lenient scoring collapses matching to (chapter, type, unordered pair) — it credits the extractor for finding that two characters interact in a chapter even off by a sentence. The co-occurrence baseline wins on raw pair recall but carries no relation type and a much higher false-positive rate; it is a floor, not a competitor.
RQ2 · which relation types are hardest
| Type | Rule-based | Rules + coreference | LLM structured extraction |
|---|---|---|---|
| kinship | 0.476 | 0.731 | 0.370 |
| marriage | 0.000 | 0.000 | 0.333 |
| friendship | 0.500 | 0.444 | 0.222 |
| conversation | 0.344 | 0.457 | 0.463 |
| conflict | 0.235 | 0.421 | 0.468 |
| killing | 0.222 | 0.316 | 0.467 |
| support | 0.069 | 0.129 | 0.171 |
Marriage and support are the hardest relation types across every method: both are rare in the gold set and marriage in particular is often stated by a noun phrase ("his wife X") the rule set does not yet cover as thoroughly as verbs.
Entity resolution & coreference
Entity linking (against the curated table)
Precision 96.7% · Recall 98.9% · F1 0.978
spaCy's generic PERSON-NER catches only 53.5% of gold mentions on this translation's proper names — the gazetteer built from the curated entity table is what makes linking viable at all.
Pronoun coreference
65.4% of all annotated pronouns resolved correctly (73.6% of those the system attempted).
| Pronoun | Correct | Wrong | Unresolved |
|---|---|---|---|
| he | 65 | 17 | 7 |
| she | 15 | 2 | 8 |
| his | 51 | 18 | 4 |
| him | 35 | 25 | 9 |
| her | 12 | 2 | 2 |
Method agreement
| Pair | Jaccard | Overlap |
|---|---|---|
| llm vs. rules | 0.089 | 34 |
| llm vs. rules+coref | 0.085 | 45 |
| rules vs. rules+coref | 0.610 | 283 |
The rule-based extractor with and without coreference agree on most edges (as expected — coreference is additive); the LLM overlaps with the rule-based extractor on barely a tenth of edges, which is itself evidence that "extraction" is not one well-defined task these methods converge on.
RQ3 · error propagation into network conclusions
Baseline graph: 160 nodes, 382 edges. Each row perturbs the graph at a multiple of the measured error rate and reports the mean, over 40 trials, of how much the perturbed graph's centralities and communities still agree with the unperturbed one.
| Perturbation | Setting | ρ betweenness | ρ PageRank | Top-10 betweenness overlap | Community NMI | Community ARI |
|---|---|---|---|---|---|---|
| miss | ×0.5 measured rate | 0.880 | 0.941 | 84.8% | 0.728 | 0.570 |
| miss | ×1 measured rate | 0.751 | 0.827 | 76.0% | 0.620 | 0.349 |
| miss | ×2 measured rate | 0.544 | 0.595 | 61.8% | 0.580 | 0.098 |
| spurious | ×0.5 measured rate | 0.961 | 0.987 | 96.5% | 0.870 | 0.793 |
| spurious | ×1 measured rate | 0.931 | 0.977 | 91.0% | 0.815 | 0.719 |
| spurious | ×2 measured rate | 0.894 | 0.960 | 88.3% | 0.754 | 0.646 |
| merge | k=2 | 0.990 | 0.995 | 95.5% | 0.913 | 0.861 |
| merge | k=5 | 0.980 | 0.986 | 92.5% | 0.867 | 0.796 |
| merge | k=10 | 0.957 | 0.972 | 83.8% | 0.800 | 0.689 |
| split | k=2 | 0.984 | 0.993 | 95.0% | 0.909 | 0.830 |
| split | k=5 | 0.966 | 0.982 | 89.3% | 0.869 | 0.774 |
| split | k=10 | 0.937 | 0.966 | 78.0% | 0.795 | 0.658 |
Rank-based centrality (who is important) is comparatively robust to plausible extraction error; community assignment (who belongs together) is not — at the measured miss rate, community agreement (ARI) is already down to roughly a third of perfect, and doubling the error rate collapses it further. Any claim in this project resting on Louvain community labels should be read with that fragility in mind; claims resting on the betweenness or PageRank ranking of the top few characters are comparatively safe.
Annotation / review queue
Verify, reject, or retype a flagged relation below, or resolve an ambiguous mention. To merge a duplicate entity, confirm an alias, or split a bundled entity in two, use the Curate this entity section on that character’s profile page.
60 of 60 low-confidence items still open, ranked worst-first. Actions here write to data/annotations/njala/revisions.jsonl and take effect the next time the pipeline runs.
kinship/child: kol-egilsson → auto-asmund (confidence 0.19, rule noun:son)
“He was the son of Asmund Ashside, east out of Smoland.”
killing/confessed: skarphedinn → sigmund-lambason (confidence 0.22, rule dialogue-first-person:slay)
“"When I slew Sigmund the white," says Skarphedinn, "Gunnar of Lithend's kinsman."”
friendship/friend: flosi → wolf-aurpriest (confidence 0.23, rule noun:friend)
“There dwelt Lodmund Wolf's son, but he was a great friend of Flosi, and there he stayed that night, and next morning Lodmund rode with him into the Dale.”
kinship/kin: hrut → sigmund-lambason (confidence 0.24, rule noun:kinsman)
“Now the time for the Thing was coming on, Unna spoke to Sigmund Auzur's son, and asked if he would ride to the Thing with her; he said he could not ride if his kinsman Hrut set his face against it.”
kinship/parent: hrodny → skarphedinn (confidence 0.24, rule noun:mother)
“His mother was Rodny, and she was Hauskuld's daughter, the sister of Ingialld of the Springs.”
kinship/sibling: auto-thorwalld → thorkel-geitisson (confidence 0.24, rule noun:brother)
“So they fared below Lagarfleet and over the heath to Njardwick; there two brothers dwelt, Thorkel the allwise, and Thorwalld his brother; they were sons of Kettle, the son of Thidrandi the wise, the son of Kettle rumble, son of Thorir Thidrandi.”
kinship/parent: kormlada → olaf-tryggvason (confidence 0.24, rule noun:mother)
“He was a son of Olaf rattle, but his mother's name was Kormlada; she was the fairest of all women, and best gifted in everything that was not in her own power, but it was the talk of men that she did all things ill over which she had any power.”
kinship/child: unna → mord-fiddle (confidence 0.28, rule noun:daughter)
“He had an only daughter, named Unna.”
killing/slaying: kari → sigmund-sigfusson (confidence 0.28, rule verb:thrust)
“He thrust with the right hand at Sigmund Sigfus' son, and smote him on his breast, and the spear came out between his shoulders, and down he fell and was dead at once.”
killing/slaying: kari → sigmund-sigfusson (confidence 0.28, rule verb:smite)
“He thrust with the right hand at Sigmund Sigfus' son, and smote him on his breast, and the spear came out between his shoulders, and down he fell and was dead at once.”
killing/slaying: kari → sigmund-sigfusson (confidence 0.28, rule verb:smite)
“He thrust with the right hand at Sigmund Sigfus' son, and smote him on his breast, and the spear came out between his shoulders, and down he fell and was dead at once.”
support/asked: asgrim → helgi-njalsson (confidence 0.30, rule asked-for-help)
“Njal's sons went to see Asgrim Ellidagrim's son, and asked him for aid.”
killing/slaying: kari → kol-thorsteinsson (confidence 0.32, rule verb:smite)
“Kari turned on Arni Kol's son and cut at him, and smote him on the shoulder, and cut asunder the shoulder blade and collar bone, and the blow went right down into his breast, and Arni fell down dead at once to earth.”
kinship/child: njal → thorgeir-starkadsson (confidence 0.32, rule noun:son)
“He was the son of Thorgeir Gelling, the son of Thorolf.”
kinship/child: thorgeir-starkadsson → auto-thorolf (confidence 0.32, rule noun:son)
“He was the son of Thorgeir Gelling, the son of Thorolf.”