arxiv-2606-10554 SRC-0361 · paper

Benchmarking Knowledge Editing using Logical Rules

Created: 2026-06-09 · Ingested: 2026-09-19

Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new benchmark designed to evaluate how knowledge editing methods handle the logical consequence

In brief

Knowledge editing methods that insert a fact into a model's weights mostly fail to propagate that fact to its logical consequences, and the gap shows up as near-zero exact match on entailed questions. The benchmark pipeline extracts entities from editing-benchmark questions with spaCy, pulls DBpedia triples via SPARQL, mines logical rules with AMIE3, then prompts an LLM to turn each rule plus its assertions into a multi-hop QA pair. Tested editors are ROME, constrained FT, and Knowledge Neurons on GPT2-medium, -large, and -xl, over MLaKE (English single-hop free-form subset) and MQuAKE-T, with 831 and 961 questions surviving entity linking from 1000 sampled prompts.

ROME on MLaKE drops from 16.36 F1 / 7.57 EM (direct) to 8.67 / 3.56 (correlated) on gpt2-medium; FT collapses further, gpt2-large_constr going 24.58 F1 to 0.45. KN inverts the pattern on gpt2-xl, scoring 1.34 F1 direct and 14.26 correlated on MLaKE.

The effect is measured, with direct and correlated evaluation on the same edited weights, so the comparison is clean within each method. But absolute scores are very low everywhere, EM is 0.0 in most cells, and the QA pairs are LLM-generated from automatically mined rules with no reported human validation; the sample rules in Table 1 include noisy predicates like thumbnail and primaryTopic.

Only GPT-2-scale models and parameter-based editors were tested; LLaMA, Mistral, and memory-based methods such as MeLLo were not. Treat edit-success numbers as uninformative about entailed knowledge.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsKnowledge-editing methods such as ROME and fine-tuning insert a stated fact reliably but do not carry its logical consequences: on multi-hop questions generated from knowledge-graph rules that the edit entails, accuracy is up to 24 points lower than on the directly edited fact, so an edited model can assert a new fact and still answer as if its implications never changed.