extracting-structured-rules-multimodal-enterprise-guideline-documents-wrapping-extraction CLM-0387When extracting structured rules from multimodal enterprise guideline documents, wrapping the extraction in deterministic schema validation, an LLM semantic judge, and a duplicate/contradiction consistency filter over a shared rule store lowers the share of extracted rules with no matching source sentence from about 15.7% to 3.2% relative to a one-pass pipeline using the same parser and models.
Capability: Stating false facts confidently
Observed on
120 real client guideline documents (PDF/DOCX/PPTX, table- and image-heavy), Qwen2.5-VL-32B extraction and Claude Sonnet 4.6 artifact generation; hallucination is defined as cosine.
Sources
- Validated by: Compared against a monolithic one-pass baseline with the same parser and same models on the same 115 documents: hallucination 3.2% vs 15.7%, duplication 3.0% vs 10.3%, artifact rule coverage 93.7% vs 62.8%; a cumulative ablation attributes duplicate removal (117 units), 113 contradictions and 1,040 gaps to the consistency layer specifically. The L2 judge was checked against 300 blind expert rule labels (F1 0.957, kappa 0.813).. Measured on 120 industry documents with a matched-model one-pass baseline, which isolates the governance layers from the base model. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When models must apply abstract, entity-independent procedural rules drawn from a shared pool, accuracy is limited by locating the right rules rather than by applying them: giving only the gold rules yields much higher exact-match than embedding those same rules in a pool of 100 or 1,000, and accuracy falls as the pool grows.Following an unfamiliar procedure · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- For synthesizing reusable XPath extractors from a few sampled web pages, a two-stage pipeline — first having a cheaper model extract both the target value and the neighbouring cue text from an attribute-stripped HTML tree, then having a stronger model write the XPath over HTML condensed around those located nodes with an execution-feedback loop — reaches usable extraction F1 on multi-domain product/listing pages while keeping prompt size far below feeding raw pages.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For structured extraction from text files, web pages and spreadsheets, a few user-highlighted example fields are enough to synthesise a reusable extraction program that pulls the remaining fields without further interaction — the locator can be inferred from examples rather than written or re-invoked per record.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A replication where the one-pass baseline, scored with human-annotated grounding rather than cosine similarity to source sentences, shows a similar ungrounded-rule rate to the governed pipeline — or where the reduction is shown to come only from the schema layer rejecting rules rather than from better grounding.