arxiv-2502-15688 SRC-0253 · paperXPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler
Yu Li, Bryce Wang, Xinyu Luan
Created: 2024-12-17 · Ingested: 2026-09-19
https://arxiv.org/abs/2502.15688(opens in a new tab)We present XPath Agent, a production-ready XPath programming agent specifically designed for web crawling and web GUI testing. A key feature of XPath Agent is its ability to automatically generate XPath queries from a set of sampled web pages using a single natural language query. To demonstrate its effectiveness, we benchmark XPath Agent against a state-of-the-art XPath programming agent across a range of web crawling tasks. Our results show that XPath Agent achieves comparable performance metrics while significantly reducing token usage and improving clock-time efficiency. The well-designed two-stage pipeline allows for seamless integration into existing web crawling or web GUI testing workflows, thereby saving time and effort in manual XPath query development. The source code for XPath Agent is available at this https URL .
In brief
Pruning a web page down to text and tags, then asking a model for both the target value and the "cue text" that labels it, is enough to get usable XPath queries from 3 sample pages. The claimed efficiency gain over a state-of-the-art competitor is asserted rather than measured.
XPath Agent splits the job into 2 stages: a weaker model extracts target information plus cue text from a sanitized page, then a stronger model writes XPath over a condensed DOM, with a feedback loop capped at 3 rounds that runs the query on seeded pages and reports missing and redundant matches. Evaluation uses SWDE, 90 websites across 9 domains and 20414 pages, with 3 seeds per task and 32 pages sampled per task.
GPT 4o leads on F1 at 0.8514 and accuracy at 0.7413; GPT 4o mini reaches 0.7961 F1, Claude 3.5 0.7647, DeepSeek V2.5 0.7337. Sanitization cuts page size to 10%–20% on average.
The headline claims about token usage and clock time carry no numbers anywhere in the text, and the AutoCrawler comparison is prose speculation with no table. The error analysis section is a TODO and the conclusion is unwritten boilerplate.
No ablation isolates the cue-text contribution despite it being listed as a contribution, and only SWDE-style semi-structured pages are tested.
Treat the architecture as a design to copy, not the efficiency claim as established.
Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.