suite-verifier-checked-tool-using-react-task-families-arithmetic-sql-analytics CLM-0492On a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.
Capability: Whether the measurement made the finding
Observed on
Four models (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) on 2,057 committed single-rollout tasks; one task per rollout, no multi-turn or long-horizon settings..
Sources
- Validated by: Four models scored on 13 families; overall scores 0.88/0.86/0.82/0.71 while gpt-oss-120b led ug_fetch (0.78) and ug_imposter (0.89) yet scored 0.24 on sql_analytics and 0.25 on tool_error_recovery. No human validation of the families' construct validity.. Measured on one paper's own benchmark with only four models and no external replication; family difficulty is not calibrated against other suites, so part of the spread could be family-specific prompt or format sensitivity rather than skill.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For small open-weight backbones (Qwen3-4B) driving long-horizon remote-sensing tool chains, inserting a step-level verifier that checks each intermediate tool output against schema, sub-goal and domain constraints and triggers replanning is the single largest architectural gain, while hierarchical decomposition without domain-aligned tool grouping adds little and can lower final accuracy relative to a flat agent with the same verifier.Completing tasks that take many steps or many hours · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.Completing tasks that take many steps or many hours · reviewed by AI
- On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A larger model set in which per-family accuracies are near-monotone in the aggregate score, so no model's ranking inverts between skill families. Drafted stance toward a-single-aggregate-benchmark-conceals-domain-trajectories-that-differ-in-shape: supports -- Both find that one blended benchmark number conceals per-domain differences that invert model ordering.