text-to-cad-program-generation-final-geometry-similarity-iou-against-reference-solid CLM-0223For text-to-CAD program generation, final-geometry similarity (IoU against a reference solid) carries almost no information about whether the generated program preserves the intended construction process: on multi-entity Text2CAD samples with Qwen3, sample-level IoU and a constraint-satisfaction rate over expected entities, Boolean roles, sketch properties and inter-entity relations correlate near zero (Pearson r = 0.022, Spearman 0.023), and high-IoU programs can omit expected entities or invert additive/subtractive roles.
Capability: Whether the measurement made the finding
Observed on
Multi-entity CadQuery programs generated from natural-language descriptions, 26,783 samples derived from Text2CAD; correlation measured on one backbone (Qwen3), on programs that ex.
Sources
- Measured: sample-level Pearson and Spearman between normalized IoU and CSR on one backbone, plus a qualitative case (IoU 90.41% with CSR 10.00%). Aggregate tables across three backbones show IoU and CSR moving by very different amounts. The design does not isolate how much of the low correlation is error in the LLM-inferred expected constraints, since no human-audited constraint ground truth is reported.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.Whether the measurement made the finding · reviewed by AI
- When NL-to-PDDL problem generation is scored by operational acceptance (parses, conforms to domain, a planner finds a plan, VAL validates it, an LLM critic accepts), the score overstates faithfulness to the described task, and the gap widens for PDDL 2.1 temporal and numeric problems where operationally accepted instances can almost never reconstruct the curated reference encoding.Whether the measurement made the finding · reviewed by AI
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- In whole-repository cross-language code migration (Java to Python), supplying an LLM with AST-derived dependency-graph context and translating parent classes before children lowers the rate of calls to nonexistent APIs relative to vector-similarity retrieval of text chunks, while lexical similarity metrics such as CodeBLEU show no difference between the two.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Measuring IoU against a construction-level constraint score on a comparable multi-entity CAD set and finding a strong positive correlation, or showing the near-zero correlation comes from noise in the LLM-inferred constraint set rather than from real construction errors. Drafted stance toward function-level-algorithmic-tasks-translated-c-model-s-functional-test: supports -- Both find that a pass/similarity-style correctness metric decorrelates from other quality dimensions of the generated code, here construction-structure correctness in CAD programs. Proposed technique, not catalogued: Constraint intent tree for CAD generation and verification.