synthesizing-regexes-user-intent-neither-natural-language-description-nor-positive CLM-0256For synthesizing regexes from user intent, neither a natural-language description nor positive/negative examples alone is sufficient — descriptions are ambiguous and small example sets underdetermine the pattern — so a system that parses the description into a structural sketch and uses it to guide example-based search solves far more tasks than either modality alone (80% vs 43% NL-only and 26% examples-only on 300+ regexes).
Capability: Writing a reusable locator: regex, SQL, XPath
Observed on
Regexes over a fixed DSL, tasks drawn from a stylized NL dataset and from StackOverflow questions that already supply both a description and examples; the NL parser is a grammar-ba.
Sources
- Validated by: Exact-match against ground-truth regexes on 200 adapted DeepRegex benchmarks and 122 StackOverflow benchmarks, against an NL-only seq2seq baseline (DeepRegex) and an examples-only ablation of the same engine, plus a 20-participant user study where users were twice as likely to produce the intended regex.. Measured on two datasets totalling 300+ tasks with both baselines run under the same time and top-k budget, including an interactive protocol adding examples over four iterations. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For translating English descriptions into regular expressions, a generic attention-based sequence-to-sequence model trained only on description/regex pairs matches a domain-engineered semantic parser on a small corpus (814 examples) and beats it by a wide margin once the training corpus is scaled to 10,000 pairs, with no semantic-equivalence machinery built in.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- When training a sequence-to-sequence model to generate regular expressions from natural language, rewarding semantic equivalence to the reference (checked by DFA equivalence and test cases) yields more semantically correct regexes than maximum-likelihood training on the reference string, because one intent has many syntactically different equivalent regexes.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- Open-weight instruction-tuned models, told in plain language to not think about a concept while transcribing a sentence, suppress that concept's residual-stream projection enough to evade linear probes, the Jacobian lens, activation oracles and natural-language autoencoders, with no fine-tuning or prompt search — detection at a 1% false-positive budget falls to roughly 5-21% under suppression while engagement is caught near ceiling.Strategic deception and detecting it · reviewed by AI
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A system given only the natural-language description, or only the examples, reaching comparable accuracy on the same benchmarks; or the sketch-guided engine failing to beat its own examples-only ablation once both get equal search budget.