architecture · digit-position-embeddings

Encode digit position in the model

Add positional embeddings that tell the model which place each digit occupies.

Abacus embeddings give every digit an embedding for its position within the number, so alignment of digits across operands becomes trivial. Small transformers trained this way generalize to numbers far longer than seen in training.

Requires: A training-time change; not applicable to a model you cannot retrain.

Addresses: Digit-level arithmetic

Contexts: Math

Does it work?

nothing measured0 supporting · 0 contesting sources

Efficacy claims — what this technique actually moves, under which conditions, and whether that has been contested.

No efficacy claim filed yet. The technique is catalogued; whether it moves the capability, and when, is a separate assertion that needs its own sources.

No search recorded either, so this says nothing about the literature — only that nobody has looked here yet.

Code

Sources