training-reversed-sequences-alongside-forward-text-largely-removesmechanismsingle paper
Training on reversed sequences alongside forward text largely removes the effect.
Capability: Not generalizing "A is B" to "B is A"
Sources
- Training on reversed sequences alongside forward text largely removes the effect.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.Losing information in long inputs · contested