good-regulator-theorem ADG-0009

Good regulator theorem

“Every good regulator of a system must be a model of that system.”

Origin

Roger Conant and W. Ross Ashby, 1970. A theorem in cybernetics: to control something well you must, in effect, contain a model of it.

Why it should, or should not, apply to models

Read against language models the theorem cuts both ways. A model that acts well in an environment must hold some model of it — and the theorem says the quality of the action is bounded by the quality of that internal model, not by how well the next step is predicted. So it predicts a specific failure: an agent that predicts familiar traces well but holds an incoherent model of the world will fail as soon as the task leaves the traces. It also predicts the fix: give the agent an explicit, externally maintained model of the environment and its regulation improves.

Standing: Holds

Standing counts claims a person has reviewed. Claims reviewed by AI alone are listed and marked, and do not move it.

Where it holds

Claims showing the heuristic applying to models.

Models that predict next steps well can still hold an incoherent implicit world model, which fails when the task deviates from familiar traces.
Tracking state through a long taskReviewed by AI and Russ Miller
Good next-step prediction with an incoherent implicit world model fails off the familiar traces — the theorem's bound, observed.
In partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.
Tracking state through a long taskReviewed by AI
Supplying an explicit state estimate of the environment raised task success over a memory of raw observations: a better model of the system made a better regulator.

Add a case where it holds or breaks

A break is worth more than a hold. If you have seen this fail for a model, with the setup written down, that is the most useful thing you can add here.