Natural language is a powerful interface, but it’s a lossy projection of what agents need to coordinate on. Intentions, timing, attention, and affect all matter, and most of them don’t survive the text bottleneck.
Open question: what does a grounding signal look like if we stop privileging text? Some candidates I want to think through:
- Embodied state traces: position, velocity, gaze direction as first-class citizens.
- Attention as a signal: what an agent isn’t looking at is as informative as what it is.
- Counterfactual rollouts: a short imagined future as part of the message.
Prof. Freda Shi’s work on how grounding actually emerges inside model computations is a useful empirical handle on this: it suggests the signal we need may already be latent in models we have, if we knew where to look.
Update, Oct 2026. I now think the first and third candidates are right, just not as messages. Positions, velocities, and goals belong in the world state itself, where every agent can read them, rather than in a signal one agent sends another. That’s the move in World models need an explicit world state. Counterfactual rollouts become the director’s tool: compare how the world unfolds with and without an intervention, which is A director, not a script.