Every conditioning choice is a hidden assumption

Lessons from extending IMLE to text-conditional generation: failures almost always traced to latent mismatch.

evergreen #generative-models #imle #lesson planted Jan 15, 2026 · tended Feb 25, 2026

At APEX Lab, I spent a summer benchmarking conditioning architectures for text-to-image IMLE: FiLM affine modulation, per-block residual injection, StyleGAN-style conditioning, mapping-network concatenation. Most of them overfit.

Each failure traced back to a single thing: a mismatch in the latent representation between the condition space and the output space.

Concrete example. Unit-normalising CLIP embeddings collapsed diversity and made overfitting worse. On paper this is fine: cleaner conditioning should help. In practice, normalised embeddings were overwhelmed by the latent noise drawn from the prior, so the condition lost its grip on the output.

Every architectural choice implies an assumption about how a conditioning signal is encoded to steer what it controls.

This generalizes beyond IMLE. Designing a shared world representation for multiple agents is the same problem scaled up: the encoding has to match the thing it’s trying to steer. Back to World models as shared substrate.

Linked notes

World models as shared substrate · Why a common representation of the world is upstream of believable multi-agent behaviour.

Backlinks

A world in a grain of sand · Blake's image, read through world models: the whole inferred from the detail, both as a design principle and as evidence the model worked.
The loss decides what the model ignores · Two projects where the objective, not the architecture, dictated what the model paid attention to and what it left blank.
The test distribution is the only one that counts · A strong validation number can be a trap. My clearest version of this lesson came from a 0.58 that became a 0.35.
The bottleneck is rarely the model · Across four very different projects, the thing that capped quality was almost never the network. It was a systems or representation cost hiding underneath.
Grounding signals beyond language · If we want agents to share more than what can be said, the grounding signal has to be richer than text.
Why now · A personal note on the path from audio processing at Huawei to wanting to spend six years on world models.
What even is a world state? · We talk about world models as if 'the state' were given. The representation is the hard part.