In SNDA, my first large-scale research project, the natural-language augmentation that lifted validation MRR to 0.58 actively hurt on the test set. The model had memorized validation-specific quirks. Only ensembling and post-processing recovered the 0.35 that placed 7th. The gap between 0.58 and 0.35 was almost entirely a domain-generalization failure, and it is the lesson that has stuck hardest.
I have since seen the same shape elsewhere. The diversity collapse behind Every conditioning choice is a hidden assumption was a generalization failure too: a conditional model with the best FID and precision returned near-duplicate samples, perfect by the headline metric and useless at the thing I cared about. And when I reproduced an emotion classifier in Face Emotion Detection, the entire accuracy gap to the paper traced back to one skipped super-resolution step, a reminder that a reported number lives somewhere specific, and usually not where the abstract points.
The discipline I took from this: treat every validation gain as a hypothesis about the test distribution, not a result. Ask what specifically the gain might be exploiting, and assume anything that helps only on the split you tuned on is a liability until the held-out distribution agrees. Reading the data closely is part of the same habit, but the structure you find there has to survive this test too: Read the data before you add machinery.