[{"content":"The naive way to animate many agents is one big model with memory over everything, emitting each agent\u0026rsquo;s behaviour in turn. That\u0026rsquo;s a puppeteer, not a society: the agents have no state of their own, they\u0026rsquo;re projections of a single process, and it shows the moment the situation drifts outside what the puppeteer already knows how to stage.\nGenerative Agents: Interactive Simulacra of Human Behavior (Park et al., 2023) is the serious answer to this. Each of the twenty-five agents in their sandbox town keeps its own memory stream, retrieves from it, reflects experiences into higher-level beliefs, and plans from there. No central controller. And it works: coordination genuinely emerges, the canonical example being one agent deciding to throw a Valentine\u0026rsquo;s party and the invitation propagating through the town until others show up.\nBut notice where the ceiling sits. Every agent runs on the same frozen base model, and everything they share passes through natural language: memory written as text, intentions communicated as text. They share a channel, not a world. The town itself is authored, not something their representations co-evolve with. So the coordination is only ever as rich as what survives the text bottleneck, which is exactly the limit I worry about in Grounding signals beyond language.\nThat\u0026rsquo;s why I read the subtitle literally. Simulacra. You can get strikingly social behaviour out of per-agent memory plus a language channel and still not have agents that share a world. Closing that gap, a substrate the agents jointly maintain rather than a channel they message across, is what World models as shared substrate has to be about.\n","permalink":"https://quangminhdinh.github.io/garden/shared-channel-not-shared-world/","summary":"\u003cp\u003eThe naive way to animate many agents is one big model with memory over everything, emitting each agent\u0026rsquo;s behaviour in turn. That\u0026rsquo;s a puppeteer, not a society: the agents have no state of their own, they\u0026rsquo;re projections of a single process, and it shows the moment the situation drifts outside what the puppeteer already knows how to stage.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003ca href=\"https://arxiv.org/abs/2304.03442\"\u003eGenerative Agents: Interactive Simulacra of Human Behavior\u003c/a\u003e\u003c/em\u003e (Park et al., 2023) is the serious answer to this. Each of the twenty-five agents in their sandbox town keeps its own memory stream, retrieves from it, reflects experiences into higher-level beliefs, and plans from there. No central controller. And it works: coordination genuinely emerges, the canonical example being one agent deciding to throw a Valentine\u0026rsquo;s party and the invitation propagating through the town until others show up.\u003c/p\u003e","title":"A shared channel is not a shared world"},{"content":" To see a World in a Grain of Sand, And a Heaven in a Wild Flower, Hold Infinity in the palm of your hand, And Eternity in an hour.\n— William Blake, Auguries of Innocence\nI read this as a statement of the world-model aspiration, which is almost certainly not what Blake meant.\nEvery line makes the same move: something vast revealed by something small. A grain holds a world, a flower a heaven, a palm infinity, an hour eternity. That move, the whole inferred from the detail, is the problem I care about, stated as an image instead of an objective function.\nThe first couplet is the cleanest version of it: a world recovered from a fragment, the rest of it implied by one local observation if you can see correctly.\nThe second couplet is the one I keep returning to, because for a world model it cuts two ways. As a design principle: the route to the infinite runs through the details. You don\u0026rsquo;t reach the whole by reaching for the whole, you get the small things right and the rest follows. As a test of success: when a model really can hold infinity in a palm, recover the vast from a sliver, that is the evidence it worked. The same image is both the aspiration and the way you would know you had met it.\nBlake meant something mystical, innocence, the divine glimpsed in the small. I\u0026rsquo;ve swapped in a claim about representation learning, and I don\u0026rsquo;t think he\u0026rsquo;d recognize my version. I like it anyway. Sometimes an image is true in a register its author never reached for.\nThe aesthetic under World models as shared substrate and What even is a world state?: the grain, seen rightly, was the world all along.\n","permalink":"https://quangminhdinh.github.io/garden/world-in-a-grain-of-sand/","summary":"\u003cblockquote\u003e\n\u003cp\u003eTo see a World in a Grain of Sand,\nAnd a Heaven in a Wild Flower,\nHold Infinity in the palm of your hand,\nAnd Eternity in an hour.\u003c/p\u003e\n\u003cp\u003e— William Blake, \u003cem\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Auguries_of_Innocence\"\u003eAuguries of Innocence\u003c/a\u003e\u003c/em\u003e\u003c/p\u003e\u003c/blockquote\u003e\n\u003cp\u003eI read this as a statement of the world-model aspiration, which is almost certainly not what Blake meant.\u003c/p\u003e\n\u003cp\u003eEvery line makes the same move: something vast revealed by something small. A grain holds a world, a flower a heaven, a palm infinity, an hour eternity. That move, the whole inferred from the detail, is the problem I care about, stated as an image instead of an objective function.\u003c/p\u003e","title":"A world in a grain of sand"},{"content":"The Legendary Moonlight Sculptor is still one of my favorite novels, and Royal Road, the virtual world inside it, is most of the reason. I read it long before I had any of the vocabulary in this garden, and I just thought the world was the coolest thing I had ever encountered.\nWhat hooked me was the freedom. Royal Road is famous in the story for an almost reckless amount of it, openness pushed nearly to the point of chaos. You aren\u0026rsquo;t handed a class and a quest log and told what to do. Weed plays a Sculptor, the class everyone laughs at, stumbles into the hidden Legendary Moonlight Sculptor class, and turns art into a form of power. The world let him matter in a way no designer had planned for.\nAnd it felt alive. The NPCs in Royal Road aren\u0026rsquo;t backdrop; they grow, remember, form relationships, become characters you actually care about. What you do leaves marks that last, all the way up to reshaping kingdoms.\nYears later I notice the slightly embarrassing thing: the reasons I loved Royal Road as a teenager are the exact properties I now study. The freedom is When discovering the action space is the game. The living NPCs are why I keep insisting on A shared channel is not a shared world. The lasting consequences presuppose What even is a world state?. I came to the research through the novel, not the other way around.\n","permalink":"https://quangminhdinh.github.io/garden/why-royal-road-fascinates-me/","summary":"\u003cp\u003e\u003cem\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Legendary_Moonlight_Sculptor\"\u003eThe Legendary Moonlight Sculptor\u003c/a\u003e\u003c/em\u003e is still one of my favorite novels, and \u003ca href=\"https://the-legendary-moonlight-sculptor.fandom.com/wiki/Royal_Road\"\u003eRoyal Road\u003c/a\u003e, the virtual world inside it, is most of the reason. I read it long before I had any of the vocabulary in this garden, and I just thought the world was the coolest thing I had ever encountered.\u003c/p\u003e\n\u003cp\u003eWhat hooked me was the freedom. Royal Road is famous in the story for an almost reckless amount of it, openness pushed nearly to the point of chaos. You aren\u0026rsquo;t handed a class and a quest log and told what to do. Weed plays a Sculptor, the class everyone laughs at, stumbles into the hidden Legendary Moonlight Sculptor class, and turns art into a form of power. The world let him matter in a way no designer had planned for.\u003c/p\u003e","title":"Royal Road, before I had the words"},{"content":"The thing that makes Royal Road feel bottomless isn\u0026rsquo;t the amount of content. It\u0026rsquo;s emergence: the best moments are the ones nobody wrote.\nIn an ordinary game you talk to an NPC by picking from a few pre-written lines and walk a fixed path the designers laid down ahead of you. Royal Road\u0026rsquo;s whole appeal is the opposite. Players keep finding solutions, stories, and roles the designers never anticipated. Weed turning a mocked Sculptor class into a legend is the headline version, but the same thing happens at every scale.\nThere\u0026rsquo;s real research on why this works. Peng et al. (2024) call it player-driven emergence: when NPC behaviour is non-deterministic instead of scripted, players discover routes to a goal that were never in the original narrative. The freedom isn\u0026rsquo;t a longer menu, it\u0026rsquo;s that the menu stops existing.\nWhat I take from it is that emergence needs two things at once. An open space of actions, the part everyone notices, and a world rich enough that an unplanned action lands as lasting consequence and other agents respond to it, the part people skip. Weed\u0026rsquo;s sculptures literally change the world, buffing a region or weakening its monsters, and everyone then lives in the changed version. When discovering the action space is the game supplies the first; What even is a world state? is the second and harder half.\nWhich is why you can\u0026rsquo;t author emergence directly. You build the substrate, and emergence is the thing that falls out. Back to World models as shared substrate.\n","permalink":"https://quangminhdinh.github.io/garden/stories-nobody-wrote/","summary":"\u003cp\u003eThe thing that makes \u003ca href=\"https://the-legendary-moonlight-sculptor.fandom.com/wiki/Royal_Road\"\u003eRoyal Road\u003c/a\u003e feel bottomless isn\u0026rsquo;t the amount of content. It\u0026rsquo;s emergence: the best moments are the ones nobody wrote.\u003c/p\u003e\n\u003cp\u003eIn an ordinary game you talk to an NPC by picking from a few pre-written lines and walk a fixed path the designers laid down ahead of you. Royal Road\u0026rsquo;s whole appeal is the opposite. Players keep finding solutions, stories, and roles the designers never anticipated. Weed turning a mocked Sculptor class into a legend is the headline version, but the same thing happens at every scale.\u003c/p\u003e","title":"The best stories in Royal Road were never written"},{"content":"Strip away the fandom and read Royal Road as a spec. Why is it great, and what would building it actually require?\nDecomposed, the magic is four properties stacked: an open, discoverable action space; a world that absorbs your actions as lasting consequence; NPCs with their own memory and goals; and emergence, which is just what you get when the first three hold at once.\nThe honest status of each, in current terms:\nGenerating the world is the closest. Shams et al. (2023), the Holodeck-style Infinitia, have LLMs and image models generate maps, quests, NPCs, and mechanics for an open-ended world from prompts alone. Local emergence works. Peng et al. (2024) get player-driven, unscripted paths out of non-deterministic LLM NPCs inside a single mystery. Small societies work. AI Town and the generative-agents line get agents to coordinate, with the ceiling I describe in A shared channel is not a shared world. The gap is that nobody has these at the same time, over one persistent world, at scale. We can generate a world, or simulate a few dozen agents, or get one emergent storyline, but not all of it in a single shared substrate that thousands of agents and players alter together and that remembers what they did.\nSo \u0026ldquo;how to make it\u0026rdquo; isn\u0026rsquo;t a content problem, it\u0026rsquo;s a representation problem: What even is a world state? and When discovering the action space is the game done well enough that World models as shared substrate can carry a whole society. That\u0026rsquo;s the part I think is Why now.\n","permalink":"https://quangminhdinh.github.io/garden/building-royal-road/","summary":"\u003cp\u003eStrip away the fandom and read \u003ca href=\"https://the-legendary-moonlight-sculptor.fandom.com/wiki/Royal_Road\"\u003eRoyal Road\u003c/a\u003e as a spec. Why is it great, and what would building it actually require?\u003c/p\u003e\n\u003cp\u003eDecomposed, the magic is four properties stacked: an open, discoverable action space; a world that absorbs your actions as lasting consequence; NPCs with their own memory and goals; and emergence, which is just what you get when the first three hold at once.\u003c/p\u003e\n\u003cp\u003eThe honest status of each, in current terms:\u003c/p\u003e","title":"What would it actually take to build Royal Road?"},{"content":"Jesse Schell makes an observation in The Art of Game Design: A Book of Lenses that I keep coming back to for reasons that have nothing to do with games.\nEarly text adventures supported hundreds of verbs, and part of playing was discovering which actions even existed. You found puzzles by guessing that you could \u0026ldquo;spin the fish\u0026rdquo; or \u0026ldquo;tickle the monkey.\u0026rdquo; Modern visual games shrank that down to a small, fully known action set, because you can\u0026rsquo;t render an arbitrary verb. Schell\u0026rsquo;s twist is that the old openness was partly an illusion: for every verb the parser knew, there were thousands it didn\u0026rsquo;t, and that mismatch was its own kind of frustration.\nWhat I take from it: this is really about action spaces, and the trade has no free side. A large, open, discoverable action space is expressive but unbounded, hard to learn, and quietly fake at the edges. A small closed one is legible but caps what can emerge.\nFor World models as shared substrate the interesting move is to refuse both. Instead of enumerating a verb list, can an agent ground its affordances in the world, so the action space is discovered from what the environment supports rather than handed over as a fixed menu? That reframes Grounding signals beyond language: agents need to discover what they can do to each other, not just what they can say.\n","permalink":"https://quangminhdinh.github.io/garden/action-space-as-discovery/","summary":"\u003cp\u003eJesse Schell makes an observation in \u003cem\u003e\u003ca href=\"https://en.wikipedia.org/wiki/The_Art_of_Game_Design:_A_Book_of_Lenses\"\u003eThe Art of Game Design: A Book of Lenses\u003c/a\u003e\u003c/em\u003e that I keep coming back to for reasons that have nothing to do with games.\u003c/p\u003e\n\u003cp\u003eEarly text adventures supported hundreds of verbs, and part of playing was discovering which actions even existed. You found puzzles by guessing that you could \u0026ldquo;spin the fish\u0026rdquo; or \u0026ldquo;tickle the monkey.\u0026rdquo; Modern visual games shrank that down to a small, fully known action set, because you can\u0026rsquo;t render an arbitrary verb. Schell\u0026rsquo;s twist is that the old openness was partly an illusion: for every verb the parser knew, there were thousands it didn\u0026rsquo;t, and that mismatch was its own kind of frustration.\u003c/p\u003e","title":"When discovering the action space is the game"},{"content":"I did not expect a course project on mobile development to restate my research agenda, but GeoMon did. The hardest part of the game was the real-time PvP duels: getting two phones to agree on whose turn it was and what had happened, without the turn order ever desyncing. The fix was a single authoritative BattleState in Firebase, applied optimistically on each device and reconciled through listeners. Ad-hoc message passing between the two clients never held together; a shared source of truth did.\nThat is the World models as shared substrate thesis in miniature. Two agents cannot coordinate on what they do not jointly represent. In a turn-based game the shared state is small and discrete, so the engineering is tractable. For social agents the shared state is rich, continuous, and partly about each other\u0026rsquo;s intentions, which is exactly why Grounding signals beyond language is hard: the substrate they would need to agree on does not fit in a BattleState object.\nWhat stays with me is that the failure mode is identical at both scales. When two clients disagree, you get desync. When two agents lack a common model, you get behavior that only imitates coordination. The open question I like is what the BattleState has to become once the world stops being a game.\n","permalink":"https://quangminhdinh.github.io/garden/shared-state-is-a-tiny-world-model/","summary":"\u003cp\u003eI did not expect a course project on mobile development to restate my research agenda, but \u003ca href=\"/projects/geomon/\"\u003eGeoMon\u003c/a\u003e did. The hardest part of the game was the real-time PvP duels: getting two phones to agree on whose turn it was and what had happened, without the turn order ever desyncing. The fix was a single authoritative \u003ccode\u003eBattleState\u003c/code\u003e in Firebase, applied optimistically on each device and reconciled through listeners. Ad-hoc message passing between the two clients never held together; a shared source of truth did.\u003c/p\u003e","title":"Shared state is a tiny world model"},{"content":"A model optimizes exactly what you wrote down, including the blind spots you wrote in by accident.\nIn Dynamic Radiant Foam, the model leaned on color gradients to learn temporal correspondence, so it poured its capacity into bright, high-intensity regions and left large parts of the scene blank. Nothing in the architecture said \u0026ldquo;ignore the dim areas.\u0026rdquo; The L1 objective did, implicitly. Adding an SSIM term, which scores local structure rather than per-pixel intensity, made it reconstruct the whole frame uniformly and converge far faster (iteration ~500 instead of 12,000–16,000).\nIn Text-Conditioned IMLE, the CLIP alignment loss was a cliff, not a dial: gentle and paired with L2 it helped, but turned up it craters quality. Classifier-free guidance behaved the same way, collapsing the model entirely when its dropout rate dropped from 0.3 to 0.1.\nThe seedling thought: a loss is an attention-allocation policy in disguise. When a model \u0026ldquo;fails to learn\u0026rdquo; some region or some mode, my first question now is what the objective is implicitly telling it to neglect, before I blame capacity. That connects to The bottleneck is rarely the model, and to the deeper point that every such choice is a hidden assumption about what matters: Every conditioning choice is a hidden assumption.\n","permalink":"https://quangminhdinh.github.io/garden/the-loss-decides-what-the-model-ignores/","summary":"\u003cp\u003eA model optimizes exactly what you wrote down, including the blind spots you wrote in by accident.\u003c/p\u003e\n\u003cp\u003eIn \u003ca href=\"/projects/dynamic-radiant-foam/\"\u003eDynamic Radiant Foam\u003c/a\u003e, the model leaned on color gradients to learn temporal correspondence, so it poured its capacity into bright, high-intensity regions and left large parts of the scene blank. Nothing in the architecture said \u0026ldquo;ignore the dim areas.\u0026rdquo; The L1 objective did, implicitly. Adding an SSIM term, which scores local structure rather than per-pixel intensity, made it reconstruct the whole frame uniformly and converge far faster (iteration ~500 instead of 12,000–16,000).\u003c/p\u003e","title":"The loss decides what the model ignores"},{"content":"PedroVerse stylized hyper-realistic 3D assets into non-photorealistic looks without writing a single engine-specific shader. The trick was to operate only on the asset\u0026rsquo;s UV maps, the albedo and object-space normal maps that almost every asset already ships with and that render cheaply on every engine and in VR. A shader built in Blender does not export; a post-process effect does not generalize; but a restyled UV map travels anywhere.\nThe decision that mattered was not an algorithm, it was choosing the representation that was already universal and defining every operation in that space. Portability then came for free, because the substrate was shared to begin with.\nThat choice has a sharp edge, though. The normal map encodes physics: shift its colors carelessly and you change the direction light bounces, so stylizing it without breaking lighting forced most of the real engineering (gradient-aware brush strokes, low SLIC compactness, a UV-space mask). Editing a representation that encodes how the world behaves is harder than editing one that only encodes how it looks.\nI keep coming back to this when I think about World models as shared substrate: a representation many agents can share is worth more than a cleverer one only you can read, and the universal one is often already sitting in the data, the engineering cousin of Read the data before you add machinery.\n","permalink":"https://quangminhdinh.github.io/garden/build-on-the-representation-that-already-travels/","summary":"\u003cp\u003e\u003ca href=\"/projects/pedroverse/\"\u003ePedroVerse\u003c/a\u003e stylized hyper-realistic 3D assets into non-photorealistic looks without writing a single engine-specific shader. The trick was to operate only on the asset\u0026rsquo;s UV maps, the albedo and object-space normal maps that almost every asset already ships with and that render cheaply on every engine and in VR. A shader built in Blender does not export; a post-process effect does not generalize; but a restyled UV map travels anywhere.\u003c/p\u003e\n\u003cp\u003eThe decision that mattered was not an algorithm, it was choosing the representation that was already universal and defining every operation in that space. Portability then came for free, because the substrate was shared to begin with.\u003c/p\u003e","title":"Build on the representation that already travels"},{"content":"The best feature I built for SNDA cost nothing to collect. Many of the retrieval queries described two nearby vehicles, not one, so I mined that second vehicle\u0026rsquo;s type and color as supervision the annotations were already handing me for free. Pairing type and color across the two vehicles was worth more than throwing every attribute at the model. In fact using all of the second vehicle\u0026rsquo;s attributes hurt, because size and motion were too sparse to be reliable.\nThe same project had a second instance of this. Every camera was static, so the dataset quietly handed me a way to encode motion without any temporal model at all. That shortcut is also a Cheap structure beats heavy machinery story.\nWhat I take from this is a sequencing rule. Before designing a loss or reaching for a bigger backbone, spend real time looking at the raw data: word frequencies, what co-occurs, what the sensor setup makes constant. The structure you find there tends to beat the structure you impose, because it matches the actual distribution instead of your assumptions about it. The flip side is that quirks found this way can be split-specific, so anything you mine still has to survive The test distribution is the only one that counts.\n","permalink":"https://quangminhdinh.github.io/garden/read-the-data-first/","summary":"\u003cp\u003eThe best feature I built for \u003ca href=\"/projects/symmetric-vehicle-retrieval/\"\u003eSNDA\u003c/a\u003e cost nothing to collect. Many of the retrieval queries described two nearby vehicles, not one, so I mined that second vehicle\u0026rsquo;s type and color as supervision the annotations were already handing me for free. Pairing \u003ccode\u003etype\u003c/code\u003e and \u003ccode\u003ecolor\u003c/code\u003e across the two vehicles was worth more than throwing every attribute at the model. In fact using \u003cem\u003eall\u003c/em\u003e of the second vehicle\u0026rsquo;s attributes hurt, because \u003ccode\u003esize\u003c/code\u003e and \u003ccode\u003emotion\u003c/code\u003e were too sparse to be reliable.\u003c/p\u003e","title":"Read the data before you add machinery"},{"content":"A standing temptation in ML work is to reach for the heaviest tool available. The opposite has paid off more often for me.\nIn SNDA I skipped the video transformer. Every camera in the dataset was static, so I collapsed each track into a single motion map: average the frames into a clean background, then paste the cropped vehicle from each frame back on top. Cheap structure carried most of the result, on one GPU.\nIn MirrorBrain I replaced Qdrant vector search with Okapi BM25 on the text-retrieval path. For a note-sized corpus, lexical BM25 was faster, good enough, and it removed an entire moving part from the hot path.\nIn Dynamic Radiant Foam, every pruning scheme I tried made training worse, so I fell back to plain densification that just fills large empty cells.\nThe thread is not that simple is virtuous. It is that the impressive method usually buys a capability I do not yet need, at a cost I pay immediately: a transformer\u0026rsquo;s data hunger, a vector DB\u0026rsquo;s operational surface, pruning\u0026rsquo;s instability. The boring choice clears the bar and keeps the system legible, and legibility is what lets me keep moving. It is the same observation as The bottleneck is rarely the model, seen from the solution side. The natural precursor is to check whether the data already hands you the structure for free, which is Read the data before you add machinery.\n","permalink":"https://quangminhdinh.github.io/garden/cheap-structure-beats-heavy-machinery/","summary":"\u003cp\u003eA standing temptation in ML work is to reach for the heaviest tool available. The opposite has paid off more often for me.\u003c/p\u003e\n\u003cp\u003eIn \u003ca href=\"/projects/symmetric-vehicle-retrieval/\"\u003eSNDA\u003c/a\u003e I skipped the video transformer. Every camera in the dataset was static, so I collapsed each track into a single motion map: average the frames into a clean background, then paste the cropped vehicle from each frame back on top. Cheap structure carried most of the result, on one GPU.\u003c/p\u003e","title":"Cheap structure beats heavy machinery"},{"content":"In SNDA, my first large-scale research project, the natural-language augmentation that lifted validation MRR to 0.58 actively hurt on the test set. The model had memorized validation-specific quirks. Only ensembling and post-processing recovered the 0.35 that placed 7th. The gap between 0.58 and 0.35 was almost entirely a domain-generalization failure, and it is the lesson that has stuck hardest.\nI have since seen the same shape elsewhere. The diversity collapse behind Every conditioning choice is a hidden assumption was a generalization failure too: a conditional model with the best FID and precision returned near-duplicate samples, perfect by the headline metric and useless at the thing I cared about. And when I reproduced an emotion classifier in Face Emotion Detection, the entire accuracy gap to the paper traced back to one skipped super-resolution step, a reminder that a reported number lives somewhere specific, and usually not where the abstract points.\nThe discipline I took from this: treat every validation gain as a hypothesis about the test distribution, not a result. Ask what specifically the gain might be exploiting, and assume anything that helps only on the split you tuned on is a liability until the held-out distribution agrees. Reading the data closely is part of the same habit, but the structure you find there has to survive this test too: Read the data before you add machinery.\n","permalink":"https://quangminhdinh.github.io/garden/the-test-distribution-is-the-only-one-that-counts/","summary":"\u003cp\u003eIn \u003ca href=\"/projects/symmetric-vehicle-retrieval/\"\u003eSNDA\u003c/a\u003e, my first large-scale research project, the natural-language augmentation that lifted validation MRR to 0.58 actively \u003cem\u003ehurt\u003c/em\u003e on the test set. The model had memorized validation-specific quirks. Only ensembling and post-processing recovered the 0.35 that placed 7th. The gap between 0.58 and 0.35 was almost entirely a domain-generalization failure, and it is the lesson that has stuck hardest.\u003c/p\u003e\n\u003cp\u003eI have since seen the same shape elsewhere. The diversity collapse behind \u003ca class=\"wikilink\" href=\"/garden/conditioning-is-an-assumption/\"\u003eEvery conditioning choice is a hidden assumption\u003c/a\u003e was a generalization failure too: a conditional model with the best FID and precision returned near-duplicate samples, perfect by the headline metric and useless at the thing I cared about. And when I reproduced an emotion classifier in \u003ca href=\"/projects/mediapipe-face-emotion/\"\u003eFace Emotion Detection\u003c/a\u003e, the entire accuracy gap to the paper traced back to one skipped super-resolution step, a reminder that a reported number lives somewhere specific, and usually not where the abstract points.\u003c/p\u003e","title":"The test distribution is the only one that counts"},{"content":"I keep running into the same surprise: I sit down to make a model better, and the lever that actually moves the result is somewhere else.\nIn Text-Conditioned IMLE, the whole study was gated by nearest-neighbor matching. Until FAISS and a fused distance made it cheap, no conditioning experiment was runnable at all. Then, the moment it was fast, the matching stopped being where the difficulty lived.\nIn Dynamic Radiant Foam, multi-view training never finished, and not because the motion model was wrong. Re-triangulating the Voronoi mesh every step, plus the quiet assumption that the whole video fits in memory, is what ran out the clock.\nIn Virtual Ring Try-On, the jump from 5 to 30 FPS came from stabilization and deleting frame conversions, not from a better hand tracker.\nThe model is usually the part I understand best, so it draws my attention, while the constraint that actually caps me sits in the plumbing around it: the matching, the data structure, the latency budget. So I now try to find the binding constraint before touching the network. Often the boring fix is the big one, which is its own note: Cheap structure beats heavy machinery. And when the model genuinely is the problem, it is usually the representation that is wrong rather than the capacity, the point behind Every conditioning choice is a hidden assumption.\n","permalink":"https://quangminhdinh.github.io/garden/bottleneck-is-rarely-the-model/","summary":"\u003cp\u003eI keep running into the same surprise: I sit down to make a model better, and the lever that actually moves the result is somewhere else.\u003c/p\u003e\n\u003cp\u003eIn \u003ca href=\"/projects/text-conditioned-imle/\"\u003eText-Conditioned IMLE\u003c/a\u003e, the whole study was gated by nearest-neighbor matching. Until FAISS and a fused distance made it cheap, no conditioning experiment was runnable at all. Then, the moment it was fast, the matching stopped being where the difficulty lived.\u003c/p\u003e\n\u003cp\u003eIn \u003ca href=\"/projects/dynamic-radiant-foam/\"\u003eDynamic Radiant Foam\u003c/a\u003e, multi-view training never finished, and not because the motion model was wrong. Re-triangulating the Voronoi mesh every step, plus the quiet assumption that the whole video fits in memory, is what ran out the clock.\u003c/p\u003e","title":"The bottleneck is rarely the model"},{"content":"Recent work on world models can already generate photorealistic streams in which a single agent wanders freely. But multi-agent social interaction is still clumsy. Scripts paper over the gap; agents don\u0026rsquo;t feel like they share a world.\nMy claim: the gap isn\u0026rsquo;t in the policy, it\u0026rsquo;s in the representation. If two agents don\u0026rsquo;t have a common substrate describing what is happening and what might happen next, they can\u0026rsquo;t coordinate. They can only imitate coordination.\nThe Reachy pilot study: what 38% means is one concrete example of this in miniature. Even with good perception, our VLM picked generic expressions because it lacked a persistent scene model of the human it was engaging with.\nRelated: Grounding signals beyond language, and a much smaller instance of the same failure in Shared state is a tiny world model.\n","permalink":"https://quangminhdinh.github.io/garden/world-models/","summary":"\u003cp\u003eRecent work on world models can already generate photorealistic streams in which a single agent wanders freely. But \u003cem\u003emulti-agent\u003c/em\u003e social interaction is still clumsy. Scripts paper over the gap; agents don\u0026rsquo;t feel like they share a world.\u003c/p\u003e\n\u003cp\u003eMy claim: the gap isn\u0026rsquo;t in the policy, it\u0026rsquo;s in the representation. If two agents don\u0026rsquo;t have a common substrate describing \u003cem\u003ewhat is happening and what might happen next\u003c/em\u003e, they can\u0026rsquo;t coordinate. They can only imitate coordination.\u003c/p\u003e","title":"World models as shared substrate"},{"content":"In the pilot study for our Reachy Mini interaction system, only 38% of participants could distinguish autonomous behaviour from human teleoperation. The knee-jerk reaction is to celebrate the 62% fooling rate.\nThe interesting data is in how participants guessed. Many of them assumed autonomous behaviour would feel more lifelike, and that teleoperation would feel repetitive and constrained. The exact inverse of what our system produced: our autonomous VLM tended to select generic, repetitive expressions.\nThe VLM recognized the action in each keyframe. What it couldn\u0026rsquo;t do was infer what the participant wanted next, or chain behaviour across turns. No persistent scene model, no anticipation.\nThis is the micro-version of the argument in World models as shared substrate: you can\u0026rsquo;t pick a good response if you don\u0026rsquo;t have a shared model of the thing you\u0026rsquo;re responding to.\n","permalink":"https://quangminhdinh.github.io/garden/reachy-pilot-study/","summary":"\u003cp\u003eIn the pilot study for our Reachy Mini interaction system, only 38% of participants could distinguish autonomous behaviour from human teleoperation. The knee-jerk reaction is to celebrate the 62% fooling rate.\u003c/p\u003e\n\u003cp\u003eThe interesting data is in \u003cem\u003ehow\u003c/em\u003e participants guessed. Many of them assumed autonomous behaviour would feel \u003cstrong\u003emore\u003c/strong\u003e lifelike, and that teleoperation would feel \u003cstrong\u003erepetitive and constrained\u003c/strong\u003e. The exact inverse of what our system produced: our autonomous VLM tended to select generic, repetitive expressions.\u003c/p\u003e","title":"Reachy pilot study: what 38% means"},{"content":"Natural language is a powerful interface, but it\u0026rsquo;s a lossy projection of what agents need to coordinate on. Intentions, timing, attention, and affect all matter, and most of them don\u0026rsquo;t survive the text bottleneck.\nOpen question: what does a grounding signal look like if we stop privileging text? Some candidates I want to think through:\nEmbodied state traces: position, velocity, gaze direction as first-class citizens. Attention as a signal: what an agent isn\u0026rsquo;t looking at is as informative as what it is. Counterfactual rollouts: a short imagined future as part of the message. Prof. Freda Shi\u0026rsquo;s work on how grounding actually emerges inside model computations is a useful empirical handle on this: it suggests the signal we need may already be latent in models we have, if we knew where to look.\nSee also: Every conditioning choice is a hidden assumption.\n","permalink":"https://quangminhdinh.github.io/garden/multi-agent-grounding/","summary":"\u003cp\u003eNatural language is a powerful interface, but it\u0026rsquo;s a lossy projection of what agents need to coordinate on. Intentions, timing, attention, and affect all matter, and most of them don\u0026rsquo;t survive the text bottleneck.\u003c/p\u003e\n\u003cp\u003eOpen question: what does a grounding signal look like if we stop privileging text? Some candidates I want to think through:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eEmbodied state traces\u003c/strong\u003e: position, velocity, gaze direction as first-class citizens.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAttention as a signal\u003c/strong\u003e: what an agent \u003cem\u003eisn\u0026rsquo;t\u003c/em\u003e looking at is as informative as what it is.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCounterfactual rollouts\u003c/strong\u003e: a short imagined future as part of the message.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eProf. Freda Shi\u0026rsquo;s work on how grounding actually emerges inside model computations is a useful empirical handle on this: it suggests the signal we need may already be latent in models we have, if we knew where to look.\u003c/p\u003e","title":"Grounding signals beyond language"},{"content":"I didn\u0026rsquo;t come into research wanting to work on world models. I came into research because, at Huawei, a problem I knew nothing about (acoustic echo cancellation on sub-200ms frames) turned out to have a beautiful fix borrowed from classical signal processing, and nobody had tried it. That was enough to convince me research was a thing I could do, and wanted to keep doing.\nThen the AI City Challenge happened, and I noticed our caption model couldn\u0026rsquo;t keep two agents\u0026rsquo; descriptions from conflicting. That pushed me toward World models as shared substrate.\nThen IMLE happened, and I learned that Every conditioning choice is a hidden assumption. That pushed me toward caring about how we encode worlds, not just generate them.\nThen the Reachy Mini pilot happened. That showed me the whole argument in a single Wizard-of-Oz setup.\nBy the time I was writing the SoP, the question had stopped being whether to do a PhD and started being which lab. That\u0026rsquo;s roughly when you know.\n","permalink":"https://quangminhdinh.github.io/garden/phd-motivation/","summary":"\u003cp\u003eI didn\u0026rsquo;t come into research wanting to work on world models. I came into research because, at Huawei, a problem I knew nothing about (acoustic echo cancellation on sub-200ms frames) turned out to have a beautiful fix borrowed from classical signal processing, and nobody had tried it. That was enough to convince me research was a thing I could do, and wanted to keep doing.\u003c/p\u003e\n\u003cp\u003eThen the AI City Challenge happened, and I noticed our caption model couldn\u0026rsquo;t keep two agents\u0026rsquo; descriptions from conflicting. That pushed me toward \u003ca class=\"wikilink\" href=\"/garden/world-models/\"\u003eWorld models as shared substrate\u003c/a\u003e.\u003c/p\u003e","title":"Why now"},{"content":"At APEX Lab, I spent a summer benchmarking conditioning architectures for text-to-image IMLE: FiLM affine modulation, per-block residual injection, StyleGAN-style conditioning, mapping-network concatenation. Most of them overfit.\nEach failure traced back to a single thing: a mismatch in the latent representation between the condition space and the output space.\nConcrete example. Unit-normalising CLIP embeddings collapsed diversity and made overfitting worse. On paper this is fine: cleaner conditioning should help. In practice, normalised embeddings were overwhelmed by the latent noise drawn from the prior, so the condition lost its grip on the output.\nEvery architectural choice implies an assumption about how a conditioning signal is encoded to steer what it controls.\nThis generalizes beyond IMLE. Designing a shared world representation for multiple agents is the same problem scaled up: the encoding has to match the thing it\u0026rsquo;s trying to steer. Back to World models as shared substrate.\n","permalink":"https://quangminhdinh.github.io/garden/conditioning-is-an-assumption/","summary":"\u003cp\u003eAt APEX Lab, I spent a summer benchmarking conditioning architectures for text-to-image IMLE: FiLM affine modulation, per-block residual injection, StyleGAN-style conditioning, mapping-network concatenation. Most of them overfit.\u003c/p\u003e\n\u003cp\u003eEach failure traced back to a single thing: a mismatch in the latent representation between the condition space and the output space.\u003c/p\u003e\n\u003cp\u003eConcrete example. Unit-normalising CLIP embeddings \u003cem\u003ecollapsed diversity and made overfitting worse\u003c/em\u003e. On paper this is fine: cleaner conditioning should help. In practice, normalised embeddings were overwhelmed by the latent noise drawn from the prior, so the condition lost its grip on the output.\u003c/p\u003e","title":"Every conditioning choice is a hidden assumption"},{"content":"GeoMon is a lightweight, location-based RPG for Android: players explore real-world maps, encounter monsters spawned around their GPS location, battle and capture them, duel other players in real time, and chat with their active monster powered by an LLM. It was my team\u0026rsquo;s project for CMPT 362: Mobile Application Development, and I built the location-tracking service, the real-time PvP duel system, and the Firebase backend (player and monster state) and authentication, plus the player\u0026rsquo;s directional sprites and a share of the monster spawning.\nDemos The full project presentation, which walks through the design end to end:\nAnd the two show-and-tell demos:\nThe live map: players roam the real world, encounter monsters that spawn around them, and see other online players nearby. What it does Location-based map. A GPS-backed location service tracks the player and streams updates to the UI through a bound service and message handler. Monster spawning. If fewer than 10 monsters exist within a radius of the player, the system spawns new ones with random species. As you move, fresh monsters appear, and all wild monsters are synchronised across players through the Firebase real-time database. Capture \u0026amp; battle. Tap a monster to fight; the first full-HP monster in your Pokedex leads. Capture probability scales with the target\u0026rsquo;s remaining HP, so you whittle it down before throwing. Real-time PvP duels. One player sends a DuelRequest; on accept, both devices launch the battle and share a single BattleState object in Firebase. Each move is applied locally first, then written to BattleState; the turn field flips and the opponent\u0026rsquo;s listener replays the change. A heartbeat of timestamps marks players online so nearby duelists show up on the map. AI companion chat. Players talk to their active monster through the Gemini API, with all network calls isolated on the IO thread. Architecture The app leans hard on Android\u0026rsquo;s threading and persistence primitives. The UI thread owns the map, markers, and battle screens; a Room database caches parsed species/skill/item data so monsters can be instantiated anywhere without re-parsing JSON; and a Monster data class acts as the bridge between Firebase\u0026rsquo;s remote representation and the local objects (mapping sprites from stored URIs along the way). Authentication uses Firebase anonymous auth per device, and a user object in the realtime DB holds each player\u0026rsquo;s monster IDs: any monster without an owner ID is, by definition, a wild monster.\nThe MVVM + threading design: UI thread, IO/Room persistence, Firebase worker pools, and the location service. Building it The work split cleanly across five of us: Brandon built the turn-based battle engine, the JSON stats repository, and the inventory and stat screens; Long handled monster spawning, the map merge, sprites, the Gemini chatbot, and the profile settings; Yizhang built the main UI shell. My half was the systems layer underneath all of that, the location-tracking service, the Firebase data model, authentication, and the PvP duels, which also made me the person constantly merging everyone\u0026rsquo;s branches and resolving the conflicts where our pieces met.\nThe single biggest structural decision was refactoring from a Room-only design to Firebase + Room once we committed to multiplayer. Local persistence alone could never let two phones agree on the world, so Room became a cache for static reference data while Firebase became the source of truth for anything live, players, monster state, and duels.\nAt our mid-project show-and-tell the UI wasn\u0026rsquo;t ready yet, so I demoed everything on a deliberately bare map of plain markers and faked a second player by running the app on my own second phone. That second device turned out to be the most useful debugging tool of the project: real-time multiplayer bugs only surface when two clients disagree, and the only honest way to see that is to hold both in your hands. The location service started as a textbook bound-service-plus-message-handler setup, and by the final build I had pulled that data flow behind a TrackingViewModel so the rest of the app never needed to know where a location update came from.\nThe duels were the part I am proudest of and the part that fought back hardest: getting two phones to share one authoritative BattleState, apply moves optimistically, and reconcile through listeners without the turn order ever desyncing took far more care than the battle maths itself. The unglamorous lessons stuck just as well, branch per feature and merge deliberately, write down how a feature works the moment you push it, and never, ever commit an API key.\nTakeaways Real-time multiplayer is mostly a state-synchronisation problem. A single shared BattleState with listener-driven turns is far simpler to reason about than ad-hoc message passing. \u0026ldquo;Optimistic local, then sync\u0026rdquo; keeps duels feeling instant even with network latency: apply the move locally, write to Firebase, let listeners reconcile. Caching reference data in Room removes a surprising amount of friction: monsters can be spawned anywhere in the app without touching the JSON seed again. ","permalink":"https://quangminhdinh.github.io/projects/geomon/","summary":"\u003cp\u003e\u003cstrong\u003eGeoMon\u003c/strong\u003e is a lightweight, location-based RPG for Android: players explore real-world maps, encounter monsters spawned around their GPS location, battle and capture them, duel other players in real time, and chat with their active monster powered by an LLM. It was my team\u0026rsquo;s project for \u003ca href=\"https://www.sfu.ca/outlines.html?2025/fall/cmpt/362/d100\"\u003e\u003cstrong\u003eCMPT 362: Mobile Application Development\u003c/strong\u003e\u003c/a\u003e, and I built the location-tracking service, the real-time PvP duel system, and the Firebase backend (player and monster state) and authentication, plus the player\u0026rsquo;s directional sprites and a share of the monster spawning.\u003c/p\u003e","title":"GeoMon"},{"content":"At APEX Lab (with Ke Li), I worked on extending IMLE (Implicit Maximum Likelihood Estimation) from unconditional image synthesis into the text-conditional setting, turning Adaptive IMLE into a testbed for one question: how should a text prompt steer a generator, and why does conditioning so often quietly kill sample diversity? Across more than a hundred runs on Oxford 102 Flowers, CelebA, and ImageNet, the answer kept pointing back at the same tension.\nSamples from the conditional IMLE pipeline on CelebA. IMLE's coverage-first objective keeps the gallery varied (pose, age, lighting, accessories) instead of collapsing onto a few prototypes. Why IMLE, and why in the small-data regime GANs maximise a likelihood through an adversarial game, which makes them prone to mode collapse, especially when data is scarce. Diffusion models need a lot of data and sampling steps to behave. IMLE flips the objective around: instead of pushing generated samples toward the data, it pulls every real sample toward a nearby generated one, which directly optimises for mode coverage:\n$$ \\min_G\\ \\mathbb{E}_{x\\sim p_{\\text{data}}}\\Big[\\min_{z}\\ \\|x-G(z)\\|^2\\Big] $$That difference matters most when training data is limited, which is exactly where the Adaptive IMLE line of work this project builds on shows its edge: GANs and diffusion degrade into artefacts and broken faces, while Adaptive IMLE stays sharp and diverse.\nThe limited-data regime, GAN baselines: FakeCLR, MixDL, and FastGAN show repeated structure and artefacts. (Adaptive IMLE foundation, APEX Lab.) Same regime, diffusion vs IMLE: EDM produces malformed faces with few samples, vanilla IMLE is faithful but soft, and Adaptive IMLE recovers sharpness without sacrificing coverage. This unconditional method is the foundation I extended. In the conditional version, the generator becomes $G(z,c)$ and we model $p(x\\mid c)$, where the condition $c$ is a text embedding (a CLIP encoding of a caption). The objective keeps its shape but now matches within each condition:\n$$ \\min_G\\ \\mathbb{E}_{(x,c)}\\Big[\\min_{z}\\ \\|x-G(z,c)\\|^2\\Big] $$Making the matching cheap Every IMLE step solves, for each real sample $x_i$, a nearest-neighbour search over the generated pool:\n$$ j^* = \\arg\\min_j\\ d\\big(x_i,\\, G(z_j)\\big) $$This is naively $O(NM)$ in the number of real samples $N$ and latent draws $M$, and it dominates training cost. Before any conditioning experiments were possible, the matching had to be fast, so the first block of work was infrastructure:\nFAISS nearest-neighbour search replacing the original MDCI routine, giving GPU-accelerated approximate search that scales with batch size, latent count, and embedding dimension. A fused distance computation that expands the squared Euclidean distance into a single matrix multiply, the form GPUs and FAISS handle most efficiently: $$ \\|a-b\\|^2 = \\|a\\|^2 + \\|b\\|^2 - 2\\,a^\\top b $$ A revised latent resampling / assignment schedule (the force_resample cadence): how often to recompute matches versus reuse cached ones, trading raw compute against fresh assignments and training stability. Resampling every 2 epochs turned out to be a good default, slightly improving FID and precision at a small recall cost. The matching is also non-differentiable, so IMLE deliberately separates the assignment step from the generator-optimisation step. The whole investigation rides on this separation working.\nHow should a prompt steer the generator? With matching cheap, the core question became architectural: where and how do you inject the text embedding so the prompt actually controls the output? I built each variant as its own commit (the history reads like a changelog) and benchmarked them on Oxford 102 Flowers at a reduced latent count $k{=}9$ for speed. Metrics are FID (lower is better) and the improved precision / recall pair, where precision tracks fidelity and recall tracks diversity, all read at a matched training budget.\nConditioning architecture FID ↓ Precision ↑ Recall ↑ Residual injection (add text to latent) 24.62 0.954 0.552 Concatenation + linear downsample 24.75 0.929 0.603 FiLM (text-modulated affine) 22.95 0.933 0.611 FiLM + light CLIP + L2 (best) 21.58 0.942 0.599 Conditional StyleGAN (noise injection) 275.5 0.293 0.000 Unconditional reference 40.40 0.919 0.609 Conditioning architectures on Oxford 102 Flowers (k = 9), at a matched training budget. FiLM leads; conditional-StyleGAN noise injection collapses entirely.\nTwo results stood out. FiLM, where the text embedding modulates per-channel affine transforms, matched or beat every other injection scheme: the prompt gets to gate features rather than fight the latent for the same channels. At the opposite end, the conditional-StyleGAN style of injecting noisy text into the mapping network collapsed completely (FID near 275, recall zero), either fundamentally mismatched here or needing far more training than the budget allowed.\nFiLM-conditioned samples on Oxford 102 Flowers (logged to W\u0026B). Sharp, varied blooms: the strongest of the injection schemes I tried. The unconditional reference is the quiet lesson in this table: its FID is much worse, yet its recall is healthy. Conditioning bought fidelity but spent diversity, and chasing that lost diversity back became the rest of the project.\nShaping the conditioning signal Beyond architecture, I swept the signal itself: the auxiliary losses and the representation of the text embedding.\nCLIP loss has to be gentle, and never alone. Adding a CLIP alignment loss between the prompt and the generated image helps only at a small weight, and only when paired with an L2 term; turn it up and quality craters.\nCLIP loss weight FID ↓ Precision ↑ Recall ↑ 0.1 22.73 0.930 0.555 0.5 31.05 0.936 0.485 1.0 48.26 0.944 0.454 Effect of the CLIP alignment-loss weight (Flowers, concatenation with per-block injection). Only a light CLIP term helps, and only alongside an L2 loss.\nThe text representation sets a floor on quality. Quantising captions into k-means codes makes the granularity tradeoff explicit. With only 8 codes the generator has too little to latch onto and never converges; push to 50 and then 100 codes and both fidelity and coverage recover, though pushing quantisation very fine eventually risks dropping the rarest flower types. Unit-normalising the CLIP embeddings, and randomly projecting them to low dimensions (64, 32, 5), both reliably hurt diversity too.\nText k-means codes FID ↓ Recall ↑ Behaviour 8 codes 196.2 0.009 too coarse, fails to converge 50 codes 78.2 0.102 recovering, slow 100 codes 45.6 0.203 sharp and more diverse Quantising captions into k-means codes (Flowers, FiLM). Too few codes fail to converge; richer codebooks restore both fidelity and coverage.\nQuantising captions into k-means codes (W\u0026B grids). Eight codes collapse into a muddy, low-quality set; fifty and a hundred codes restore sharpness and variety. Coarse conditioning, not the matching, is the bottleneck here. Supporting machinery underneath all of this: a text sampler, CLIP and L2-CLIP losses, k-means clustering of captions, text unit normalisation, and dataset normalisation statistics, with W\u0026amp;B image and metric logging throughout, and early exploration on ImageNet (32×32) alongside the main Flowers and CelebA runs.\nThe diversity problem, and what actually moved it The same pattern followed me from Flowers to CelebA: conditional models produced clean faces but a narrowing range of them. The clearest way to see it is to fix a prompt and draw several latents. A plain conditional model returns near-duplicates; the prompt has effectively pinned the output. The single most effective fix was not a loss or an injection trick, it was initialisation, starting the conditional generator from a strong unconditional IMLE prior and letting the text branch adapt on top of it.\nSame prompt per row, several latents each (W\u0026B grids). Top: the base conditional model returns near-identical faces, the diversity collapse in one picture. Bottom: initialising from an unconditional IMLE prior and adapting a small text branch restores real within-prompt variation. Classifier-free guidance helped too, but proved brittle to its conditioning-dropout probability $p$.\nCelebA setup FID ↓ Precision ↑ Recall ↑ Base conditional 12.94 0.995 0.668 From unconditional pretrain 18.37 0.939 0.784 Pretrain + 5 text layers 18.63 0.976 0.825 Classifier-free guidance, $p{=}0.3$ 17.59 0.979 0.695 Classifier-free guidance, $p{=}0.1$ 77.72 0.047 0.035 Restoring diversity on CelebA. Initialising from an unconditional prior lifts recall the most; classifier-free guidance helps but is brittle to the dropout rate p.\nNote the tension in the first rows: the base model has the best FID and precision but the worst within-prompt variety. Initialising from the unconditional prior trades a few FID points to lift recall from the mid-0.6s into the low-0.8s, the clearest single intervention in the whole study. CFG at $p{=}0.3$ lands in a similar place, but dropping $p$ to $0.1$ collapses the model entirely (precision 0.05): the conditioning-dropout rate is a cliff, not a slider. Smaller architectures, frozen backbones, and random horizontal flipping all underperformed.\nWhat I took away The bottleneck is the matching, then it isn\u0026rsquo;t. Making nearest-neighbour search cheap (FAISS plus fused distances) is what makes conditional IMLE feasible at all, but once it\u0026rsquo;s fast, the matching machinery is no longer where the difficulty lives. Conditioning is a representation problem, and a diversity tax. A prompt only steers generation if its embedding is compatible with the generator\u0026rsquo;s latent space (FiLM works, noisy StyleGAN injection doesn\u0026rsquo;t), and even when it works, conditioning trades coverage for fidelity unless you fight to keep it. The strongest lever was a good prior. Initialising from an unconditional IMLE model, not a fancier loss, did the most to restore diversity, with light CLIP+L2 guidance and a well-tuned CFG dropout as supporting acts. Several intuitive ideas reliably backfired: heavy CLIP loss, unit-normalised embeddings, low-dimensional projections, coarse text quantisation, and conditional-StyleGAN noise injection each cost quality, diversity, or both. Every run, metric curve, and qualitative grid is public on Weights \u0026amp; Biases (organised into per-experiment reports), and the implementation, committed feature by feature, is on GitHub.\n","permalink":"https://quangminhdinh.github.io/projects/text-conditioned-imle/","summary":"\u003cp\u003eAt \u003ca href=\"https://sfuapex.ca/\"\u003e\u003cstrong\u003eAPEX Lab\u003c/strong\u003e\u003c/a\u003e (with \u003ca href=\"https://www.sfu.ca/~keli/\"\u003eKe Li\u003c/a\u003e), I worked on extending \u003cstrong\u003eIMLE\u003c/strong\u003e (Implicit Maximum Likelihood Estimation) from unconditional image synthesis into the \u003cstrong\u003etext-conditional\u003c/strong\u003e setting, turning \u003ca href=\"https://github.com/quangminhdinh/AdaIMLE\"\u003eAdaptive IMLE\u003c/a\u003e into a testbed for one question: \u003cem\u003ehow\u003c/em\u003e should a text prompt steer a generator, and why does conditioning so often quietly kill sample diversity? Across more than a hundred runs on Oxford 102 Flowers, CelebA, and ImageNet, the answer kept pointing back at the same tension.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"demo.jpg\" alt=\"Grid of CelebA faces generated by the conditional IMLE model\"\u003e\n  \u003cfigcaption\u003eSamples from the conditional IMLE pipeline on CelebA. IMLE's coverage-first objective keeps the gallery varied (pose, age, lighting, accessories) instead of collapsing onto a few prototypes.\u003c/figcaption\u003e\n\u003c/figure\u003e\n\u003ch2 id=\"why-imle-and-why-in-the-small-data-regime\"\u003eWhy IMLE, and why in the small-data regime\u003c/h2\u003e\n\u003cp\u003eGANs maximise a likelihood through an adversarial game, which makes them prone to \u003cstrong\u003emode collapse\u003c/strong\u003e, especially when data is scarce. Diffusion models need a lot of data and sampling steps to behave. IMLE flips the objective around: instead of pushing generated samples toward the data, it pulls \u003cem\u003eevery real sample\u003c/em\u003e toward a nearby generated one, which directly optimises for \u003cstrong\u003emode coverage\u003c/strong\u003e:\u003c/p\u003e","title":"Text-Conditioned IMLE"},{"content":"Most real-world environments move, yet most high-quality radiance-field methods assume the scene stands perfectly still. For my final project in CMPT 469: Rendering and Visual Computing for AI, I set out to add a temporal axis to Radiant Foam and reconstruct dynamic scenes from monocular and multi-view video. It was an ambitious swing, and an honest one: it did not fully converge, but the failure modes were the interesting part, and each one had a satisfying remedy.\nMonocular reconstruction on a Neural 3D Video scene. Left: the model's rendering. Middle: the ground-truth frame. Right: the per-pixel L1 error map. Locally sharp, but the single viewpoint leaves the model unable to render correct novel angles. From static to moving scenes Given a monocular video or several synchronized multi-view videos, the goal of dynamic reconstruction is to learn a representation that captures how objects, people, lighting, and surfaces change over time, directly useful for animation, urban planning, and AR/VR. Existing work tends to split two ways: methods that learn a 6D plenoptic function and let a neural network absorb the motion implicitly, and methods that model motion explicitly through a deformation field or time-conditioned structure. Both directions usually trade rendering quality for speed, or speed for quality, and most can only hold a scene together for a few seconds.\nRadiant Foam is a real-time static method that sidesteps the rasterization bottleneck of 3D Gaussian Splatting by ray-tracing through a dense Voronoi tessellation: 3D space is partitioned into cells, every point belongs to exactly one cell, and the density and radiance are held constant inside each cell. A ray accumulates the contributions of the cells it cuts through. Starting from the volume-rendering integral along a ray $\\mathbf{r}$ between $q_{\\min}$ and $q_{\\max}$,\n$$ \\mathbf{c_r}=\\int_{q_{\\min}}^{q_{\\max}} T(q)\\,\\sigma(\\mathbf{r}(q))\\,\\mathbf{c}(\\mathbf{r}(q))\\,dq, \\qquad T(q)=\\exp\\!\\Big(-\\!\\int_{q_{\\min}}^{q}\\sigma(\\mathbf{r}(u))\\,du\\Big), $$the constant-per-cell assumption collapses the integral into a sum over the $M$ segments the ray spends inside successive cells:\n$$ \\mathbf{c_r}=\\sum_{m=1}^{M} T_m\\big(1-e^{-\\sigma_m\\delta_m}\\big)\\,\\mathbf{c}_m, \\qquad T_m=\\prod_{j=1}^{m} e^{-\\sigma_j\\delta_j}, $$where $\\sigma_m$ and $\\mathbf{c}_m$ are the cell\u0026rsquo;s density and radiance and $\\delta_m$ is the width of segment $m$. That gives photorealistic renderings with complex lighting at real-time speed, which makes it a tempting base for dynamic reconstruction. My project asked a simple question: can the foam move through time without giving up its speed?\nPutting time into the foam Radiant Foam stores three per-point quantities as learnable parameters: position, density, and the spherical-harmonic (SH) coefficients that produce view-dependent color. To make the scene dynamic, I turned each of these into a function of time $t$, borrowing the temporal radial-basis idea from Spacetime Gaussian but with modifications that trained more stably on the foam.\nTemporal density. Each point\u0026rsquo;s density is gated by a Gaussian bump in time, so a point only contributes around the moment it is actually visible:\n$$ \\sigma_i(t)=\\sigma_i^{s}\\,\\exp\\!\\left(-\\frac{\\lvert t/T-\\mu_i^{\\tau}\\rvert^{2}}{\\exp(2 s_i^{\\tau})}\\right), $$where $T$ is the scene duration, $\\sigma_i^{s}$ is the time-independent (canonical) density reused from Radiant Foam, $\\mu_i^{\\tau}$ is the temporal center (the timestamp at which point $i$ is most visible), and $s_i^{\\tau}$ is a learnable temperature setting how long it stays visible.\nTemporal position. Motion is a polynomial offset from the canonical position $\\mu_i^{s}$, expanded around the point\u0026rsquo;s own temporal center:\n$$ \\mu_i(t)=\\mu_i^{s}+\\sum_{k=1}^{n_p} b_{i,k}\\,(t/T-\\mu_i^{\\tau})^{k}. $$I used degree $n_p=5$ rather than the degree 3 of Spacetime Gaussian; lower degrees trained noticeably worse on the foam.\nTemporal spherical harmonics. To let color change over time, I modulated each spatial SH coefficient by a 1D Fourier term, which keeps the expensive CUDA SH evaluation untouched and only rescales the coefficients:\n$$ sh_i(t)=\\sum_{j=1}^{l}\\cos\\!\\left(\\frac{2\\pi j}{T}\\,t\\right)a_i^{s,j}\\,sh_i^{s,j}, $$where $a_i^{s,j}$ and $sh_i^{s,j}$ are the $j$-th magnitude and basis coefficient of the static SH, and $j$ doubles as the order of the Fourier series.\nTwo instabilities, and two fixes Training surfaced two recurring problems. Both turned out to be informative, and each had a clean remedy.\nStructure collapse, fixed by temporal perturbation With discrete frame times, structure the model had cleanly learned by a few hundred iterations would suddenly collapse: color jitter spread across the image and local geometry fell apart, and it took hundreds more iterations just to recover. The likely cause is that discrete timestamps give the model no signal about temporal continuity, so I made time continuous by jittering it. For each training frame I sampled a perturbed time from a Gaussian centered on the true time $t$, with standard deviation one quarter of the average frame duration $\\Delta t$, clipped so it never strays more than half a frame:\n$$ t'\\sim\\mathcal{N}\\!\\big(t,\\,(\\Delta t/4)^2\\big), \\qquad \\lvert t'-t\\rvert \\le \\Delta t/2. $$This small change largely eliminated the collapses and made the model markedly more robust for the rest of training.\nTop: without perturbation, coherent structure at iteration 1,500 (left) collapses by iteration 2,000 (right). Bottom: with temporal perturbation, structure learned at iteration 1,500 is preserved through iteration 2,000. Color-gradient dependency, fixed by an SSIM loss The second failure was subtler. The model leaned on color gradients to learn temporal correspondence, so it fixated on high-intensity regions and ignored the rest of the scene. The error map makes it obvious: bright areas get reconstructed while large swaths stay blank.\nThe color-gradient dependency problem. Left: rendering. Middle: ground truth. Right: L1 error map. The model pours its capacity into high-intensity regions and leaves the rest unreconstructed. The fix was to add a Structural Similarity (SSIM) loss on top of Radiant Foam\u0026rsquo;s objective. SSIM is a perceptual loss that weighs luminance, contrast, and, crucially, local structure:\n$$ \\mathcal{L}_{\\text{SSIM}}(x,y)=1-\\frac{(2\\mu_x\\mu_y+c_1)(2\\sigma_{xy}+c_2)}{(\\mu_x^2+\\mu_y^2+c_1)(\\sigma_x^2+\\sigma_y^2+c_2)}, $$with $\\mu$, $\\sigma^2$, and $\\sigma_{xy}$ the local pixel means, variances, and covariance. The final training objective just adds it to Radiant Foam\u0026rsquo;s loss $\\mathcal{L}_{\\text{RF}}$:\n$$ \\mathcal{L}=\\mathcal{L}_{\\text{RF}}+\\lambda\\,\\mathcal{L}_{\\text{SSIM}},\\qquad \\lambda=0.25. $$The effect was dramatic. The model switched to reconstructing the whole scene uniformly, and it converged far faster: the rendering below was reached at iteration ~500 with SSIM, whereas comparable quality without it took 12,000–16,000 iterations, a roughly 25× speedup.\nWith the SSIM loss. Left: rendering. Middle: ground truth. Right: L1 error map. Error is now spread evenly instead of concentrating on bright regions. I also tried a learned alternative: a small fully-connected deformation network that takes a canonical position, density, and time $t$ and predicts motion and density offsets. Even on simple scenes its VRAM use blew up and it ran out of memory, so it never became a serious contender.\nResults I evaluated on both settings. For multi-view, I used the coffee-martini scene from Neural 3D Video (six real scenes, roughly 10 seconds each), extracted at 30 FPS using only the first 30 frames, holding out camera 0 for testing, initializing points with COLMAP, and downsampling frames 4×. For monocular, I used the bouncing balls scene from the synthetic D-NeRF dataset, plus a monocular cut of Neural 3D Video (150 frames from camera 0 for training, 150 from camera 7 for testing).\nTraining used Adam with degree-3 spherical harmonics for 50,000 iterations: the first 2,000 are warm-up, and the cell count grows until iteration 31,000. Because each point\u0026rsquo;s neighbours change with time, I could not rely on Radiant Foam\u0026rsquo;s lazy triangulation. Its AABB tree was rebuilt incrementally every iteration and fully rebuilt every 1,000 iterations, where the original codebase only rebuilds after densification and near the end. That triangulation cost is exactly what made full multi-view training too expensive to finish. With no completed baseline to compare against, models were ranked by how closely their renderings matched the ground-truth frames.\nMonocular, synthetic (D-NeRF). The model failed to converge and rendered incorrect colors. Some point positions land roughly right, but most drift far from ground truth, most likely because the data is simply too sparse.\nD-NeRF bouncing balls, monocular. Left: rendering. Middle: ground truth. Right: L1 error map. The model never converges to the correct colors or geometry. Monocular, real (Neural 3D Video cut). Renderings are locally sharp (the teaser above), but the model overfits the training views and cannot produce correct novel angles. A single viewpoint just does not span enough angular range to generalize.\nMulti-view (Neural 3D Video). Far more demanding in compute and time, so training never finished. But the intermediate renderings are encouraging: the model generalizes to the correct viewing angle, cleanly learns static regions, and starts to pick up motion.\nMulti-view reconstruction on Neural 3D Video. Left: rendering. Middle: ground truth. Right: L1 error map. Even without completed training, the model renders from the correct novel viewpoint and recovers static structure. Takeaways Temporal radial-basis modelling struggles on monocular video, especially when viewing angles barely vary; the multi-view setting is where it shows promise. Mesh-based dynamic reconstruction is extremely sensitive to pruning. Every pruning scheme I tried degraded training, so I fell back to a simple densification that just fills large empty cells. Future methods should confirm gains with densification alone before adding any pruning. Point density is delicate. It took many candidate temporal functions before one trained stably, and degree-5 motion polynomials clearly beat degree 3. The \u0026ldquo;fast training\u0026rdquo; claims of many methods quietly assume the whole dataset fits in memory. That assumption breaks for multi-view video, where the data volume and the repeated triangulation dominate the runtime. Where it goes next Speed is everything here, so the open problems are systems-shaped: a dynamic AABB tree for triangulation, a way to maintain the adjacency list without re-triangulating every step, a better pruning and densification strategy tuned for this representation, and a richer motion model for the monocular case. The full write-up lives in the report, the slides are here, and the implementation, with the proposal and milestone reports, is in the project repository.\n","permalink":"https://quangminhdinh.github.io/projects/dynamic-radiant-foam/","summary":"\u003cp\u003eMost real-world environments move, yet most high-quality radiance-field methods assume the scene stands perfectly still. For my final project in \u003ca href=\"https://www.sfu.ca/outlines.html?2025/spring/cmpt/469/d100\"\u003e\u003cstrong\u003eCMPT 469: Rendering and Visual Computing for AI\u003c/strong\u003e\u003c/a\u003e, I set out to add a temporal axis to \u003ca href=\"https://radfoam.github.io/\"\u003e\u003cstrong\u003eRadiant Foam\u003c/strong\u003e\u003c/a\u003e and reconstruct \u003cem\u003edynamic\u003c/em\u003e scenes from monocular and multi-view video. It was an ambitious swing, and an honest one: it did not fully converge, but the failure modes were the interesting part, and each one had a satisfying remedy.\u003c/p\u003e","title":"Dynamic Radiant Foam"},{"content":"PedroVerse is a Blender add-on that turns hyper-realistic 3D assets into vibrant non-photorealistic (NPR) styles, without engine-specific shaders or external photo-editing tools. The trick: instead of touching geometry or writing shaders, we operate entirely on the asset\u0026rsquo;s UV maps (albedo and object-space normal maps), which almost every asset already ships with and which render cheaply everywhere. It was a team project for CMPT 461: Computational Photography at SFU.\nPedroVerse stylizes 3D assets by directly editing their albedo and object-space normal maps: a wide range of NPR looks, no engine-specific shaders. The key insight A shader built in Blender often can\u0026rsquo;t be exported to a game engine for rapid prototyping, and post-processing effects don\u0026rsquo;t generalize across engines or VR environments. But UV maps are universal. They carry strong visual priors, they\u0026rsquo;re inexpensive to render, and, crucially, the normal map preserves dynamic lighting even after stylization. So we define every stylization operation in UV space, combining classical image processing with lightweight deep learning into a modular, permutable pipeline.\nThe pipeline From input textures through style transfer, palette recoloring, and geometric abstraction, applied to both albedo and normal maps. Style transfer: a lightweight pretrained model encodes content and style images, then interpolates by a user-controlled content-to-style ratio (high-quality textures in ~2–3 s). Palette recoloring: palette-based photo recoloring with a modified k-means in LAB space; the artist edits the extracted palette through an HSV color picker. Geometric abstraction: five interchangeable structural filters spanning classical stroke-based rendering (Bézier brush strokes swept along image gradients), the neural Paint Transformer, SLIC superpixels, Pyxelate 8-bit pixelation, and an anime-style Voronoi watercolor effect. The hard part is applying these to the normal map: a wrong color shift there changes the direction light bounces off the surface, producing artifacts. We found that gradient-aware methods (brush strokes) are far safer than noisy splatting (neural paint), that low SLIC compactness suppresses sudden curvature shifts, and that a bitwise mask between the stylized and original normal map keeps everything inside valid UV space.\nImplementation \u0026amp; results The UI is built on the Blender Python API; style transfer runs in TensorFlow, classical filters in OpenCV / scikit-image, and the Paint Transformer in PyTorch, with each component invoked via subprocess to dodge Blender\u0026rsquo;s module-import quirks, then packaged as a single add-on. Even the least-optimized method takes at most ~1 minute per asset.\nResults across both object-centric assets and full 3D environments. Takeaways Working in UV space buys portability for free. By never touching geometry or engine shaders, the same stylization travels across engines and VR: the whole reason the approach is worth it. Normal maps are unforgiving. Stylizing albedo is easy; stylizing the normal map without breaking lighting forced most of the interesting engineering (gradient-awareness, compactness tuning, UV masking). Classical + lightweight learning is a sweet spot for interactive tools: fast enough to keep an artist in the loop, flexible enough to permute many looks. We\u0026rsquo;re also prototyping CLIP-based style transfer (CLIPstyler with ViT-B/32) to replace the VGG-based approach for text-driven stylization. Code is on GitHub.\n","permalink":"https://quangminhdinh.github.io/projects/pedroverse/","summary":"\u003cp\u003e\u003cstrong\u003ePedroVerse\u003c/strong\u003e is a \u003cstrong\u003eBlender add-on\u003c/strong\u003e that turns hyper-realistic 3D assets into vibrant \u003cstrong\u003enon-photorealistic (NPR)\u003c/strong\u003e styles, \u003cem\u003ewithout\u003c/em\u003e engine-specific shaders or external photo-editing tools. The trick: instead of touching geometry or writing shaders, we operate entirely on the asset\u0026rsquo;s \u003cstrong\u003eUV maps\u003c/strong\u003e (albedo and object-space normal maps), which almost every asset already ships with and which render cheaply everywhere. It was a team project for \u003ca href=\"https://yaksoy.github.io/cpim/\"\u003e\u003cstrong\u003eCMPT 461: Computational Photography\u003c/strong\u003e\u003c/a\u003e at SFU.\u003c/p\u003e\n\u003cfigure\u003e\n  \u003cimg src=\"teaser.jpg\" alt=\"PedroVerse stylized 3D assets\"\u003e\n  \u003cfigcaption\u003ePedroVerse stylizes 3D assets by directly editing their albedo and object-space normal maps: a wide range of NPR looks, no engine-specific shaders.\u003c/figcaption\u003e\n\u003c/figure\u003e\n\u003ch2 id=\"the-key-insight\"\u003eThe key insight\u003c/h2\u003e\n\u003cp\u003eA shader built in Blender often can\u0026rsquo;t be exported to a game engine for rapid prototyping, and post-processing effects don\u0026rsquo;t generalize across engines or VR environments. But \u003cstrong\u003eUV maps are universal\u003c/strong\u003e. They carry strong visual priors, they\u0026rsquo;re inexpensive to render, and, crucially, the \u003cstrong\u003enormal map preserves dynamic lighting\u003c/strong\u003e even after stylization. So we define every stylization operation in \u003cstrong\u003eUV space\u003c/strong\u003e, combining classical image processing with lightweight deep learning into a modular, permutable pipeline.\u003c/p\u003e","title":"PedroVerse"},{"content":"MirrorBrain was an LLM-augmented, Zettelkasten-style note-taking app: an Obsidian-like graph of atomic notes where the model auto-generates the connections, context, and metadata between ideas. It was the flagship product of KaleidoAI, my startup attempt through the Ethos Fund New Venture Challenge. I led a team of 5, building on Next.js/TypeScript, FastAPI, Convex, and Qdrant.\nIt did not succeed, and the reason was execution, not idea. I\u0026rsquo;m keeping it here precisely because the post-mortem taught me more than a polished win would have.\nThe concept The product centered on a few interlocking ideas:\nNotes with an Obsidian-style structure, editable like Notion, linkable to knowledge sources, and quick to scaffold from a short description. Knowledge sources ingested from PDFs, docs, slides, websites, YouTube, and podcasts, then auto-summarized into chapter summaries, Q\u0026amp;A, related works, and tags. A glossary that auto-builds definitions and related concepts from terms extracted out of your notes. A chat interface that retrieves across your notes and their attached sources, behind a fast, Notion-like command bar. It began life as a browser extension for capturing notes on the fly, then grew into a full workspace: editable block-based pages, a knowledge graph that suggests links and extrapolates context, and an AI-powered editor with continue and grammar assists. The buttons above open full-resolution versions of each design.\nThe UX wireframe: routing from auth, the note/source/term pages, and a Notion-style command bar wired to the API. What I built and led The stack was Next.js + TypeScript + Tailwind with BlockNote as the editor, a FastAPI backend, Convex for real-time data, and Qdrant for vector search. The system fanned out into four paths: ingestion (Tavily/SerpAPI lookups plus context-aware, block-level chunk splitting), search and RAG, block-level injection back into pages, and a websocket chat scoped to a note\u0026rsquo;s attached sources.\nSystem architecture: the ingestion, search/RAG, block-injection, and chat paths over the Next.js, FastAPI, and storage layers. The data model stayed deliberately small: pages typed as note, source, or topic, assembled from a linked list of blocks, with each block mirrored into Qdrant by id.\nThe data model: pages, blocks, sources, and users in Convex, each block synced to a Qdrant payload by id. Two engineering details I\u0026rsquo;m still fond of:\nReverse-engineering BlockNote\u0026rsquo;s side-menu UI to bend the editor into the linked-note interactions the product needed. Replacing Qdrant vector search with Okapi BM25 for the text-retrieval path: for our note-sized corpus, lexical BM25 was faster and good enough, and it removed an entire moving part from the hot path. The milestone roadmap: how the extension, note structure, editor, and chat were meant to evolve from v0.1 to v0.3. Post-mortem The idea wasn\u0026rsquo;t the problem; execution was. We over-invested in product surface and architecture before pressure-testing demand and shipping cadence: the classic founder trap. Sometimes the boring choice wins. Swapping a vector DB for BM25 was a small reminder that the simplest tool that clears the bar beats the impressive one that doesn\u0026rsquo;t. Leading five people taught me delivery discipline (Kanban, scoping, and saying no) more than any single technical task did. Code for the v0 build lives in the KaleidoAI repo.\n","permalink":"https://quangminhdinh.github.io/projects/mirrorbrain/","summary":"\u003cp\u003e\u003cstrong\u003eMirrorBrain\u003c/strong\u003e was an LLM-augmented, \u003cstrong\u003eZettelkasten-style\u003c/strong\u003e note-taking app: an Obsidian-like graph of atomic notes where the model auto-generates the connections, context, and metadata between ideas. It was the flagship product of \u003cstrong\u003eKaleidoAI\u003c/strong\u003e, my startup attempt through the \u003cstrong\u003e\u003ca href=\"https://www.ethosfund.vc/\"\u003eEthos Fund\u003c/a\u003e New Venture Challenge\u003c/strong\u003e. I led a team of 5, building on Next.js/TypeScript, FastAPI, Convex, and Qdrant.\u003c/p\u003e\n\u003cp\u003eIt did not succeed, and the reason was \u003cstrong\u003eexecution, not idea\u003c/strong\u003e. I\u0026rsquo;m keeping it here precisely because the post-mortem taught me more than a polished win would have.\u003c/p\u003e","title":"MirrorBrain"},{"content":"Most discussion of world models treats \u0026ldquo;the world state\u0026rdquo; as a given object you read off and predict forward. I keep getting stuck one step earlier: what is a world state, concretely?\nThe open questions I can\u0026rsquo;t yet answer cleanly:\nThere\u0026rsquo;s no uniform representation of a 3D world, and no uniform representation of its state. So which one do you commit to, and what does that choice quietly assume? Does the state have to be maintained in real time, or is it something you reconstruct on demand? Where does the data defining it even come from? The reason this isn\u0026rsquo;t just bookkeeping: the representation is the problem, not a detail you settle before the real work. That\u0026rsquo;s the same lesson as Every conditioning choice is a hidden assumption, the encoding has to match the thing it\u0026rsquo;s trying to steer. A \u0026ldquo;world state\u0026rdquo; you can\u0026rsquo;t ground in a representation is just a word.\nA seedling, deliberately. Back to World models as shared substrate.\n","permalink":"https://quangminhdinh.github.io/garden/what-is-a-world-state/","summary":"\u003cp\u003eMost discussion of world models treats \u0026ldquo;the world state\u0026rdquo; as a given object you read off and predict forward. I keep getting stuck one step earlier: what \u003cem\u003eis\u003c/em\u003e a world state, concretely?\u003c/p\u003e\n\u003cp\u003eThe open questions I can\u0026rsquo;t yet answer cleanly:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThere\u0026rsquo;s no uniform representation of a 3D world, and no uniform representation of its state. So which one do you commit to, and what does that choice quietly assume?\u003c/li\u003e\n\u003cli\u003eDoes the state have to be maintained in real time, or is it something you reconstruct on demand?\u003c/li\u003e\n\u003cli\u003eWhere does the data defining it even come from?\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe reason this isn\u0026rsquo;t just bookkeeping: the representation \u003cem\u003eis\u003c/em\u003e the problem, not a detail you settle before the real work. That\u0026rsquo;s the same lesson as \u003ca class=\"wikilink\" href=\"/garden/conditioning-is-an-assumption/\"\u003eEvery conditioning choice is a hidden assumption\u003c/a\u003e, the encoding has to match the thing it\u0026rsquo;s trying to steer. A \u0026ldquo;world state\u0026rdquo; you can\u0026rsquo;t ground in a representation is just a word.\u003c/p\u003e","title":"What even is a world state?"},{"content":"This was my first large-scale research project, and my first run at the AI City Challenge. The task, Track 2 of the 7th AI City Challenge (CVPR 2023), is retrieving the right vehicle track from a natural-language description: given a sentence like \u0026ldquo;a gray SUV turns left at a busy intersection,\u0026rdquo; find the matching video track among thousands. My solution, SNDA (Symmetric Network with Dual-vehicle Attributes Augmentation), reached 35.44% MRR, 7th place on the leaderboard.\nSNDA: four branches learn local and global representations of both the text queries and the track images, fused and aligned with a symmetric InfoNCE loss, then enhanced by a dual-vehicle attribute system on the visual side. Why the task is hard Natural-language vehicle retrieval sits between video understanding and fine-grained re-identification, and it inherits the worst of both. Three problems make it genuinely difficult:\nAmbiguous, information-poor queries. A single sentence rarely pins down a vehicle. \u0026ldquo;A white sedan goes straight\u0026rdquo; matches hundreds of tracks. Tiny inter-class variation. Vehicles with different identities routinely share appearance and motion attributes, so the visual signal that separates them is thin. Data scarcity. There simply isn\u0026rsquo;t much annotated track/vehicle data to train a robust cross-modal model, and image-query re-identification pipelines don\u0026rsquo;t transfer cleanly to text queries. Architecture SNDA adapts the symmetric SSM design and uses four branches to capture local and global representations of each modality. Text features come from a frozen RoBERTa; visual features from an EfficientNet-B2 backbone. The idea is that local features carry instance detail (the cropped vehicle, the augmented subject phrase) while global features carry context (the whole motion image, the full sentence with location cues), and aligning both gives the model two complementary views of the same track.\nCross-modal representation learning Each backbone embedding passes through a small projection head before alignment. The two text heads share a layer-normalized form, and the two visual heads a batch-normalized one:\n$$ f_t = g_t(h_t) = W_2\\,\\sigma\\!\\big(\\mathrm{LN}(W_1 h_t)\\big), \\qquad f_i = g_i(h_i) = W_2\\,\\sigma\\!\\big(\\mathrm{BN}(W_1 h_i)\\big) $$where $h$ is a backbone embedding and $\\sigma$ is ReLU. The local and global text features are concatenated into the language representation $E_t = [\\,f_t^l \\,\\|\\, f_t^g\\,]$, and likewise the visual features into $E_i = [\\,f_i^l \\,\\|\\, f_i^g\\,]$. Alignment then happens at four levels: the two local/global pairs, the fused pair $\\langle E_i, E_t\\rangle$, and an augmented pair introduced below.\nModeling motion without a video model Every camera in the dataset is static and its background is stable, which I turned into a shortcut. Instead of a video transformer, I collapse each track into a single motion map. First average all frames into a clean background,\n$$ B = \\frac{1}{N}\\sum_{i=1}^{N} F_i, $$then paste the cropped vehicle from each frame\u0026rsquo;s bounding box back on top. Because boxes from consecutive frames overlap and clutter the image, I skip any box whose IoU with the previous one exceeds a threshold $T = 0.05$. The result is one motion-bearing image the visual branch can consume cheaply, capturing the vehicle\u0026rsquo;s trajectory without ever running a temporal model.\nThe dual-vehicle attribute system The core idea: many queries describe two nearby vehicles, and that second vehicle is free supervision. I extract attributes by frequency-analyzing the dataset vocabulary and grouping the most common words into categories: vehicle type (suv, sedan, truck, van), color (red, blue, black, white, and a merged gray/grey/silver), size (small, mid-sized, large), motion (left, right, stop, else go straight), and an intersection flag for long-distance scene context.\nFrequencies of words describing vehicle type, motion, size, and color: the empirical basis for the attribute categories and their merges. Critically, type and color come in pairs. Many sentences name the target and a nearby vehicle, so I prioritize finding two same-category words that co-occur in one query, yielding type1/type2 and color1/color2 (with color2 discarded if no type2 is found). Motion and size are extracted once as motion1 and size1. Because each attribute is mined across a shuffled, merged set of every description of a scene, the value pulled in can differ from the words in any single query, which adds detail rather than echoing it.\nExample Primary descriptions A gray SUV turns left from intersection. A gray SUV turns left through a busy intersection. A gray minivan takes a left at an intersection. Other camera views A big SUV making a left turn at the intersection. A gray SUV turn left followed by another pickup truck. A gray van turns left. Attributes are mined across the primary descriptions and the descriptions of the same scene from other cameras. Here size (big), a second type (truck), and the intersection flag all surface from views beyond the primary sentence.\nThese attributes drive three mechanisms.\nText augmentation. Extracted attributes are prepended (and, for location, appended) to two sampled queries to enrich sparse descriptions, separately for the local and global text inputs.\nInput Augmented query Local big gray suv. A gray minivan takes a left at an intersection. Global left. A gray SUV turns left from intersection. intersection. Subject augmentation (size + color + type) on the local input; motion and location augmentation on the global input.\nVisual attribute heads. Seven projection heads predict the attributes directly from the visual features. For an attribute with $n_{attr}$ categories I build a one-hot label of length $n_{attr}+1$, reserving the extra slot for missing or out-of-vocabulary values so \u0026ldquo;unknown\u0026rdquo; is a first-class class rather than noise. The heads are trained with cross-entropy, forcing the visual side to learn explicit attribute signals it can match against the text.\nAugmented representation pair. The visual attribute predictions are fused with the visual features into an augmented vector $E_i^a$, paired with a projected language vector $E_t^a$, giving the fourth and most attribute-aware alignment pair.\nLoss functions Every visual-language pair is aligned with a symmetric InfoNCE loss. For a batch of $N$ pairs, the two directions are\n$$ \\mathcal{L}_{t2i}^{z} = -\\frac{1}{N}\\sum_{i=1}^{N}\\log \\frac{\\exp\\!\\big(\\cos(z_{\\text{text}}^i, z_{\\text{img}}^i)/\\tau\\big)} {\\sum_{j=1}^{N}\\exp\\!\\big(\\cos(z_{\\text{text}}^i, z_{\\text{img}}^j)/\\tau\\big)}, \\qquad \\mathcal{L}_{\\text{SNCE}}^{z} = \\tfrac{1}{2}\\big(\\mathcal{L}_{t2i}^{z} + \\mathcal{L}_{i2t}^{z}\\big), $$with $\\mathcal{L}_{i2t}^{z}$ defined symmetrically and $\\tau$ a learned temperature. The representation loss sums this over all four pairs,\n$$ \\mathcal{L}_{\\text{rep}} = \\sum_{z\\in\\mathcal{Z}} \\mathcal{L}_{\\text{SNCE}}^{z}, \\qquad \\mathcal{Z} = \\big\\{\\langle f_i^l, f_t^l\\rangle,\\ \\langle f_i^g, f_t^g\\rangle,\\ \\langle E_i, E_t\\rangle,\\ \\langle E_i^a, E_t^a\\rangle\\big\\}, $$and the seven attribute heads contribute an averaged cross-entropy term, giving the total objective:\n$$ \\mathcal{L}_{\\text{aug}} = \\frac{1}{7}\\sum_{attr}\\mathcal{L}_{attr}^{a}, \\qquad \\mathcal{L} = \\mathcal{L}_{\\text{rep}} + \\mathcal{L}_{\\text{aug}}. $$Post-processing At inference, two relationship terms refine the similarity matrix. Long-distance modeling detects intersections (in text via the intersection flag; in video by checking whether a vehicle stays roughly still across $n$ frames) and scores their agreement as $S_l$. Short-distance modeling scores vehicle-to-vehicle relationships by matching the local features of every vehicle in a track against the query, giving $S_r$. The final matrix combines the four learned similarities with both terms:\n$$ S = \\sum_{z\\in\\mathcal{Z}} S_z + \\alpha S_r + \\beta S_l, \\qquad \\alpha = 1,\\ \\beta = 0.2. $$Dataset and metrics The track uses a variation of CityFlow-NL: 666 target vehicles across 3,598 single-view tracks from 46 calibrated cameras, each annotated with three distinct natural-language descriptions, for 2,155 training tracks plus a held-out 184 query/track test split. Descriptions cover color, maneuver, traffic scene, and relations to other vehicles.\nAn example query–track pair from CityFlow-NL: three sentences describe a single tracked vehicle, with appearance, motion, and scene cues spread across them. Performance is the official Mean Reciprocal Rank, with Recall@5 and Recall@10 as secondary metrics:\n$$ \\mathrm{MRR} = \\frac{1}{|Q|}\\sum_{i=1}^{|Q|}\\frac{1}{\\mathrm{rank}_i}, \\qquad \\mathrm{Recall}@k = \\frac{|R \\cap D(k)|}{|R|}, $$where $\\mathrm{rank}_i$ is the position of the correct track for query $i$, and $D(k)$ the top-$k$ retrieved set.\nImplementation. Frozen RoBERTa text encoder, EfficientNet-B2 visual backbone (ImageNet-pretrained), images resized to $228\\times228$, batch size 64, 400 epochs of AdamW (weight decay $10^{-2}$, initial LR $0.01$) with a 40-epoch warm-up and a step decay every 80 epochs, all on a single NVIDIA A100-80G.\nResults, and the lesson that stuck The ablations told a two-sided story. On the validation set, every technique helped, and the final model reached 0.58 MRR / 0.87 R@5 / 0.94 R@10.\nMethod MRR ↑ Recall@5 ↑ Recall@10 ↑ Baseline (symmetric net, no attributes) 0.47 0.32 0.69 + all second-vehicle attributes 0.44 0.25 0.68 + selected attributes (type2, color2 only) 0.45 0.60 0.87 + NL augmentation 0.58 0.78 0.93 + feature engineering (final) 0.58 0.87 0.94 Ablation on the validation set. Using *all* of the second vehicle's attributes hurts (size2 and motion2 are too sparse); keeping only type2 and color2 recovers it, and text augmentation drives the biggest jump.\nBut on the test set, the same natural-language augmentation that helped validation actually hurt: the model had learned validation-specific quirks and failed to cross the domain gap. Only feature engineering, ensembling, and post-processing recovered the score, to the 0.35 MRR that placed 7th.\nMethod MRR ↑ Recall@5 ↑ Recall@10 ↑ Baseline 0.23 0.33 0.50 + selected attributes 0.25 0.44 0.57 + NL augmentation 0.23 0.36 0.51 + feature engineering 0.24 0.39 0.58 Ensemble + post-processing (final) 0.35 0.53 0.64 Ablation on the test set. NL augmentation *regresses* the score here, the mirror image of the validation table, and only ensembling plus the post-processing terms close the gap.\nA strong validation number can be a trap. The gap between 0.58 (val) and 0.35 (test) was almost entirely a domain-generalization failure: my clearest early lesson that the test distribution is the only one that counts. Cheap structure beats heavy machinery. Motion maps and attribute heads delivered most of the gains without any video transformer, on a single GPU. Free supervision is everywhere if you read the data closely: the second vehicle in a sentence was hiding in plain sight, and pairing type/color was worth more than throwing every attribute at the model. The future direction I\u0026rsquo;d take from here is squarely domain adaptation, plus a better fusion mechanism for the augmentation signals. The full method is in the report and the code is on GitHub. This project also set the stage for TrafficVLM, my later AI City Challenge work that placed 3rd.\n","permalink":"https://quangminhdinh.github.io/projects/symmetric-vehicle-retrieval/","summary":"\u003cp\u003eThis was my \u003cstrong\u003efirst large-scale research project\u003c/strong\u003e, and my first run at the \u003ca href=\"https://www.aicitychallenge.org/\"\u003e\u003cstrong\u003eAI City Challenge\u003c/strong\u003e\u003c/a\u003e. The task, Track 2 of the 7th AI City Challenge (CVPR 2023), is \u003cem\u003eretrieving the right vehicle track from a natural-language description\u003c/em\u003e: given a sentence like \u003cem\u003e\u0026ldquo;a gray SUV turns left at a busy intersection,\u0026rdquo;\u003c/em\u003e find the matching video track among thousands. My solution, \u003cstrong\u003eSNDA (Symmetric Network with Dual-vehicle Attributes Augmentation)\u003c/strong\u003e, reached \u003cstrong\u003e35.44% MRR, 7th place\u003c/strong\u003e on the leaderboard.\u003c/p\u003e","title":"Symmetric Network with Dual-vehicle Attributes Augmentation"},{"content":"A fun, early experiment, one of my first attempts at reproducing a research idea end-to-end. Instead of feeding raw pixels to a network, this project classifies facial emotion from the angular geometry of facial landmarks detected by MediaPipe, replicating the preprocessing pipeline of a 2022 emotion-recognition paper.\nThe idea Raw-pixel emotion classifiers entangle identity, lighting, and pose with expression. The alternative explored here is geometric: detect the face mesh with MediaPipe, then encode the angles between facial landmarks as features. Those angle features feed classical classifiers (KNN, SVM, and an auto-sklearn AutoML baseline) trained on the FER2013 expression dataset.\nThe pipeline is deliberately small and legible:\nencode.py: the angular feature encoding preprocess.py: data preparation model.py: classifier definitions run.py: a CLI, e.g. python run.py fit --model svm --data fer2013 Results On FER2013 (48×48 images), the models reached roughly 52% (auto-sklearn), 48% (KNN), and 40% (SVM) accuracy, below the original paper, mainly because I omitted the super-resolution enhancement step the authors used before landmark detection. At 48×48, MediaPipe\u0026rsquo;s landmarks are noisy, and the angular encoding inherits that noise.\nTakeaways Resolution matters before geometry does. The biggest gap to the paper came from skipping super-resolution: landmark-based features are only as good as the landmarks, and tiny inputs starve them. Geometric features are interpretable but brittle. Encoding angles strips away identity and lighting nicely, yet it\u0026rsquo;s far more sensitive to detector error than a pixel CNN would be. Reproducing a paper teaches you where its accuracy actually comes from, often in an unglamorous preprocessing step the abstract barely mentions. Code and the CLI are on GitHub.\n","permalink":"https://quangminhdinh.github.io/projects/mediapipe-face-emotion/","summary":"\u003cp\u003eA fun, early experiment, one of my first attempts at reproducing a research idea end-to-end. Instead of feeding raw pixels to a network, this project classifies facial emotion from the \u003cstrong\u003eangular geometry of facial landmarks\u003c/strong\u003e detected by \u003cstrong\u003eMediaPipe\u003c/strong\u003e, replicating the preprocessing pipeline of a 2022 emotion-recognition paper.\u003c/p\u003e\n\u003ch2 id=\"the-idea\"\u003eThe idea\u003c/h2\u003e\n\u003cp\u003eRaw-pixel emotion classifiers entangle identity, lighting, and pose with expression. The alternative explored here is \u003cstrong\u003egeometric\u003c/strong\u003e: detect the face mesh with MediaPipe, then encode the \u003cstrong\u003eangles\u003c/strong\u003e between facial landmarks as features. Those angle features feed classical classifiers (\u003cstrong\u003eKNN\u003c/strong\u003e, \u003cstrong\u003eSVM\u003c/strong\u003e, and an \u003cstrong\u003eauto-sklearn\u003c/strong\u003e AutoML baseline) trained on the \u003cstrong\u003eFER2013\u003c/strong\u003e expression dataset.\u003c/p\u003e","title":"Face Emotion Detection with Angular Encoding"},{"content":"At YITEC, as a Machine Learning Engineer (Oct 2020 – Jun 2021), I helped build Vietnam\u0026rsquo;s first real-time hand-tracking and AR ring try-on mobile app, letting customers try on jewellery rings from their phones instead of visiting a store. This was my first ML-engineering role, before I had any clue about AI research; it was pure, hands-on exploration, and it became one of the most formative projects of my early career.\nThe project was a major success, matching or beating the solutions of FPT and Viettel at the time, and laid the foundation for one of YITEC\u0026rsquo;s later core businesses.\nThe virtual ring rendered onto a tracked hand in real time. Demo The engineering problem Putting a convincing virtual ring on a moving finger in real time is a stack of hard sub-problems (tracking, rendering, lighting, and latency), all at once, on a phone. My work spanned:\nHand tracking with MediaPipe Graph on Android Studio, including research into frame conversion and exporting to AAR format for integration with Unity. Real-time camera processing in Unity via AR Foundation, and using Google ARCore to read the scene\u0026rsquo;s lighting cues so the virtual ring and its reflection probe render under the same conditions as the real hand. Placement maths: deriving formulas for the ring\u0026rsquo;s position, scale, and rotation from MediaPipe\u0026rsquo;s relative landmark positions, plus an acceptable range for those factors that accounts for latency and vibration. Porting the MediaPipe API to C#, one call at a time, and attempting to integrate it with Unity Barracuda to eliminate frame conversion and cut latency. Stabilisation \u0026amp; denoising: experimenting with video-stabilisation and vibration-denoising algorithms that took the experience from 5 FPS to 30 FPS. Takeaways Real-time AR is a latency budget first, an ML problem second. The biggest single win (5 → 30 FPS) came from stabilisation and removing frame conversions, not from a better model. Physical realism sells the illusion. Matching the ring\u0026rsquo;s lighting and reflections to ARCore\u0026rsquo;s scene cues mattered more to perceived quality than landmark precision. This was my on-ramp to ML. Approaching the field as an engineer first, shipping something real under hard constraints, shaped how I think about research today. The most recent version of this work is under NDA, so the public repository reflects the earlier exploration. See also the YITEC project page and JLM.\n","permalink":"https://quangminhdinh.github.io/projects/virtual-ring-tryon/","summary":"\u003cp\u003eAt \u003cstrong\u003e\u003ca href=\"https://yitec.net/\"\u003eYITEC\u003c/a\u003e\u003c/strong\u003e, as a Machine Learning Engineer (Oct 2020 – Jun 2021), I helped build \u003cstrong\u003eVietnam\u0026rsquo;s first real-time hand-tracking and AR ring try-on mobile app\u003c/strong\u003e, letting customers try on jewellery rings from their phones instead of visiting a store. This was my first ML-engineering role, \u003cem\u003ebefore I had any clue about AI research\u003c/em\u003e; it was pure, hands-on exploration, and it became one of the most formative projects of my early career.\u003c/p\u003e","title":"Virtual Ring Try-On"}]