Generative traffic video forecasting aims to synthesize long-horizon, temporally coherent future videos of traffic scenes from a short observation history and textual descriptions. In this paper, we present CosmosAlign, a generative traffic video forecasting framework built upon the pretrained Cosmos3-Nano world foundation model. Our approach is motivated by the observation that successfully adapting large pretrained world models to downstream forecasting tasks depends primarily on distribution alignment rather than increased model capacity.
To this end, we propose a two-stage LoRA adaptation strategy that first aligns the conditioning-mode distribution with the target forecasting task, and then aligns the training captions with the model's native structured prompting interface through an LLM-based re-captioning pipeline. During inference, we further improve prediction quality using a fully training-free procedure consisting of consensus-based medoid sample selection and motion-adaptive blending of static scene regions.
CosmosAlign achieves a final score of 76.49 on the AI City Challenge 2026 Track 5 benchmark, ranking first on the final leaderboard while achieving the best PSNR and LPIPS and the joint-highest SSIM.
Track 5 of the AI City Challenge 2026 provides a short history of observed frames from a traffic safety scenario, together with textual descriptions of the pedestrian and the vehicle, and asks for the next 51–120 future frames of the same scene at 1280×720. The benchmark is built on the Woven Traffic Safety (WTS) dataset, recorded from fixed overhead cameras and vehicle-mounted cameras, and submissions are scored by a normalized average of PSNR, SSIM, LPIPS, CLIP score, FID, and FVD.
| Rank | Team | Final↑ | PSNR↑ | SSIM↑ | LPIPS↓ | CLIP↑ | FID↓ | FVD↓ |
|---|---|---|---|---|---|---|---|---|
| 1 | Qyn (ours) | 76.49 | 20.12 | 0.650 | 0.246 | 0.950 | 22.41 | 21.79 |
| 2 | SSUPER | 76.04 | 19.73 | 0.630 | 0.249 | 0.938 | 21.16 | 19.46 |
| 3 | Latent Painter | 75.13 | 19.72 | 0.650 | 0.266 | 0.945 | 26.52 | 24.79 |
| 4 | CHTTL_A30 | 74.05 | 18.86 | 0.597 | 0.281 | 0.942 | 23.78 | 24.11 |
| 5 | VGU_ai_lab | 73.28 | 19.74 | 0.647 | 0.294 | 0.945 | 33.61 | 29.35 |
Our solution ranks first with a final score of 76.49, 0.45 points ahead of the second-ranked team, attaining the best PSNR, LPIPS, CLIP, and joint-highest SSIM of all teams. Guidance tuning, Stage 2 caption alignment, medoid selection, and motion-adaptive blending each improve the final score, with the last step contributing the largest gain (+1.17).
(a) Overhead view: the pedestrian is advanced along the ground-truth trajectory across the crosswalk, with the static camera and the road markings preserved. (b) Vehicle view at dusk: the lead vehicle, brake lights, and light sources are continued stably. (c) Behavioral divergence: the generation stays sharp and realistic, but the predicted pedestrian behavior differs from the ground truth. Divergence of this kind is exactly the stochasticity that our medoid selection exploits: futures on which independent samples agree are much less likely to contain such idiosyncratic behavior errors.
The history frames of this validation clip show a dashboard clock; the ground truth (top) cuts to the road scene, whereas our prediction (bottom) carries the glowing digits into the generated road scene and holds them for the whole horizon, although the prompt describes only the road scene. This behavior indicates that the five conditioning frames, not the prompt, anchor the generated scene.
@inproceedings{dinh2026cosmosalign,
author = {Dinh, Quang Minh and Doan, Tuan Kiet},
title = {CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops},
year = {2026},
}