Overview

Learning control from pixels without extrinsic rewards is a foundational challenge for generalist agents, particularly in domains where reward design is brittle, unavailable, or easily exploited. We propose Rank-Then-Act (RTA), a framework for reinforcement learning from video in which supervision is defined over ordinal structure rather than scalar reward prediction. In Stage 1, we train a VLM as a progressive ordinal estimator using GRPO on shuffled video segments with anchor conditioning, explicitly removing access to absolute frame positions. In Stage 2, we do not use the VLM as a scalar reward model; instead, the RL signal is a correlation functional-Spearman rank correlation between predicted ordinal progress and true temporal indices within a sliding window-producing a bounded, scale-invariant signal that depends only on ordering structure.

Unlike prior video reward models that directly regress scalar progress or binary success signals, RTA uses a correlation objective over ordinal structure, making the reward invariant to scale, calibration drift, and cross-task shifts. This design enables policy learning driven entirely by progress inferred from expert demonstrations and environment interactions.

Contributions

• A VLM progress scorer trained offline on shuffled expert video clips with GRPO, taught to assign progress ranks on shuffled clips, and kept frozen during policy learning.

• A correlation-only reward for online control: Spearman correlation between predicted progress ranks and time indices over sliding windows, enabling dense, shaped, reward-free feedback.

• Effective control across discrete and continuous benchmarks without environment rewards, outperforming strong video-based baselines including Rank2Reward and VLM-RM.

• Cross-task and cross-environment reuse of a single pretrained progress scorer, suggesting strong generality of ordinal video supervision.

Method

Stage 1: Listwise Progress Scorer via GRPO

Given an expert video, we anchor the first frame and randomly shuffle the remaining frames to break temporal shortcuts. A VLM progress scorer is prompted to assign each frame a progress rank (higher = later). The training reward is Spearman rank correlation between predicted ranks and ground-truth temporal indices:

$R = \mathrm{spr}(\mathbf{p}, \mathbf{q})$

We optimize this reward with Group Relative Policy Optimization (GRPO). After convergence, the scorer is frozen for Stage 2.

Stage 2: Online Control from Progress-Time Consistency

At query steps (every $N = 15$ steps), we form a sliding window of recent observations, shuffle non-anchor frames $L = 2$ times, and query the frozen scorer. The scalar reward is the average Spearman correlation between predicted progress ranks and window timestamps:

$r_t = \mathrm{spr}(\mathbf{p}_t, \mathbf{q}_t) \in [-1, 1]$

This correlation is the sole reward for policy training. No extrinsic environment rewards are used at any point. For discrete PyBoy tasks we use Qwen2.5-VL-7B with VL-DAC and GAE; for continuous control (PointMaze, MetaWorld) we use DrQv2 as the policy backbone.

Results

PyBoy: Catrap and Kirby

We evaluate RTA on Catrap levels 2, 4, and 6, and Kirby level 0. Levels 4 and 6 require advanced reasoning and backtracking, while Kirby is a long-horizon control game. RTA is compared against Rank2Reward, VLM-RM, GVL, and oracle-reward baselines. RTA achieves the strongest performance across all Catrap levels and is the only method that attains a non-zero success rate on Kirby without any environment-provided rewards.

Cross-Domain Generalization

We train Stage-1 scorers on diverse video sources-YouTube GameBoy playthroughs, full-game Catrap/Kirby runs, MetaWorld demonstrations, and COIN AssembleSofa videos-and evaluate on PyBoy levels. Scorers trained on different visual domains still provide meaningful rewards, enabling agents to complete levels as well as or better than the baselines in Table 1. Generalization is also preserved on MetaWorld when the ranker is trained on YouTube or COIN data.

PointMaze-UMaze

To evaluate RTA beyond discrete tasks, we test it on PointMaze-UMaze with DrQv2 as the backbone, training for one million environment steps. We compare RTA with Rank2Reward (R2R), including an ablation that adds a GAIL-style reward to the ranking-based signal. RTA consistently outperforms R2R on UMaze, both in the ranking-only and mixed reward settings.

MetaWorld

On six MetaWorld manipulation tasks, Stage-1 RTA is trained once on a mixed dataset of demonstrations, while Rank2Reward is trained separately per task. RTA surpasses R2R on the majority of tasks under a matched compute budget. Ranking-only RTA can lag methods that incorporate an additional classifier-based (GAIL-style) reward on some tasks, likely because MetaWorld requires extensive exploration and RTA occasionally assigns high scores to plausible but unsuccessful trajectories. However, R2R relies on per-task pretraining, whereas RTA allows a single pretrained scorer to be reused across tasks.

Discussion

By learning policies directly from expert videos using a progress-time correlation signal, RTA suggests a scalable approach to reward-free learning that reduces the need for manually designed reward functions and may help avoid reward hacking.

Effect of Window Size on Cyclic Trajectories

We study how the windowed rank-correlation reward behaves under different temporal scales using cyclic trajectories composed of forward and reversed segments, explicitly removing global monotonic progress. We find scale-dependent behavior: (i) very small windows are overly sensitive to local temporal consistency and can produce monotonically increasing reward even in cyclic trajectories; (ii) intermediate windows suppress these local artifacts and better reflect the absence of global progress; and (iii) large windows lead to signal dilution due to temporal averaging. Despite these differences, all window sizes consistently assign high reward to the final segment where the goal is achieved.

Conclusion

We present Rank-Then-Act (RTA), a simple two-stage method that teaches vision-language models to act using only expert video and progress ranks (via Spearman correlation), with no extrinsic rewards. In both challenging discrete-action games and long-horizon continuous environments, this purely reward-free signal trains agents that achieve high success rates and outperform strong baselines. Because real-world rewards are often sparse or hard to design, learning from expert video alone offers a scalable path to multimodal agency while reducing risks like reward hacking. We see this as a step toward robust, generalist agents that can understand and act in open, dynamic, and underspecified environments.

Citation

@inproceedings{maksyuta2026rankthenact,
      title={Rank-Then-Act: Reward-Free Control from Frame-Order Progress},
      author={Yuriy Maksyuta and George Bredis and Ruslan Rakhimov and Daniil Gavrilov},
      booktitle={ICML 2026 Workshop on RLxF: Reinforcement Learning from World Feedback},
      year={2026},
      url={https://corl-team.github.io/rank-then-act/static/paper.pdf},
}