TokenGazer: Learning Where to Look in Token Space for Visuomotor Control
Checkpoints for TokenGazer. A learned gaze policy selects K of the frozen DINOv2 patch tokens per frame, and a diffusion policy acts on only those K tokens.
- Stage 1: the gaze selector is trained with GRPO against a frozen action-prediction reward head. On LIBERO, the selector and the reward head also receive the task instruction.
- Stage 2: the diffusion policy is trained on steps that alternate between the gaze's tokens and a random K-subset, and is deployed with the gaze's top-K.
Code: github.com/ostadabbas/TokenGazer. All policies are vision-only (no proprioception).
Rollouts
Selected rollouts from the greedy selector alone: among each benchmark's successful episodes, these are the ones whose gaze lands most often on the task-relevant parts of the scene (the T-block, the cubes, or the gripper and the instructed objects), so they show the gaze at its clearest rather than its typical behavior. Each clip shows the camera view next to the same frame with the gaze overlaid; selected patches are bright and the rest is dimmed. Cube and LIBERO clips are brightened uniformly for viewing.
| LIBERO-Goal, K=8: "turn on the stove" | LIBERO-10, K=8: "turn on the stove and put the moka pot on it" |
|---|---|
![]() |
![]() |
| LIBERO-Object, K=8: "pick up the chocolate pudding and place it in the basket" | LIBERO-Spatial, K=8: "pick up the black bowl next to the plate and place it on the plate" |
|---|---|
![]() |
![]() |
The LIBERO clips come from one multi-task policy trained on all four LIBERO suites
(libero_4suite/). More clips, including PushT-Distract under live clutter, are in rollouts/.
Files
Each benchmark folder holds the selectors (Stage 1) and the policies (Stage 2) behind the paper's reported numbers.
| Folder | Selector (Stage 1) | Policy (Stage 2) | Paper result |
|---|---|---|---|
pusht/ |
gazer_pusht_patchpolicy_K{16,64}_s{0,1,2}.pt |
dp_pusht_grpo_K{16,64}_s{0,1,2}_patchpolicy_mix50_np.pt |
score 0.80 (K=16), 0.85 (K=64), 3 seeds |
pusht_distract/ |
gumbel_pusht_distract2_patchpolicy_K16_s{0,1,2}.pt |
dp_pusht_distract2_gumbel_K16_s{0,1,2}_patchpolicy.pt |
score 0.29 (K=16), 3 seeds |
cube/ |
gazer_cube_double_action_K{8,16,32,64}_s0.pt, ..._K64_s{1,2}.pt |
dp_cube_double_grpo_K{16,32,64}_s0_action_noprop_mix50.pt, ..._K64_s{1,2}_..., and dp_cube_double_grpo_K8_s0_action_noprop_sample.pt |
0.86 cubes placed of 2 (K=64), 3 seeds; 0.40 at K=8 (seed 0) |
libero_goal/ |
grpo_libero_goal_action_K{8,16,32,64}_s5_langgaze.pt |
dp_libero_goal_grpo_K8_s{0,1,2}_action_langgaze_mix50_noprop.pt |
success 0.735 (K=8), 3 seeds |
libero_4suite/ |
grpo_libero_4suite_action_K8_s5_langgaze_pc.pt (one camera-balanced selector, 4 tokens per camera) |
dp_libero_4suite_grpo_K8_s0_action_langgaze_pc_mix50_noprop.pt |
one multi-task policy on all four LIBERO suites: average success 0.663 over 3 seeds (seed 0 included), tied with random selection and above dense (0.606) |
The reward heads (head_action_*.pt) are included for Cube and LIBERO. The PushT selectors were
scored candidate subsets with a diffusion network trained beforehand on the full patch grid and used only as a reward model.
Loading
Each file is a torch.save dictionary. Selectors store state_dict and args. Stage-2 policies
store the EMA weights under state_dict, plus args and the evaluation metrics. Load them with
the release code:
from tokengazer.policy.train_diffusion import load_gazer # selector -> gaze callable
# Stage-2 policies are evaluated with --eval-only in the per-benchmark train_diffusion scripts.
Evaluation protocol:
- Replanning: one action per replan on Cube and LIBERO, eight on PushT.
- Episodes: 50 on PushT, 100 on Cube, 20 per task on LIBERO.
License
MIT.





