TokenGazer: Learning Where to Look in Token Space for Visuomotor Control

Checkpoints for TokenGazer. A learned gaze policy selects K of the frozen DINOv2 patch tokens per frame, and a diffusion policy acts on only those K tokens.

  • Stage 1: the gaze selector is trained with GRPO against a frozen action-prediction reward head. On LIBERO, the selector and the reward head also receive the task instruction.
  • Stage 2: the diffusion policy is trained on steps that alternate between the gaze's tokens and a random K-subset, and is deployed with the gaze's top-K.

Code: github.com/ostadabbas/TokenGazer. All policies are vision-only (no proprioception).

Rollouts

Selected rollouts from the greedy selector alone: among each benchmark's successful episodes, these are the ones whose gaze lands most often on the task-relevant parts of the scene (the T-block, the cubes, or the gripper and the instructed objects), so they show the gaze at its clearest rather than its typical behavior. Each clip shows the camera view next to the same frame with the gaze overlaid; selected patches are bright and the rest is dimmed. Cube and LIBERO clips are brightened uniformly for viewing.

PushT, K=16 Cube, K=8, both cubes placed
PushT Cube
LIBERO-Goal, K=8: "turn on the stove" LIBERO-10, K=8: "turn on the stove and put the moka pot on it"
LIBERO goal LIBERO 10
LIBERO-Object, K=8: "pick up the chocolate pudding and place it in the basket" LIBERO-Spatial, K=8: "pick up the black bowl next to the plate and place it on the plate"
LIBERO object LIBERO spatial

The LIBERO clips come from one multi-task policy trained on all four LIBERO suites (libero_4suite/). More clips, including PushT-Distract under live clutter, are in rollouts/.

Files

Each benchmark folder holds the selectors (Stage 1) and the policies (Stage 2) behind the paper's reported numbers.

Folder Selector (Stage 1) Policy (Stage 2) Paper result
pusht/ gazer_pusht_patchpolicy_K{16,64}_s{0,1,2}.pt dp_pusht_grpo_K{16,64}_s{0,1,2}_patchpolicy_mix50_np.pt score 0.80 (K=16), 0.85 (K=64), 3 seeds
pusht_distract/ gumbel_pusht_distract2_patchpolicy_K16_s{0,1,2}.pt dp_pusht_distract2_gumbel_K16_s{0,1,2}_patchpolicy.pt score 0.29 (K=16), 3 seeds
cube/ gazer_cube_double_action_K{8,16,32,64}_s0.pt, ..._K64_s{1,2}.pt dp_cube_double_grpo_K{16,32,64}_s0_action_noprop_mix50.pt, ..._K64_s{1,2}_..., and dp_cube_double_grpo_K8_s0_action_noprop_sample.pt 0.86 cubes placed of 2 (K=64), 3 seeds; 0.40 at K=8 (seed 0)
libero_goal/ grpo_libero_goal_action_K{8,16,32,64}_s5_langgaze.pt dp_libero_goal_grpo_K8_s{0,1,2}_action_langgaze_mix50_noprop.pt success 0.735 (K=8), 3 seeds
libero_4suite/ grpo_libero_4suite_action_K8_s5_langgaze_pc.pt (one camera-balanced selector, 4 tokens per camera) dp_libero_4suite_grpo_K8_s0_action_langgaze_pc_mix50_noprop.pt one multi-task policy on all four LIBERO suites: average success 0.663 over 3 seeds (seed 0 included), tied with random selection and above dense (0.606)

The reward heads (head_action_*.pt) are included for Cube and LIBERO. The PushT selectors were scored candidate subsets with a diffusion network trained beforehand on the full patch grid and used only as a reward model.

Loading

Each file is a torch.save dictionary. Selectors store state_dict and args. Stage-2 policies store the EMA weights under state_dict, plus args and the evaluation metrics. Load them with the release code:

from tokengazer.policy.train_diffusion import load_gazer  # selector -> gaze callable
# Stage-2 policies are evaluated with --eval-only in the per-benchmark train_diffusion scripts.

Evaluation protocol:

  • Replanning: one action per replan on Cube and LIBERO, eight on PushT.
  • Episodes: 50 on PushT, 100 on Cube, 20 per task on LIBERO.

License

MIT.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading