TULLUS/paligemma2-3b-lerobot-adapter

PaliGemma 2 (3B-448) SO-100 Pick & Place Adapter

This repository contains a PEFT LoRA adapter fine-tuned on top of Google's PaliGemma 2 3B (448px) pre-trained vision-language model. It is configured for visual action prediction and end-to-end motor trajectory generation for the SO-100 6-DOF robotic arm.

Model Summary

• Adapter Repository: TULLUS/paligemma2-3b-lerobot-adapter
• Base Model: google/paligemma2-3b-pt-448
• Architecture: PEFT / LoRA
• Target Hardware: SO-100 6-DOF Robotic Arm (Feetech STS3215 bus motors)
• Framework Integration: LeRobot (lerobot) & Hugging Face Transformers
• Training Dataset: lerobot/svla_so100_pickplace
• Output: Continuous 6-DOF motor angle float string predictions

---

Quickstart & Usage

1. Installation

pip install transformers peft torch pillow

2. Loading the Model with Adapter

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForMultimodalLM
from peft import PeftModel

base_model_id = "google/paligemma2-3b-pt-448"
adapter_id = "TULLUS/paligemma2-3b-lerobot-adapter"

# Load base model & processor
processor = AutoProcessor.from_pretrained(base_model_id)
base_model = AutoModelForMultimodalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Attach the LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# Prepare inputs
image = Image.open("camera_frame.jpg")
prompt = "pick and place"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)

# Generate raw action string
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=64)

action_text = processor.decode(output[0], skip_special_tokens=True)
print("Predicted motor actions:", action_text)

Base Model Compatibility Note: This adapter was trained specifically on the raw pre-trained (pt) base model (google/paligemma2-3b-pt-448). Using this adapter with instruction-tuned or mixed (mix) checkpoints is not recommended, as shifted internal weight spaces and token distributions can degrade the precise coordinate output formatting required for robotic control.

---

Action Space & Joint Mapping

The model produces space-delimited string outputs containing continuous joint position targets:

0.707 -87.188 74.619 72.598 89.686 0.106

These values map sequentially to the 6 motor channels on the SO-100 arm:

Index	Motor / Joint	Description
1	shoulder_pan	Base rotation
2	shoulder_lift	Lower arm pitch
3	elbow_flex	Upper arm pitch
4	wrist_flex	Wrist pitch
5	wrist_roll	Wrist roll
6	gripper	End-effector open/close

---

Hardware Execution & Action Smoothing

To prevent sudden torque spikes or jerky motion on physical hardware, predicted action vectors should be passed through an Exponential Moving Average (EMA) joint smoother (e.g. PerJointSmoother) before dispatching commands to the motor bus via lerobot.common.robot_devices.robots.feetech.

---

Disclaimer & Safety Warning

Use at Your Own Risk: This model generates experimental machine learning control policies for physical robotic hardware. The authors assume no responsibility or liability for any hardware damage, servo burnout, mechanical wear, property damage, or personal injury that may occur during live hardware testing or autonomous execution. Always keep an emergency stop (E-stop) handy, test with low motor torque limits, and supervise physical robot operations closely.

---

Citation

@article{
    title={PaliGemma 2: A Family of Versatile VLMs for Transfer},
    author={Andreas Steiner and André Susano Pinto and Michael Tschannen and Daniel Keysers and Xiao Wang and Yonatan Bitton and Alexey Gritsenko and Matthias Minderer and Anthony Sherbondy and Shangbang Long and Siyang Qin and Reeve Ingle and Emanuele Bugliarello and Sahar Kazemzadeh and Thomas Mesnard and Ibrahim Alabdulmohsin and Lucas Beyer and Xiaohua Zhai},
    year={2024},
    journal={arXiv preprint arXiv:2412.03555}
}

---

Trained and maintained by TULLUS.