Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback
Abstract
Inhabitant enables a language-model agent to plan and execute long-horizon human-scene interactions by specifying motions spatially, measuring deviations, and revising control, achieving high task completion on the Inhabit benchmark.
Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7\% of the tasks with Claude Opus 5.5, and 93.3\% and 82.7\% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.
Community
Inhabitant is a training-free agent framework for long-horizon human-scene interaction. An LLM agent writes spatial control for a pretrained motion generator, measures the executed motion in a scene simulator, and revises until the task is done. On our new benchmark Inhabit (75 household tasks, 31 layouts), it completes 74 of 75 tasks with Claude Opus 5.5.
Project page: https://ryenhails.github.io/Inhabitant/
Code: https://github.com/yaozhang182/inhabitant
Benchmark: https://huggingface.co/datasets/zyyy12138/Inhabit
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation (2026)
- HITCH: Teaching a Vision-Language Model to Operate a Humanoid through a Whole-Body Harness (2026)
- Recursive Video In-Context Learning for Agentic Robot (2026)
- HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World (2026)
- World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal (2026)
- RoboQuest: Generalist Physical Agents that Search, Inspect and Test (2026)
- RoboTeam: Benchmarking Vision-Based Multi-Agent Control of Many Robot Arms in Real Time (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.25005 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper