LLM-Drop

🤗 LLM-Drop hosts research artifacts for efficient foundation models, with a focus on large language models and unified multimodal models.

Our work studies how modern foundation models can be made more efficient while preserving their core capabilities. This page collects model weights, code links, project pages, and related resources from our research projects.

📌 Projects

🧩 LLM-Drop

Uncovering the Redundancy in Transformers via a Unified Study of Layer Dropping
TMLR 2026

• 📄 Paper: https://openreview.net/forum?id=1I7PCbOPfe
• 💻 Code: https://github.com/CASE-Lab-UMD/LLM-Drop
• 🤗 Models: https://huggingface.co/collections/LLM-Drop/llm-drop

🔍 Pruning on Representations

Demystifying When Pruning Works via Representation Hierarchies

• 🌐 Project Page: https://case-lab-umd.github.io/Pruning-on-Representations/
• 📄 Paper: https://arxiv.org/abs/2603.24652
• 💻 Code: https://github.com/CASE-Lab-UMD/Pruning-on-Representations

🌐 Sparse Unified Models

Understanding and Harnessing Sparsity in Unified Multimodal Models

• 🌐 Project Page: https://shwai-he.github.io/SparseUnifiedModel/
• 📄 Paper: https://huggingface.co/papers/2512.02351
• 💻 Code: https://github.com/Shwai-He/SparseUnifiedModel
• 🤗 Models:
  • https://huggingface.co/LLM-Drop/BAGEL-MoE-7B-GEN-16to8
  • https://huggingface.co/LLM-Drop/BAGEL-MoE-7B-GEN-32to16

📬 Contact

For questions or collaborations, please contact:

• Shwai He: shwaihe@umd.edu
• Guoheng Sun: ghsun@umd.edu