Instructions to use tiiuae/falcon-mamba-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tiiuae/falcon-mamba-7b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="tiiuae/falcon-mamba-7b")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b") model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use tiiuae/falcon-mamba-7b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "tiiuae/falcon-mamba-7b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-mamba-7b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/tiiuae/falcon-mamba-7b
- SGLang
How to use tiiuae/falcon-mamba-7b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-mamba-7b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-mamba-7b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "tiiuae/falcon-mamba-7b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "tiiuae/falcon-mamba-7b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use tiiuae/falcon-mamba-7b with Docker Model Runner:
docker model run hf.co/tiiuae/falcon-mamba-7b
Commit History
Update README.md 2a2ae6e verified
Update README.md 856badf verified
Update README.md 6ddf4e2 verified
Update README.md 09172dd verified
Chore: update Nemo12B eval results b8d360a verified
chore: add rwkv-v6 eval results 2cf50a0 verified
Update README.md a2f4009 verified
Update config.json 503c3d4 verified
Update tokenizer_config.json b0a607a verified
Update generation_config.json 49b4ef3 verified
Update generation_config.json 0464d38 verified
Update config.json a3b68db verified
Update tokenizer_config.json 2d34e76 verified
Update tokenizer_config.json f48fe69 verified
Update tokenizer_config.json da17011 verified
Update generation_config.json 04eb035 verified
Update config.json d6f50d2 verified
Update README.md 4a2cff9 verified
Update README.md 1e2f267 verified
Update README.md 4c94e06 verified
Update README.md ef7ba29 verified
Update license c2a45ff verified
Adding Evaluation Results (#4) acc82ee verified
Update README.md 230022c verified
Update README.md 7f04837 verified
Update README.md 9bc71f5 verified
Update README.md d58940c verified
Update README.md e79efbe verified
Update README.md e93d9bf verified
Update README.md c78f432 verified
Update README.md 3adb85d verified
Update README.md 2b10f59 verified
Update README.md 3f922ac verified
Update README.md 7952d52 verified
Update README.md bdad334 verified
Update README.md ded105d verified
Update README.md 54ec485 verified
Update README.md d2cd831 verified
Update README.md 02688fc verified
Update README.md 124f971 verified
Update README.md a011b2a verified
Update README.md 4141e0d verified
Update README.md 8c8f700 verified
Update README.md 89fee4a verified
Update README.md 1afab62 verified
Update README.md 61fb990 verified
Ilyas Chahed commited on