Instructions to use ibm-granite/granite-4.0-h-tiny-base-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ibm-granite/granite-4.0-h-tiny-base-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
Use Docker
docker model run hf.co/ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use ibm-granite/granite-4.0-h-tiny-base-GGUF with Ollama:
ollama run hf.co/ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use ibm-granite/granite-4.0-h-tiny-base-GGUF with Docker Model Runner:
docker model run hf.co/ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
- Lemonade
How to use ibm-granite/granite-4.0-h-tiny-base-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ibm-granite/granite-4.0-h-tiny-base-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.granite-4.0-h-tiny-base-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
granite-4.0-h-tiny-base (GGUF)
This repository contains models that have been converted to the GGUF format with various quantizations from an IBM Granite base model.
Please reference the base model's full model card here: https://huggingface.co/ibm-granite/granite-4.0-h-tiny-base
Running with llama.cpp
The examples below use the Q4_K_M quantization (granite-4.0-h-tiny-base-Q4_K_M.gguf). Other quantizations in this repository work the same way; only the file name changes. See the llama.cpp documentation for complete usage and options, including the llama cli and llama serve flag references.
example: llama cli
Start an interactive chat:
llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf
Or run a single prompt and exit:
llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf -p "Explain what a GGUF file is in two sentences." --single-turn
example: llama serve using OpenAI API
- Start llama serve with the model
llama serve -m granite-4.0-h-tiny-base-Q4_K_M.gguf \
--port 8080
- Send a chat request to the llama serve /v1/chat/completions endpoint
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{ "role": "user", "content": "Explain what a GGUF file is in two sentences." }
]
}'
The response is a JSON message with the generated text in the choices[0].message.content field.
llama.cpp recommended parameters for granite-4.0-h-tiny-base
Set the context length explicitly with -c / --ctx-size. While this model can support arbitrarily long context since it does not use positional embeddings, it was trained on text up to 128k, so we recommend limiting context to 131072 at runtime.
llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf -c 131072
llama serve -m granite-4.0-h-tiny-base-Q4_K_M.gguf -c 131072 --port 8080
Build Information
- llama.cpp build tag:
b11798 - Transformers version:
5.20.2
- Downloads last month
- 1,522
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Model tree for ibm-granite/granite-4.0-h-tiny-base-GGUF
Base model
ibm-granite/granite-4.0-h-tiny-base