granite-4.0-h-tiny-base (GGUF)

This repository contains models that have been converted to the GGUF format with various quantizations from an IBM Granite base model.

Please reference the base model's full model card here: https://huggingface.co/ibm-granite/granite-4.0-h-tiny-base

Running with llama.cpp

The examples below use the Q4_K_M quantization (granite-4.0-h-tiny-base-Q4_K_M.gguf). Other quantizations in this repository work the same way; only the file name changes. See the llama.cpp documentation for complete usage and options, including the llama cli and llama serve flag references.

example: llama cli

Start an interactive chat:

llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf

Or run a single prompt and exit:

llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf -p "Explain what a GGUF file is in two sentences." --single-turn

example: llama serve using OpenAI API

  1. Start llama serve with the model
llama serve -m granite-4.0-h-tiny-base-Q4_K_M.gguf \
  --port 8080
  1. Send a chat request to the llama serve /v1/chat/completions endpoint
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Explain what a GGUF file is in two sentences." }
    ]
  }'

The response is a JSON message with the generated text in the choices[0].message.content field.


llama.cpp recommended parameters for granite-4.0-h-tiny-base

Set the context length explicitly with -c / --ctx-size. While this model can support arbitrarily long context since it does not use positional embeddings, it was trained on text up to 128k, so we recommend limiting context to 131072 at runtime.

llama cli -m granite-4.0-h-tiny-base-Q4_K_M.gguf -c 131072
llama serve -m granite-4.0-h-tiny-base-Q4_K_M.gguf -c 131072 --port 8080

Build Information

  • llama.cpp build tag: b11798
  • Transformers version: 5.20.2
Downloads last month
1,522
GGUF
Model size
7B params
Architecture
granitehybrid
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ibm-granite/granite-4.0-h-tiny-base-GGUF

Quantized
(5)
this model

Collections including ibm-granite/granite-4.0-h-tiny-base-GGUF