CMF: new, memory-mappable format for quantized LLMs

Beginners
1

CMF: a single-file, memory-mappable format for quantized LLMs

Hi all,

I’ve been building CMF (Cortiq Model Format) single-file container for quantized models with a Rust runtime. Most of the converted models live on the Hub, so I wanted to share it here.

What one .cmf file holds

  • a fixed 128-byte envelope that addresses every section
  • header JSON: architecture, quantization defaults, chat template, provenance
  • a binary tensor directory with dtype, shape, offsets and a 64-bit hash per tensor
  • a page-aligned weight blob you can mmap and read in place
  • optional extras: tokenizer, task masks, and skill overlays (replacement tensors on top of one backbone)

Quantization is chosen per tensor: f16 / q8 / q8_2f for sensitive tensors, q4tp / q2tp for the large matrices. cortiq verify checks every section’s bounds and every tensor hash, and there is a small independent

Try it

cargo install cortiq-cli
cortiq convert --model Qwen/Qwen3-0.6B --quant q8 --output qwen.cmf
cortiq run qwen.cmf --prompt "What is the capital of France?" --greedy --no-think

Already have a GGUF: cortiq import-gguf model.gguf --output model.cmf. cortiq serve exposes an OpenAI-compatible API. It runs on CPU, and on GPU through Vulkan / DX12 (wgpu) or Metal.

On the Hub
53 converted models tagged cortiq: infosave (Oleg Kirichenko)

I’d welcome feedback on the format itself, especially from people who have written safetensors or GGUF loaders: what would you want in the header or tensor directory that isn’t there?

GitHub: GitHub - infosave2007/cmf: One backbone, many specialists. CMF is a self-describing, memory-mappable single-file format and a dependency-free runtime for quantized LLMs — zero-copy execution on CPU or GPU (Vulkan · Metal · DX12) that overlays task-specialized skills onto one shared model, with no per-model duplication. · GitHub

2

Interesting project! The memory-mapped design and per-tensor quantization are particularly worth exploring.

A few questions from an interoperability and reliability perspective:

  1. How do you handle endianness and compatibility across different CPU architectures?

  2. Is hash64 intended only for accidental-corruption detection, with signatures providing authenticity?

  3. How are unknown tensor dtypes, quantization schemes, and required features handled by older readers?

  4. Have you tested the independent Python reader against files generated by the Rust converter, including malformed or truncated files?

  5. Do you have reproducible benchmarks comparing cold-start time, peak RSS, and inference throughput against GGUF on identical models and hardware?

I’d be interested in exploring the format and its reader implementation. Thanks for sharing the specification!

3 
Thanks for the questions.

**1. Endianness.** Everything in the file is little-endian, period. Readers decode with explicit LE routines, so x86-64 and arm64 see the same bytes. Weights are read in place, no byte-swap path — big-endian hosts aren't supported and I don't plan to.

**2. hash64.** Yes. `hash64` is murmur3-style, not cryptographic. It's there to catch corruption and to dedup tensors between files. Authenticity is a separate detached `.sig`: Ed25519 over the file's SHA-256 (`cortiq sign` / `cortiq verify`).

**3. Unknown dtypes / features.** Fail fast. Unknown `required_features` bit, wrong version, unknown dtype id — the file is refused, no partial reads. New layouts get a new dtype id and, if they change how existing data is read, a feature bit. Header JSON is the additive part, unknown keys are ignored.

**4. Python reader.** The gates cover every converter dtype against the source weights, shards vs single file, hashes checked across Rust and Python, flipped bytes, bad magic/version/feature bits. On truncated files the Rust reader returns `Bounds` wherever the cut is. The Python reader catches cuts in the weight blob through tensor hashes but not in the tail sections, and it doesn't dequantize `q4tp`/`q2tp`/`q1t` yet — both on my list. No fuzzing so far.

**5. Benchmarks vs GGUF.** Not yet, honestly. `cortiq bench --json` gives throughput/TTFT/allocations, but it isn't comparable to `llama-bench` one-to-one, and I don't measure cold start or peak RSS. Plan is paired runs on the same checkpoint via `import-gguf`, scripts in the repo.

The reader is `python/cmf_reader.py`, ~350 lines, stdlib + numpy. Byte layouts are duplicated there on purpose so it works without any of our code. Happy to go through it.