Add CONVERSION_SUMMARY.md

CONVERSION_SUMMARY.md

@@ -0,0 +1,172 @@
+# IQuest-Loop-Instruct GGUF Conversion Summary
+
+**Date**: 2026-01-07
+**Model**: IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct
+**Achievement**: World's first IQuest-Loop-Instruct GGUF conversion
+
+## Files Created
+
+| File | Size | Format | SHA256 | Completion Time |
+|------|------|--------|--------|----------------|
+| IQuest-Coder-V1-40B-Loop-Instruct-f16.gguf | 75GB | F16 | `b70d3bb48753e786c8afca7556b818341fc9258e29083be4b0375c5a8b788289` | 2m 6s |
+| IQuest-Coder-V1-40B-Loop-Instruct-q4_k_m.gguf | 23GB | Q4_K_M | `b665999c8d6660ba0ea29cbbb072056052ef965a233ef65661ec16a16b39a9e3` | 2m 23s |
+| IQuest-Coder-V1-40B-Loop-Instruct-q5_k_m.gguf | 27GB | Q5_K_M | `a15814998038c8c6334f69bc11b776bce785350c933ce95fe9c41c4c7ec708ba` | 1m 41s |
+| IQuest-Coder-V1-40B-Loop-Instruct-q8_0.gguf | 40GB | Q8_0 | `a9323b7ca583a842737dd4ec1f7422101c68ededf2a86c75a8d5e9da70eaae06` | 53s |
+
+## Technical Implementation
+
+### Architecture Support
+
+Created `IQuestLoopCoderModel` class in llama.cpp's `convert_hf_to_gguf.py`:
+- Inherits from `LlamaModel` (compatible architecture base)
+- Maps 160 loop-specific `gate_projections` tensors to GGUF format
+- Preserves loop parameters in metadata:
+  - `llama.loop.num`: 2
+  - `llama.loop.window_size`: 64
+
+### Tensor Mapping
+
+**Gate Projections** (160 tensors total):
+- Source: `model.gate_projections.{0-79}.{weight|bias}`
+- Shape: `[128, 40]` weight + `[40]` bias per layer
+- Target: `blk.{layer}.loop_gate.{weight|bias}`
+- Quantization: Uses fallback q5_0/q5_1 for Q4_K_M/Q5_K_M (tensors too small for standard quantization)
+
+**Standard Tensors** (721 tensors):
+- Uses LlamaModel's standard tensor mapping
+- Attention: Q, K, V, Output projections
+- FFN: Gate, Up, Down projections
+- Normalization: Attention & FFN RMS norms
+
+## Conversion Statistics
+
+- **Total Tensors**: 883
+  - Standard Llama: 721
+  - Loop Gates: 160 (80 layers × 2 per layer)
+  - Embeddings: 2
+- **Vocabulary Size**: 76,800 tokens
+- **Context Length**: 131,072 tokens
+- **Hidden Layers**: 80
+- **Attention Heads**: 40 (8 KV heads)
+- **Hidden Size**: 5,120
+- **FFN Size**: 27,648
+
+## Current Status
+
+### What Works ✅
+
+1. **Conversion**: Successfully converts HuggingFace → GGUF F16
+2. **Quantization**: All standard quantization levels work (Q4_K_M, Q5_K_M, Q8_0, etc.)
+3. **Metadata**: Loop parameters correctly stored in GGUF metadata
+4. **Tensor Preservation**: All 883 tensors including loop gates successfully converted
+5. **Ollama Import**: Ollama accepts and imports the GGUF file
+
+### What Needs Work 🔧
+
+1. **Runtime Support**: llama.cpp runtime needs loop attention mechanism implementation
+2. **Inference**: Model loads but cannot run inference yet (loop gates not used)
+3. **Testing**: Need to validate loop attention behavior matches original PyTorch
+
+## Implementation Details
+
+### Modified Files
+
+**`/tmp/convert_hf_to_gguf.py`** (lines 2695-2733):
+```python
+@ModelBase.register("IQuestLoopCoderForCausalLM")
+class IQuestLoopCoderModel(LlamaModel):
+    """IQuest Loop Coder model with recurrent loop attention mechanism."""
+    model_arch = gguf.MODEL_ARCH.LLAMA
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.loop_num = self.hparams.get('loop_num', 2)
+        self.loop_window_size = self.hparams.get('loop_window_size', 64)
+
+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+        self.gguf_writer.add_uint32("llama.loop.num", self.loop_num)
+        self.gguf_writer.add_uint32("llama.loop.window_size", self.loop_window_size)
+
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
+        if "gate_projections" in name:
+            parts = name.split('.')
+            if len(parts) >= 4 and parts[1] == "gate_projections":
+                layer_num = parts[2]
+                param_type = parts[3]
+                new_name = f"blk.{layer_num}.loop_gate.{param_type}"
+                return [(new_name, data_torch)]
+        return super().modify_tensors(data_torch, name, bid)
+```
+
+## Next Steps for Community
+
+### For llama.cpp Maintainers
+
+1. **Implement Loop Attention Runtime**:
+   - Read `llama.loop.num` and `llama.loop.window_size` from GGUF metadata
+   - Load `blk.{layer}.loop_gate.{weight|bias}` tensors
+   - Implement recurrent loop attention mechanism in CUDA/CPU kernels
+   - Reference: Original implementation at IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct
+
+2. **Add Unit Tests**:
+   - Verify tensor loading
+   - Validate loop parameter reading
+   - Test against PyTorch reference implementation
+
+3. **Documentation**:
+   - Add Loop architecture to supported models list
+   - Document loop parameter usage
+   - Provide conversion examples
+
+### For Model Users
+
+1. **Wait for Runtime Support**: These GGUFs will work once llama.cpp implements loop attention
+2. **Use Regular Variant**: For immediate use, IQuest-Coder (non-Loop) is fully supported
+3. **Contribute**: Help implement loop attention in llama.cpp runtime
+
+## Performance Expectations (Once Runtime Supports Loop)
+
+Based on quantization levels:
+
+- **Q4_K_M (23GB)**: Recommended for most deployments, 30% of original size
+- **Q5_K_M (27GB)**: Better quality, 35% of original size
+- **Q8_0 (40GB)**: Excellent quality, 53% of original size, minimal loss
+- **F16 (75GB)**: Full precision reference
+
+## Docker Build System
+
+**Image**: `avarok/dgx-spark-complete:latest`
+**Base**: `dgx-vllm:cutlass-nvfp4-v15`
+**Includes**:
+- vLLM v15 with IQuest Loop Coder support
+- llama.cpp with CUDA support
+- Conversion scripts (convert_to_gguf.sh, quantize.sh)
+- Optimized for NVIDIA GB10 (SM 12.1)
+
+## References
+
+- **Original Model**: https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct
+- **llama.cpp Issue**: #18517 - Request for Loop-Instruct support
+- **PR Inspiration**: #18524 - Regular IQuestCoder support
+- **Debugging Journey**: /workspace/builds/DEBUGGING_JOURNEY.md
+
+## Credits
+
+- **Hardware**: Dual NVIDIA DGX Spark with GB10 GPUs
+- **Model**: IQuestLab team for Loop architecture innovation
+- **Tools**: llama.cpp (ggerganov), vLLM team
+- **First GGUF**: This conversion is the first Loop-Instruct variant in GGUF format
+
+## Verification
+
+SHA256 checksums provided for all files. Verify before use:
+```bash
+sha256sum IQuest-Coder-V1-40B-Loop-Instruct-*.gguf
+```
+
+---
+
+**Status**: Conversion successful, runtime support pending
+**Date**: 2026-01-07
+**Next**: Submit PR to llama.cpp with implementation + publish to HuggingFace