GoldenNet-Diwan-Analyzer-Qwen2.5-3B-v2

A specialized Arabic NLP model fine-tuned for analyzing Iraqi government correspondence. Built on top of Qwen2.5-3B-Instruct using QLoRA.

Model Description

GoldenNet-Diwan Analyzer is designed to process and analyze official Iraqi government letters and correspondence. It performs 7 specialized tasks and responds in structured JSON format.

Supported Tasks

Task	Description (Arabic)	Description (English)
تقييم الأولوية	Priority assessment	Assess urgency and priority level
تصنيف المراسلات	Correspondence classification	Classify document type/category
تحليل المشاعر	Sentiment analysis	Analyze tone and sentiment
استخراج الكيانات	Entity extraction	Extract named entities (people, orgs, dates, etc.)
التوجيه	Routing	Recommend appropriate department routing
استخراج المواعيد	Date/deadline extraction	Extract dates, deadlines, and timeframes
تصنيف السرية	Confidentiality classification	Classify security/confidentiality level

Training Details

Base Model

• Model: Qwen/Qwen2.5-3B-Instruct
• Architecture: Qwen2ForCausalLM (3.09B parameters, 36 layers)
• Precision: bfloat16

Fine-tuning Configuration

• Method: QLoRA (4-bit quantization + LoRA)
• LoRA Rank: 32
• LoRA Alpha: 64
• LoRA Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• LoRA Dropout: 0.05
• Quantization: 4-bit (bitsandbytes)

Training Hyperparameters

• Epochs: 5
• Batch Size: 2 (per device)
• Gradient Accumulation Steps: 8 (effective batch size: 16)
• Learning Rate: 2e-4
• LR Scheduler: Cosine
• Warmup Ratio: 0.05
• Weight Decay: 0.01
• Max Grad Norm: 0.3
• Optimizer: paged_adamw_8bit
• Cutoff Length: 1024 tokens

Dataset

• Format: ShareGPT conversation format
• Training Samples: 2,237
• Validation Samples: 252
• Language: Arabic (Iraqi government domain)

Training Results

Epoch	Train Loss	Eval Loss
1	0.1867	0.1803
2	0.1166	0.1563
3	0.0505	0.1515
4	0.0236	0.1623
5	0.0118	0.1716

• Best Checkpoint: Epoch 3 (eval_loss = 0.1515)
• Total Training Time: ~69 minutes
• Total FLOPs: 5.06e16
• Hardware: NVIDIA RTX 5070 Laptop (8GB VRAM)

Training Framework

• Framework: LLaMA-Factory
• PyTorch: 2.10.0+cu128
• CUDA: 12.9

Usage

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Alamori/GoldenNet-Diwan-Analyzer-Qwen2.5-3B-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

system_prompt = """أنت نظام GoldenNet-Diwan المتخصص في تحليل المراسلات الحكومية العراقية. يمكنك تنفيذ المهام التالية: تقييم الأولوية، تصنيف المراسلات، تحليل المشاعر، استخراج الكيانات، التوجيه، استخراج المواعيد، وتصنيف السرية. أجب بصيغة JSON فقط."""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "صنف المراسلة التالية:\n\nكتاب شكوى من مواطن بخصوص تأخر معاملته في دائرة الجنسية"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1, top_p=0.9)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

With Ollama (GGUF)

A quantized GGUF version (Q4_K_M, 1.8 GB) is available for use with Ollama:

ollama run GoldenNet-qwen2.53B_analyzer-v2

Limitations

• Optimized specifically for Iraqi government correspondence; may not generalize well to other Arabic text domains
• Best results with formal Arabic government writing style
• JSON output format may occasionally need post-processing

License

Apache 2.0 (following the base Qwen2.5 license)