| ---
|
| language: en
|
| tags:
|
| - jamba
|
| - lora
|
| - chat
|
| - fine-tuning
|
| license: apache-2.0
|
| ---
|
|
|
| # Jamba Chat LoRA
|
|
|
| This is a LoRA fine-tuned version of the Jamba model trained on chat conversations.
|
|
|
| ## Model Description
|
|
|
| - **Base Model:** LaferriereJC/jamba_550M_trained
|
| - **Training Data:** UltraChat dataset
|
| - **Task:** Conversational AI
|
| - **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
|
|
|
| ## Usage
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
| from peft import PeftModel, PeftConfig
|
|
|
| # Load the model
|
| model = AutoModelForCausalLM.from_pretrained(
|
| "LaferriereJC/jamba_550M_trained",
|
| trust_remote_code=True
|
| )
|
| model = PeftModel.from_pretrained(model, "your-username/jamba-chat-lora")
|
|
|
| # Load the tokenizer
|
| tokenizer = AutoTokenizer.from_pretrained("LaferriereJC/jamba_550M_trained")
|
|
|
| # Example usage
|
| text = "User: How are you today?\nAssistant:"
|
| inputs = tokenizer(text, return_tensors="pt")
|
| outputs = model.generate(**inputs, max_length=100)
|
| response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
| print(response)
|
| ```
|
|
|
| ## Training Details
|
|
|
| - **Training Data:** UltraChat dataset (subset)
|
| - **LoRA Config:**
|
| - Rank: 16
|
| - Alpha: 32
|
| - Target Modules: Last layer feed forward experts
|
| - Dropout: 0.1
|
| - **Training Parameters:**
|
| - Learning Rate: 5e-4
|
| - Optimizer: AdamW (32-bit)
|
| - LR Scheduler: Cosine
|
| - Warmup Ratio: 0.03
|
|
|
