Instructions to use leesm/llama-2-7b-hf-lora-oki10p with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leesm/llama-2-7b-hf-lora-oki10p with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="leesm/llama-2-7b-hf-lora-oki10p")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("leesm/llama-2-7b-hf-lora-oki10p")
model = AutoModelForCausalLM.from_pretrained("leesm/llama-2-7b-hf-lora-oki10p", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leesm/llama-2-7b-hf-lora-oki10p with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leesm/llama-2-7b-hf-lora-oki10p"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leesm/llama-2-7b-hf-lora-oki10p",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/leesm/llama-2-7b-hf-lora-oki10p

SGLang

How to use leesm/llama-2-7b-hf-lora-oki10p with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leesm/llama-2-7b-hf-lora-oki10p" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leesm/llama-2-7b-hf-lora-oki10p",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leesm/llama-2-7b-hf-lora-oki10p" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leesm/llama-2-7b-hf-lora-oki10p",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use leesm/llama-2-7b-hf-lora-oki10p with Docker Model Runner:
```
docker model run hf.co/leesm/llama-2-7b-hf-lora-oki10p
```

Llama-2 model fine tuning (TREX-Lab at Seoul Cyber University)

Summary

Base Model : meta-llama/Llama-2-7b-hf
Dataset : heegyu/open-korean-instructions (10%)
Tuning Method
- PEFT(Parameter Efficient Fine-Tuning)
- LoRA(Low-Rank Adaptation of Large Language Models)
Related Articles : https://arxiv.org/abs/2106.09685
Fine-tuning the Llama2 model with a random 10% of Korean chatbot data (open Korean instructions)
Test whether fine tuning of a large language model is possible on A30 GPU*1 (successful)

Developed by: [TREX-Lab at Seoul Cyber University]
Language(s) (NLP): [Korean]
Finetuned from model : [meta-llama/Llama-2-7b-hf]

Fine Tuning Detail

alpha value 16
r value 64 (it seems a bit big...@@)

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias='none',
    task_type='CAUSAL_LM'
)

Mixed precision : 4bit (bnb_4bit_use_double_quant)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype='float16',
)

Use SFT trainer (https://huggingface.co/docs/trl/sft_trainer)

trainer = SFTTrainer(
    model=peft_model,
    train_dataset=dataset,
    dataset_text_field='text',
    max_seq_length=min(tokenizer.model_max_length, 2048),
    tokenizer=tokenizer,
    packing=True,
    args=training_args
)

Train Result

time taken : executed in 2d 0h 17m

TrainOutput(global_step=2001,
            training_loss=0.6940358212922347,
            metrics={
               'train_runtime': 173852.2333,
               'train_samples_per_second': 0.092,
               'train_steps_per_second': 0.012,
               'train_loss': 0.6940358212922347,
               'epoch': 3.0})

Downloads last month: 14

Safetensors

Model size

7B params

Tensor type

F16

Dataset used to train leesm/llama-2-7b-hf-lora-oki10p

Paper for leesm/llama-2-7b-hf-lora-oki10p

LoRA: Low-Rank Adaptation of Large Language Models

Paper • 2106.09685 • Published Jun 17, 2021 • 65