Instructions to use DisOOM/Yi-70B-200k-RPMerge-Franken with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DisOOM/Yi-70B-200k-RPMerge-Franken with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DisOOM/Yi-70B-200k-RPMerge-Franken")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DisOOM/Yi-70B-200k-RPMerge-Franken")
model = AutoModelForCausalLM.from_pretrained("DisOOM/Yi-70B-200k-RPMerge-Franken")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DisOOM/Yi-70B-200k-RPMerge-Franken with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DisOOM/Yi-70B-200k-RPMerge-Franken"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DisOOM/Yi-70B-200k-RPMerge-Franken",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DisOOM/Yi-70B-200k-RPMerge-Franken

SGLang

How to use DisOOM/Yi-70B-200k-RPMerge-Franken with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DisOOM/Yi-70B-200k-RPMerge-Franken" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DisOOM/Yi-70B-200k-RPMerge-Franken",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DisOOM/Yi-70B-200k-RPMerge-Franken" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DisOOM/Yi-70B-200k-RPMerge-Franken",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DisOOM/Yi-70B-200k-RPMerge-Franken with Docker Model Runner:
```
docker model run hf.co/DisOOM/Yi-70B-200k-RPMerge-Franken
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Yi-70B-200k-RPMerge-Franken

--This is a 71B frankenmerge of Yi-34B-200K-RPMerge created by interleaving layers of Yi-34B-200K-RPMerge with itself using mergekit.--

By attempting to merge the yi-34B (RPMerge, which I consider to be a better-performing version), to create a 70B-level Yi, what surprised me was that it didn't seem to exhibit the increased logical confusion and linguistic errors that many models with more than double the original parameter count do. It appeared to just get stronger with the increase in parameters. I also tried several other fine-tuned versions of Yi, and the results were satisfactory.

-Quantize

GGUF Here：Coming soon

-Merge Configuration

This yaml below:

dtype: float16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 4]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [4, 14]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [8, 18]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [12, 22]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [16, 26]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [20, 30]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [24, 34]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [28, 38]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [32, 42]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [36, 46]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [40, 50]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [44, 54]
    model: brucethemoose/Yi-34B-200K-RPMerge
- sources:
  - layer_range: [48, 60]
    model: brucethemoose/Yi-34B-200K-RPMerge

-Performance

Tips:I don't have the capability to conduct benchmark tests, nor can I even use it extensively enough, so my test results might not be accurate.

It has better performance than the 34B version in most of my own tests (subjective) including comprehension, reasoning and coherence and also writing skills. If you believe in this model's performance, feel free to test it out or offer evaluations. Everyone's tests or evaluations are welcome.

Downloads last month: 3

Safetensors

Model size

71B params

Tensor type

F16

Model tree for DisOOM/Yi-70B-200k-RPMerge-Franken

Quantizations

2 models