Instructions to use allura-org/GLM4-9B-Neon-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use allura-org/GLM4-9B-Neon-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="allura-org/GLM4-9B-Neon-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("allura-org/GLM4-9B-Neon-v2")
model = AutoModelForCausalLM.from_pretrained("allura-org/GLM4-9B-Neon-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use allura-org/GLM4-9B-Neon-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "allura-org/GLM4-9B-Neon-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-org/GLM4-9B-Neon-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/allura-org/GLM4-9B-Neon-v2

SGLang

How to use allura-org/GLM4-9B-Neon-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "allura-org/GLM4-9B-Neon-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-org/GLM4-9B-Neon-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "allura-org/GLM4-9B-Neon-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allura-org/GLM4-9B-Neon-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use allura-org/GLM4-9B-Neon-v2 with Docker Model Runner:
```
docker model run hf.co/allura-org/GLM4-9B-Neon-v2
```

What about refusals?

by MrDevolver - opened Apr 26, 2025

Discussion

MrDevolver

Apr 26, 2025

•

edited Apr 26, 2025

Hello,

models with roleplay as primary use usually perform better when they are free of restrictions of the refusals usually present in models for general purpose. The regular GLM-4 model is quite impressive, but unfortunately not very suitable for wide range of roleplay scenarios due to its tendency to resort to refusals.

My question is the following:
Is this GLM4-9B-Neon-v2 model a roleplay finetune on top of the regular, unmodified GLM-4 model, or was the model modified in any way to address the aforementioned issue?

LyraNovaHeart

Allura org Apr 26, 2025

Hi,

Great question! Given that the training data had lots of synth RP and short story data, this should alleviate most of the refusals, any remaining ones usually can be dealt with by system prompt. Your mileage may vary though

AuriAetherwiing

Allura org Apr 26, 2025

•

edited Apr 26, 2025

Model was trained on large quantity of uncensored roleplaying and short story gen data, so most refusals should be dealt with. Testing with master import settings, provided in this repo, I have not encountered any refusals.

MrDevolver

Apr 26, 2025

Thank you both. I consider the base GLM-4-0414 a very intelligent model for its size, in both 9B and 32B versions. Since I do a lot of roleplaying, I'm thrilled to test this particular intelligent model enhanced with roleplay data. If I find any issues with it regarding roleplay, I'll report them to you. Thank you again and thank you for using this particular model for creating this roleplaying model, I think it can help this model to gain more popularity. 😉👍

AuriAetherwiing

Allura org Apr 26, 2025

Hope you enjoy!
I'm planning to do a training run on 32B version soon

MrDevolver

Apr 26, 2025

Hope you enjoy!
I'm planning to do a training run on 32B version soon

Thank you. 32B version would be lovely. Not too suitable for my current hardware (even Q2_K of 32B is usually very slow for me, under 3 t/s), but I can imagine people with better hardware would have a good time with it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment