karpathy

/

nanochat-d34

like 42

This PR will allow users to run the model in transformers. They can convert and test like so:

1. Download the nanochat-d34 checkpoint

hf download karpathy/nanochat-d34 --local-dir nanochat-d34

2. Convert the checkpoint to transformers format

uv run \
--with "transformers @ git+https://github.com/huggingface/transformers.git@nanochat-implementation" \
--with "tiktoken>=0.12.0" \
https://raw.githubusercontent.com/huggingface/transformers/nanochat-implementation/src/transformers/models/nanochat/convert_nanochat_checkpoints.py \
--input_dir ./nanochat-d34 \
--output_dir ./nanochat-d3-hf

3. (optional) Upload the checkpoint to the Hugging Face Hub

hf upload <username>/nanochat-d34 nanochat-d34

4. Test the model

import torch
from transformers import AutoTokenizer, NanoChatForCausalLM

tokenizer = AutoTokenizer.from_pretrained("./nanochat-d3-hf")
model = NanoChatForCausalLM.from_pretrained("./nanochat-d3-hf")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

prompt = "Hello, how are you?"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
inputs.pop("token_type_ids", None)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))