evals/README.md

ModelChorus Multilingual Benchmark Evaluations

Automated benchmark evaluation framework for models on the ModelChorus platform, built on lm-evaluation-harness.

Setup

# Python 3.10+ required
cd evals/
pip install -r requirements.txt

Set your environment variables:

export OPENAI_API_KEY="sk-or-..."     # OpenRouter API key (default provider)
export HF_TOKEN="hf_..."              # HuggingFace token (for dataset access + result uploads)

For self-hosted models, set additional API keys as configured in eval_config.toml:

export FUNCTIONARY_API_KEY="sk-..."   # Example: self-hosted Functionary endpoint
export VLLM_API_KEY="sk-..."          # Example: vLLM deployment

Production token map (HF Jobs pipeline)

The weekly benchmark pipeline (dispatch.py → HF child jobs → Convex sync → Convex Slack alerts) uses several HF tokens, each scoped narrowly. Renewing a token in the wrong place produces a silent zero-benchmarks Sunday, so the inventory below is the source of truth.

Token name	Where it lives	Used by	Scope needed
HF_WRITE_TOKEN	GitHub repo secret	.github/workflows/mirror-evals-to-hf.yml	Write to meetkai/modelchorus-evals Space
HF_TOKEN_ADMIN	Local shell (one-shot, not stored)	evals/register_dispatcher.py	Manage scheduled jobs in meetkai
HF_TOKEN_RUNNER	Stored as a secret on the dispatcher schedule	Dispatcher (run_job calls) and every child job (HF_TOKEN in eval container)	Run jobs in meetkai and write to meetkai/modelchorus-eval-results
HF_API_TOKEN	Convex deployment env var	convex/benchmarkAlerts.ts (weekly cron)	Read jobs in meetkai
HF_TOKEN	Convex deployment env var (existing, unrelated to this pipeline)	convex/benchmarkSync.ts (existing weekly cron)	Read meetkai/modelchorus-eval-results dataset
EVAL_DISCOVERY_TOKEN	Same value on Convex env and dispatcher schedule secret (and local)	models:listForEvals (Convex query) ↔ evals/model_discovery.py	Shared secret — gates the alias-stripped roster query

When in doubt, hf auth whoami against the token confirms which org it has access to. Tokens are minted at https://huggingface.co/settings/tokens (use fine-grained for least-privilege scoping).

Model identifier conventions

The eval pipeline distinguishes two fields per model, with strict roles:

Field	Source	Example	Used for
model_id	Convex models.modelId; provider's canonical ID	openai/gpt-5-nano, meetkai:functionary-large	The actual provider API call; embedded in JSONL row metadata for DB attribution
name	Convex models.name; UI display string	OpenAI: GPT-5 Nano	Component of the routing slug; Slack output, log lines. models.alias is intentionally never used.
provider_name	Convex providers.name	OpenRouter, MeetKai Gateway Direct	Third slug component — disambiguates two rows that share both name and model_id
canonical slug	_hf_path_segment(f"{name}__{model_id}__{provider_name}")	OpenAI__GPT-5_Nano__openai_gpt-5-nano__OpenRouter	--models CLI arg, HF Job model label, HF dataset upload folder — single source of truth

run_eval.py --models X does per-arg fallback: try the full slug, then model_id, then name, evaluated independently per argument so mixed CLI invocations like --models openai/gpt-5-nano gpt-5-mini work in local dev. evals/dispatch.py always passes the full slug.

HF Job labels can't contain /, :, spaces, or other display characters, so the slug is sanitized via [^a-zA-Z0-9._-]+ → _, edge ._- chars stripped, falls back to "unknown" for empty input. The Convex query models:listForEvals (alias-stripped) is gated behind a shared-secret EVAL_DISCOVERY_TOKEN so callers can't bypass the blind-testing alias overlay.

Quick Start

# Run all models x all tasks
python run_eval.py

# Run a specific model
python run_eval.py --models gpt-5-mini

# Run a specific task group
python run_eval.py --tasks swahili_classification

# Run specific model + task
python run_eval.py --models gpt-5-mini --tasks english_classification

# Dry run (print config only, no API calls)
python run_eval.py --dry-run

Configuration

Edit eval_config.toml to configure models and tasks.

Adding a model

# OpenRouter model (uses default base_url)
[[models]]
name = "my-model"
model_id = "provider/model-name"

# Self-hosted model (custom endpoint + API key)
[[models]]
name = "my-local-model"
model_id = "my-model-id"
base_url = "https://my-endpoint.example.com/v1/chat/completions"
api_key_env = "MY_API_KEY"
custom_headers = { "X-Custom-Header" = "value" }

Adding a task group

[[tasks]]
name = "my_task_group"

Available Benchmarks

Language	Benchmarks	Group Names
English	SIB-200, Belebele, MGSM	english_classification, english_mcq, english_math
Spanish	AQuAS, XQuAD, MGSM + more	spanish, spanish_aquas, spanish_qa
Swahili	SIB-200, AfriMMLU, Belebele + more	swahili, swahili_classification, swahili_mcq, swahili_qa
French	SIB-200, Belebele, MGSM	french_classification, french_mcq, french_math
Arabic	SIB-200, Belebele	arabic_classification, arabic_mcq
Hausa	SIB-200, AfriMMLU, Belebele	hausa_classification, hausa_mcq

Adding a New Language/Benchmark

1. Create a new directory: evals/<language>/
2. Create task YAML files following the lm-eval task guide
3. Add a root group YAML (e.g., <language>.yaml) that aggregates subtasks
4. Add utils.py with any custom processing functions
5. Add the task group to eval_config.toml

Use existing language folders as reference for the YAML structure.

Multi-Provider Support

Each model in eval_config.toml can specify its own provider:

• base_url — API endpoint (defaults to OpenRouter)
• api_key_env — environment variable name for the API key (defaults to OPENAI_API_KEY)
• custom_headers — additional HTTP headers

This maps to how the ModelChorus platform supports multiple providers (openrouter, openai_compatible) with custom endpoints.

Results

Results are saved to evals/output/results/ by default. When HF Hub upload is enabled in eval_config.toml, results are also pushed to the configured HuggingFace dataset repository.

Scoring Metrics

Metric	Description	Used By
exact_match	Case-sensitive string match	MCQ, classification
f1_macro	Macro-averaged F1 across labels	Classification tasks
sas_encoder	Cosine similarity via sentence-transformers	Spanish QA
llm_judge_score	LLM-as-judge (gpt-5-mini)	QA tasks
rouge1	ROUGE-1 F-measure	Summarization