# ModelChorus Multilingual Benchmark Evaluations

Automated benchmark evaluation framework for models on the ModelChorus platform, built on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

## Setup

```bash
# Python 3.10+ required
cd evals/
pip install -r requirements.txt
```

Set your environment variables:

```bash
export OPENAI_API_KEY="sk-or-..."     # OpenRouter API key (default provider)
export HF_TOKEN="hf_..."              # HuggingFace token (for dataset access + result uploads)
```

For self-hosted models, set additional API keys as configured in `eval_config.toml`:

```bash
export FUNCTIONARY_API_KEY="sk-..."   # Example: self-hosted Functionary endpoint
export VLLM_API_KEY="sk-..."          # Example: vLLM deployment
```

## Production token map (HF Jobs pipeline)

The weekly benchmark pipeline (`dispatch.py` → HF child jobs → Convex sync → Convex Slack alerts) uses several HF tokens, each scoped narrowly. Renewing a token in the wrong place produces a silent zero-benchmarks Sunday, so the inventory below is the source of truth.

| Token name             | Where it lives                                                              | Used by                                                                             | Scope needed                                                              |
| ---------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `HF_WRITE_TOKEN`       | GitHub repo secret                                                          | `.github/workflows/mirror-evals-to-hf.yml`                                          | Write to `meetkai/modelchorus-evals` Space                                |
| `HF_TOKEN_ADMIN`       | Local shell (one-shot, not stored)                                          | `evals/register_dispatcher.py`                                                      | Manage scheduled jobs in `meetkai`                                        |
| `HF_TOKEN_RUNNER`      | Stored as a secret on the dispatcher schedule                               | Dispatcher (`run_job` calls) **and** every child job (`HF_TOKEN` in eval container) | Run jobs in `meetkai` **and** write to `meetkai/modelchorus-eval-results` |
| `HF_API_TOKEN`         | Convex deployment env var                                                   | `convex/benchmarkAlerts.ts` (weekly cron)                                           | Read jobs in `meetkai`                                                    |
| `HF_TOKEN`             | Convex deployment env var (existing, unrelated to this pipeline)            | `convex/benchmarkSync.ts` (existing weekly cron)                                    | Read `meetkai/modelchorus-eval-results` dataset                           |
| `EVAL_DISCOVERY_TOKEN` | **Same value** on Convex env **and** dispatcher schedule secret (and local) | `models:listForEvals` (Convex query) ↔ `evals/model_discovery.py`                   | Shared secret — gates the alias-stripped roster query                     |

When in doubt, `hf auth whoami` against the token confirms which org it has access to. Tokens are minted at https://huggingface.co/settings/tokens (use fine-grained for least-privilege scoping).

## Model identifier conventions

The eval pipeline distinguishes two fields per model, with strict roles:

| Field           | Source                                                     | Example                                             | Used for                                                                                                |
| --------------- | ---------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `model_id`      | Convex `models.modelId`; provider's canonical ID           | `openai/gpt-5-nano`, `meetkai:functionary-large`    | The actual provider API call; embedded in JSONL row metadata for DB attribution                         |
| `name`          | Convex `models.name`; UI display string                    | `OpenAI: GPT-5 Nano`                                | Component of the routing slug; Slack output, log lines. **`models.alias` is intentionally never used.** |
| `provider_name` | Convex `providers.name`                                    | `OpenRouter`, `MeetKai Gateway Direct`              | Third slug component — disambiguates two rows that share both name and model_id                         |
| canonical slug  | `_hf_path_segment(f"{name}__{model_id}__{provider_name}")` | `OpenAI__GPT-5_Nano__openai_gpt-5-nano__OpenRouter` | `--models` CLI arg, HF Job `model` label, HF dataset upload folder — single source of truth             |

`run_eval.py --models X` does per-arg fallback: try the full slug, then `model_id`, then `name`, evaluated independently per argument so mixed CLI invocations like `--models openai/gpt-5-nano gpt-5-mini` work in local dev. `evals/dispatch.py` always passes the full slug.

HF Job labels can't contain `/`, `:`, spaces, or other display characters, so the slug is sanitized via `[^a-zA-Z0-9._-]+ → _`, edge `._-` chars stripped, falls back to `"unknown"` for empty input. The Convex query `models:listForEvals` (alias-stripped) is gated behind a shared-secret `EVAL_DISCOVERY_TOKEN` so callers can't bypass the blind-testing alias overlay.

## Quick Start

```bash
# Run all models x all tasks
python run_eval.py

# Run a specific model
python run_eval.py --models gpt-5-mini

# Run a specific task group
python run_eval.py --tasks swahili_classification

# Run specific model + task
python run_eval.py --models gpt-5-mini --tasks english_classification

# Dry run (print config only, no API calls)
python run_eval.py --dry-run
```

## Configuration

Edit `eval_config.toml` to configure models and tasks.

### Adding a model

```toml
# OpenRouter model (uses default base_url)
[[models]]
name = "my-model"
model_id = "provider/model-name"

# Self-hosted model (custom endpoint + API key)
[[models]]
name = "my-local-model"
model_id = "my-model-id"
base_url = "https://my-endpoint.example.com/v1/chat/completions"
api_key_env = "MY_API_KEY"
custom_headers = { "X-Custom-Header" = "value" }
```

### Adding a task group

```toml
[[tasks]]
name = "my_task_group"
```

## Available Benchmarks

| Language | Benchmarks                         | Group Names                                                      |
| -------- | ---------------------------------- | ---------------------------------------------------------------- |
| English  | SIB-200, Belebele, MGSM            | `english_classification`, `english_mcq`, `english_math`          |
| Spanish  | AQuAS, XQuAD, MGSM + more          | `spanish`, `spanish_aquas`, `spanish_qa`                         |
| Swahili  | SIB-200, AfriMMLU, Belebele + more | `swahili`, `swahili_classification`, `swahili_mcq`, `swahili_qa` |
| French   | SIB-200, Belebele, MGSM            | `french_classification`, `french_mcq`, `french_math`             |
| Arabic   | SIB-200, Belebele                  | `arabic_classification`, `arabic_mcq`                            |
| Hausa    | SIB-200, AfriMMLU, Belebele        | `hausa_classification`, `hausa_mcq`                              |

## Adding a New Language/Benchmark

1. Create a new directory: `evals/<language>/`
2. Create task YAML files following the [lm-eval task guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md)
3. Add a root group YAML (e.g., `<language>.yaml`) that aggregates subtasks
4. Add `utils.py` with any custom processing functions
5. Add the task group to `eval_config.toml`

Use existing language folders as reference for the YAML structure.

## Multi-Provider Support

Each model in `eval_config.toml` can specify its own provider:

- `base_url` — API endpoint (defaults to OpenRouter)
- `api_key_env` — environment variable name for the API key (defaults to `OPENAI_API_KEY`)
- `custom_headers` — additional HTTP headers

This maps to how the ModelChorus platform supports multiple providers (`openrouter`, `openai_compatible`) with custom endpoints.

## Results

Results are saved to `evals/output/results/` by default. When HF Hub upload is enabled in `eval_config.toml`, results are also pushed to the configured HuggingFace dataset repository.

## Scoring Metrics

| Metric            | Description                                 | Used By              |
| ----------------- | ------------------------------------------- | -------------------- |
| `exact_match`     | Case-sensitive string match                 | MCQ, classification  |
| `f1_macro`        | Macro-averaged F1 across labels             | Classification tasks |
| `sas_encoder`     | Cosine similarity via sentence-transformers | Spanish QA           |
| `llm_judge_score` | LLM-as-judge (gpt-5-mini)                   | QA tasks             |
| `rouge1`          | ROUGE-1 F-measure                           | Summarization        |