Leaderboard / AGENTS.md
schaeferbasti's picture
Leaderboard v0.12.0: vector PDF downloads (jsPDF, svg2pdf.js, embedded font)
78a8a90
|
Raw History Blame Contribute Delete
6.62 kB

AGENTS.md

The SelectArena leaderboard: a static Hugging Face Space (sdk: static) that serves files generated in the SelectArena repo. Static Spaces run no compute, so they are free; do not switch it to Gradio or Docker (those need a paid plan).

Files

  • index.html: the whole page: header (on screens wider than 640px it stays at the top together with the tabs, see initStickyHeader in app.js), the four tab panels (Leaderboard, which opens first: the Elo figure above the table; Performance: Performance-Time Trade-Off (Pareto front), performance per dataset characteristic, model, task type and budget, Budget- and Time-constraint Ablations and complete results; Stability; Validity), all user-facing text (each explanation sits next to the figure or table it describes; there is no About tab; the Paper and GitHub links are the header buttons), the citation (always visible in the footer) and <meta name="selectarena-version"> (not displayed; there is no version history, the commit messages are the changelog).
  • assets/app.js: plain JavaScript, no build step. Fetches data/meta.json, renders the selector chips (task type, downstream model, budget; combinations that are not published are disabled; shared by the Leaderboard and Performance Trade-offs tabs; task type, view and noise level of the lasagna plots; each tab's chips sit in a collapsed <details class="config"> whose "Customize" button shows the current choices), sets the iframes' src and the figures' <img src> to the generated files and resizes the iframes when a page posts {saFrameHeight}. Keeps the tab and the selections in the URL hash (tab keys leaderboard, tradeoffs, stability, validity; old names such as summary, performance, data and about still work).
  • assets/downloads.js: SVG / PNG / PDF downloads of the figures drawn in index.html (the .fig-dl buttons of the 22-hour and Lasso/ElasticNet ablation figures, and of Figure D.3), in the light paper look, and the PDFs of the generated figure pages, which send their SVG (message {saPdf}, answered by {saPdfAck}). The PDFs are vector PDFs made with jsPDF and svg2pdf.js, with the text in SelectArena Sans (a renamed subset of Liberation Sans) embedded; these files in assets/vendor/ (see its README) are loaded at the first PDF download only. Where they cannot be loaded, or a figure page has no leaderboard around it, the PDF holds the figure rendered at 4x.
  • assets/style.css: color tokens on :root (light) and for dark mode, which follows the system setting unless the theme button (after "Paper", "Cite" and "Add your method") set data-theme on <html>. The choice is kept in localStorage (selectarena-theme) and passed on to the embedded pages, whose CSS (SelectArena's website/html.py) honours the same data-theme attribute.
  • smoke_test.py: standard-library check of data/ and of the version tag (see below).

Data contract

data/ is SelectArena's generated_website_artifacts/clean_website_artifacts/website_data/:

  • meta.json: counts, models (key -> name), budgets, total_budget, task_types, has_stability, has_validity, validity_noise, validity_tables, generated_at, subsets (each with tasks, model, budget, path, n_methods, n_datasets, n_comparisons, pareto), lasagna (variants, variant_labels, and when present stability = {path, task_types} and validity = a list of {noise, path, task_types}), analysis (path, groupings = a list of {key, label}, improvement_by_model, improvement_by_task_type, rank_by_size, elo_without_timeouts) and has_results_explorer.
  • subsets/tasks_<t>/models_<m>/budgets_<b>/: website_leaderboard.csv, leaderboard_table.html, elo_explorer.html, and pareto.html / pareto.csv when the subset's pareto is true.
  • lasagna/stability/tasks_<t>/ and lasagna/validity/noise_<level>/tasks_<t>/: one <variant>.html per entry of lasagna.variants and per_epv.csv (the plotted values).
  • analysis/: the figures of the Performance tab's sections per dataset characteristic, model, task type and budget and of its Ablations (all results, not filtered by the chips): heatmap_<key>.html per entry of analysis.groupings (chips "Group by", hash key group), improvement_by_model.html, improvement_by_task_type.html and rank_by_size.html when listed (heatmap_budget.html is also shown as a figure of its own), elo_no_timeouts.html (Elo on the datasets where no method ran out of time next to the default Elo, as in the paper's Figure D.3; leaderboard of those datasets in leaderboard_no_timeouts.csv) when listed, and their CSV files.
  • results.csv and results_explorer.html: the results per dataset, method, downstream model and budget (Performance Trade-offs tab, below the Pareto front; not filtered by the chips).
  • per_dataset.csv (mean rank per dataset; data only, not shown), and optionally stability.csv, stability_table.html, validity.csv and one validity_noise_<level>_table.html per noise level (listed in meta.json as validity_tables).
  • website_leaderboard.csv columns: Rank, Method, Category (Search / Non-search / Baseline), Subgroup (the paper's taxonomy, e.g. Tree, Backward search), Elo and its interval, the metrics.

The generated pages are self-contained (inline CSS/JS, no network) and post their height to the parent window. All figures (Elo, Pareto front, lasagna plots) are such pages with an inline SVG: tooltips on hover or tap, colours that follow the page's theme, and SVG / PNG / PDF / CSV buttons whose downloads always use the light paper look.

subsetDir() in app.js mirrors selectarena.website.subsets.Subset.rel_path; change both together.

Conventions

  • Never hand-edit data/: refresh it with the update-leaderboard skill in the SelectArena repo (../SelectArena/.claude/skills/update-leaderboard/), which deletes the old content before copying.
  • Every refresh bumps selectarena-version in index.html (patch for a data refresh, minor for site changes) and says what changed in the commit message. smoke_test.py checks the tag exists.
  • No external scripts, fonts or CDNs; everything is served from this repo.
  • Verification: python smoke_test.py, then python -m http.server 8000 and a look at http://localhost:8000 (all four tabs, a few selections, light and dark mode).
  • Do not commit or push without an explicit request: pushing deploys the Space.