Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
PhysiQuanty 
posted an update 1 day ago
Post
2540
🧠 Arithmetic-SLM : A 30M model that manages to compute simple arithmetic better than a 3B model 🚀
WhirlwindAI/Arithmetic-SLM
WhirlwindAI/arithmetic-slm

🏆 Leaderboard ArithMark-2 🏆
🥇 Qwen/Qwen2.5-Math-1.5B = 82.08%
🥈 WhirlwindAI/Arithmetic-SLM = 78.60% (31.7M Params)
🥉 Qwen/Qwen2.5-3B = 78.44%

Example WhirlwindAI/Arithmetic-SLM =
0.5 * 0.5 = 0.25 ✅
105 + 45 / 8 = 110 ✅
(132 / 12) + (46 - 15) = 42 ✅
(10 + 28) * 3 = 114 ✅
1 * (16 + 28) = 44 ✅
(21 + 27) * (14 - 7) = 336 ❌ 

leaderboard = """
|              Model               |    Params    |   Score   |
|----------------------------------|--------------|-----------|
|      Qwen/Qwen2.5-Math-1.5B      |     1.54B    |   82.08%  |
|    WhirlwindAI/Arithmetic-SLM    |    31.70M    |   78.60%  | <=
|         Qwen/Qwen2.5-3B          |     3.09B    |   78.44%  |
|        Qwen/Qwen2.5-1.5B         |     1.54B    |   77.72%  |
|    Qwen/Qwen2.5-Coder-1.5B       |     1.54B    |   74.88%  |
|   HuggingFaceTB/SmolLM2-1.7B     |     1.71B    |   66.12%  |
|        Qwen/Qwen2.5-0.5B         |      494M    |   63.04%  |
| facebook/MobileLLM-R1-140M-base  |      140M    |   53.88%  |
|     SupraLabs/Supra-50M-Base     |       52M    |   27.12%  |
"""

Bench =
AxiomicLabs/ArithMark-2.0
DataSet =
WhirlwindAI/Arithmetic
By Science AND FOR SCIENCE <3
dipankarsarkar

The headline number hides the interesting part.

78.6% at 31.7M matching a 3B is real, but look at the one it missed: (21+27)(14-7). Both sub-expressions are easy, the model gets 48 and 7, then the final 487 is where it breaks. That is not an arithmetic gap, it is a magnitude gap: a two-digit times one-digit crossing into three.

So the number I would want from ArithMark-2 is accuracy stratified by the digit-length of intermediate results, not one aggregate. A 30M model can memorize the small-magnitude table and still fall off a cliff exactly where carries compound.

Does the score hold if you filter to problems whose intermediates all exceed two digits?

·
PhysiQuanty
•
edited about 4 hours ago

Thanks for your message, yes exactly.

This first dataset was mostly chained calculations, so it likely tests pattern memorization more than true arithmetic generalization. It was mainly a POC to measure the baseline of a very small model.

The next step is to train for procedural generalization: decomposing calculations into simpler sub-calculations, for example 3-digit addition into 1-digit additions with carries.

I agree that the next evaluation should be stratified by intermediate magnitude, carry depth, and digit length.
In this post