Zyphra
/

ZAYA1-reasoning-base

@@ -8,14 +8,14 @@ ZAYA1 is an 800m active/8.3B total parameter MoE model, and the first trained en
 Our ZAYA1 base model benchmark performance is extremely competitive with the SoTA Qwen3 series of models of comparable scale, and outperforms comparable western open-source models such as SmolLM3, and Phi4. ZAYA1-base excels especially at complex and challenging mathematical and STEM reasoning tasks, nearly matching the performance of SoTA Qwen3 thinking models under high pass@k settings even prior to explicit post-training for reasoning, and exceeds other strong reasoning models such as Phi4-reasoning, and Deepseek-R1-Distill.
-Details of our pretraining efforts, hardware specific optimizations, and ZAYA1 base model benchmarks are described in the [accompanying technical report](-/TODO)
 This version of the model has undergone an additional 1T tokens of reasoning-focused midtraining.
 ## Model Details
-ZAYA1's architecture includes several innovations developed at Zyphra. These include
 - **Compressed Convolutional Attention (CCA)**: [This novel attention](-/TODO) mechanism performs attention entirely in the latent space enabling significant reductions in parameter count, prefill compute, and KV cache size compared to alternative attention mechanisms, while also being more performant in loss/flop.
 - **ZAYA1 Router**: The ZAYA1 router makes fundamental improvements to the linear router used in almost all existing large-scale MoE models. The ZAYA1 router replaces the linear with a downprojection followed by a depth-mixing EDA layer then a three-layer MLP per expert to add significant nonlinear expressivity to the router.
@@ -24,7 +24,7 @@ ZAYA1's architecture includes several innovations developed at Zyphra. These inc
 ![zaya_arch](https://cdn-uploads.huggingface.co/production/uploads/65c05e75c084467acab2f84a/Ih8RnOPNbtRzaVcH16ar-.png)
-ZAYA1-reasoning-base uses the [Gemma3](https://ai.google.dev/gemma/terms) tokenizer
 ## Performance

 Our ZAYA1 base model benchmark performance is extremely competitive with the SoTA Qwen3 series of models of comparable scale, and outperforms comparable western open-source models such as SmolLM3, and Phi4. ZAYA1-base excels especially at complex and challenging mathematical and STEM reasoning tasks, nearly matching the performance of SoTA Qwen3 thinking models under high pass@k settings even prior to explicit post-training for reasoning, and exceeds other strong reasoning models such as Phi4-reasoning, and Deepseek-R1-Distill.
+Details of our pretraining efforts, hardware specific optimizations, and ZAYA1 base model benchmarks are described in the [accompanying technical report](-/TODO).
 This version of the model has undergone an additional 1T tokens of reasoning-focused midtraining.
 ## Model Details
+ZAYA1's architecture includes several innovations developed at Zyphra. These include:
 - **Compressed Convolutional Attention (CCA)**: [This novel attention](-/TODO) mechanism performs attention entirely in the latent space enabling significant reductions in parameter count, prefill compute, and KV cache size compared to alternative attention mechanisms, while also being more performant in loss/flop.
 - **ZAYA1 Router**: The ZAYA1 router makes fundamental improvements to the linear router used in almost all existing large-scale MoE models. The ZAYA1 router replaces the linear with a downprojection followed by a depth-mixing EDA layer then a three-layer MLP per expert to add significant nonlinear expressivity to the router.
 ![zaya_arch](https://cdn-uploads.huggingface.co/production/uploads/65c05e75c084467acab2f84a/Ih8RnOPNbtRzaVcH16ar-.png)
+ZAYA1-reasoning-base uses the [Gemma3](https://ai.google.dev/gemma/terms) tokenizer.
 ## Performance