Upload folder using huggingface_hub

README.md

@@ -0,0 +1,135 @@
+# [Puro-2B](https://huggingface.co/thu-pacman/Puro-2B)
+
+Puro-2B is a 2B-class causal language model trained with an openly documented,
+cost-aware pretraining recipe. This collection contains Qwen3-compatible
+Transformers exports for the Phase 1 checkpoint, curriculum and uniform Phase 2
+variants, and the six late checkpoints used for the final equal-weight average.
+
+## Model Catalog
+
+| Model | Role |
+| --- | --- | --- |
+| [Puro-2B-Base](https://huggingface.co/thu-pacman/Puro-2B-Base) | Canonical exported final; equal-weight average of six late Phase 2 checkpoints. |
+| [Puro-2B-Base-Phase1](https://huggingface.co/thu-pacman/Puro-2B-Base-Phase1) | End of Phase 1, before the Phase 2 data distribution. |
+| [Puro-2B-Curriculum-DecayFinal](https://huggingface.co/thu-pacman/Puro-2B-Curriculum-DecayFinal) | Final checkpoint from the curriculum-ordered Phase 2 run with the decayed schedule. |
+| [Puro-2B-Curriculum-SMA6-Inputs](https://huggingface.co/thu-pacman/Puro-2B-Curriculum-SMA6-Inputs) | Six input checkpoints used by the equal-weight simple moving average; an artifact set, not an additional averaged model. |
+| [Puro-2B-Uniform](https://huggingface.co/thu-pacman/Puro-2B-Uniform) | Full-budget uniform-data-ordering control. |
+| [Puro-2B-Uniform-Phase2-1of2](https://huggingface.co/thu-pacman/Puro-2B-Uniform-Phase2-1of2) | Uniform Phase 2 run at one half of the full ladder budget. |
+| [Puro-2B-Uniform-Phase2-1of4](https://huggingface.co/thu-pacman/Puro-2B-Uniform-Phase2-1of4) | Uniform Phase 2 run at one quarter of the full ladder budget. |
+| [Puro-2B-Uniform-Phase2-1of8](https://huggingface.co/thu-pacman/Puro-2B-Uniform-Phase2-1of8) | Uniform Phase 2 run at one eighth of the full ladder budget. |
+| [Puro-2B-Uniform-Phase2-1of16](https://huggingface.co/thu-pacman/Puro-2B-Uniform-Phase2-1of16) | Uniform Phase 2 run at one sixteenth of the full ladder budget. |
+
+## Training Summary
+
+The recipe described in the accompanying Prom Technical Report has two
+pretraining phases:
+
+- Phase 1: approximately 439B tokens on 24 RTX 5090 GPUs.
+- Phase 2: approximately 961B additional tokens on 96 RTX 5090 GPUs.
+- Total: approximately 1.4T tokens.
+- Sequence length: 4,096; global batch size: 1,536; micro-batch size: 2.
+- Optimizer: MuonH with weight decay 0.1.
+- Training precision: blockwise E4M3 FP8.
+- Learning-rate schedule: power decay in Phase 1 and a decayed/continued Phase 2 schedule, depending on the released variant.
+
+The curriculum and uniform variants share the Phase 2 pool. The curriculum
+variant changes the data order using the documented curriculum construction;
+the uniform control removes that ordering with a deterministic global
+reshuffle. The ladder checkpoints are intended for controlled studies of data
+ordering and budget, not as independently tuned deployment models.
+
+## Reported Results Snapshot
+
+The following headline results are reproduced from the current technical-report
+draft and should be treated as provisional until the public evaluation table,
+model revision, and uncertainty protocol are frozen:
+
+- Under the report's aggregate evaluation protocol, Puro-2B is comparable to
+  Qwen2.5-1.5B and exceeds Qwen2-1.5B.
+- In the controlled clean mathematical SFT probe, the curriculum initialization
+  is ahead of the matched uniform initialization by about 4 percentage points
+  on GSM8K at the endpoint.
+- In the scaled mathematical SFT setup, the curriculum advantage is about 2
+  percentage points.
+- In the broad Tulu SFT transfer setup, curriculum improves the Core15
+  macro-average by 1.59 points and is higher on 13 of 15 component benchmarks;
+  HumanEval and BoolQ are documented regressions.
+
+These results describe matched experiments in the report. They are not claims
+that every checkpoint in this collection has the same downstream score.
+
+## Architecture and File Format
+
+All exported checkpoints use the `Qwen3ForCausalLM` architecture and the same
+tokenizer family. The checked-in configuration reports:
+
+| Property | Value |
+| --- | --- |
+| Hidden size | 2,048 |
+| Transformer layers | 28 |
+| Attention heads / KV heads | 16 / 8 |
+| Feed-forward size | 6,144 |
+| Vocabulary size | 151,936 |
+| Maximum position embeddings | 4,096 |
+| Weight file | `model.safetensors` |
+
+Each single-model repo contains model weights, `config.json`, generation
+configuration, and tokenizer files. The exports do not contain optimizer
+states, training dataloaders, or the original training checkpoints. The
+[averaging-input repo](https://huggingface.co/thu-pacman/Puro-2B-Curriculum-SMA6-Inputs)
+contains six complete model exports:
+
+```text
+iter_0222100
+iter_0222200
+iter_0222300
+iter_0222400
+iter_0222500
+iter_0222569
+```
+
+The canonical averaged model is the equal-weight parameter average
+
+```text
+1/6 * (iter_0222100 + iter_0222200 + iter_0222300
+       + iter_0222400 + iter_0222500 + iter_0222569)
+```
+
+Optimizer moments are not averaged into the exported model. The public bundle
+records the averaging equation above without exposing machine-local paths.
+
+## Loading a Checkpoint
+
+After selecting one model repo, load it with a Transformers version that
+supports the Qwen3 configuration:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "thu-pacman/Puro-2B-Base"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype="auto",
+    device_map="auto",
+)
+```
+
+The final release preserves the tokenizer files alongside each weight file.
+Each linked repo has a lightweight README pointing back to this collection
+description.
+
+## Interpretation Notes
+
+- `Puro-2B-Base` is a post-processed average, not a raw training checkpoint.
+- `Puro-2B-Curriculum-SMA6-Inputs` is the reproducibility input set for that average and should be presented as an artifact bundle rather than a ninth model.
+- `Puro-2B-Curriculum-DecayFinal` is the decay-final export; it is distinct from the six-checkpoint averaged base model.
+- The uniform shortened checkpoints form a budget ladder. Their `1ofN` labels refer to the Phase 2 training budget, not to parameter count or model width.
+- Downstream benchmark claims belong to the technical report and its evaluation protocol. They should not be inferred from the directory names alone.
+
+## Reference
+
+See the accompanying **Prom Technical Report** for the full data recipe,
+training traces, curriculum construction, cost assumptions, and evaluation
+protocol. This README is intentionally limited to the artifacts in this
+collection.

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 4096,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.13.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": false,
+  "eos_token_id": [
+    151643,
+    151645
+  ],
+  "max_new_tokens": 2048,
+  "pad_token_id": 151643,
+  "transformers_version": "4.37.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85e8b5e0122077d63ad8d4f45c99c913adafe708356ad4303107088a3b44456b
+size 4063515640

tokenizer_config.json

@@ -0,0 +1,40 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": ["<|im_start|>", "<|im_end|>"],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}