farrelmahaztra/Dongeng-35M

Dongeng is a model family trained on a dataset of the same name, which consists of ~2M Indonesian children's stories in the style of the SimpleStories dataset. Dongeng is the Indonesian word for story or folktale.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "farrelmahaztra/Dongeng-35M"
model = AutoModelForCausalLM.from_pretrained(repo).eval()
tok = AutoTokenizer.from_pretrained(repo)

prompt = "Suatu hari, seekor kucing kecil"
# add_special_tokens=False so the tokenizer doesn't append [EOS] and end the story early.
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(
    inputs.input_ids, max_new_tokens=400, do_sample=True,
    temperature=0.7, eos_token_id=tok.eos_token_id,
)

# Decode and undo the morphological segmentation ("me + baca" -> "membaca").
print(tok.decode(out[0], skip_special_tokens=True).replace(" + ", ""))

For an unconditional story, seed generation with [EOS] (the start-of-story token) instead of a prompt: model.generate(torch.tensor([[tok.eos_token_id]]), ...).

Note on prompts. The training corpus is morphologically segmented (e.g. membaca -> me + baca), so an affixed word like seekor (se + ekor in training) tokenizes slightly off-distribution when passed raw. You may want to run the prompt through the dongeng package's segment() first, although the generation should remain coherent either way and unconditional generation is unaffected.

Model Variants

Model Name	n_params	n_layers	d_model	n_heads	n_ctx	d_vocab
Dongeng-35M	35 million	12	512	8	512	4096
Dongeng-30M	30 million	10	512	8	512	4096
Dongeng-11M	11 million	6	384	6	512	4096
Dongeng-5M	5 million	6	256	4	512	4096
Dongeng-1.25M	1.25 million	4	128	4	512	4096