Command Line Interfaces (CLIs)

TRL provides a powerful command-line interface (CLI) to fine-tune large language models (LLMs) using methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and more. The CLI abstracts away much of the boilerplate, letting you launch training jobs quickly and reproducibly.

Commands

Currently supported commands are:

Training Commands

trl distillation: distill a teacher model into a student with on-policy knowledge distillation
trl dpo: fine-tune a LLM with DPO
trl grpo: fine-tune a LLM with GRPO
trl kto: fine-tune a LLM with KTO
trl reward: train a Reward Model
trl rloo: fine-tune a LLM with RLOO
trl sft: fine-tune a LLM with SFT

Other Commands

trl env: get the system information
trl vllm-serve: serve a model with vLLM (deprecated, use vllm serve directly, see vLLM integration)

Fine-Tuning with the TRL CLI

Basic Usage

You can launch training directly from the CLI by specifying required arguments like the model and dataset:

SFT

DPO

Reward

GRPO

RLOO

KTO

Using Configuration Files

To keep your CLI commands clean and reproducible, you can define all training arguments in a YAML configuration file:

SFT

DPO

Reward

GRPO

RLOO

KTO

Scaling Up with torchrun

The TRL CLI launches training with torchrun, one process per available GPU. Any torchrun argument can be passed directly to trl, such as --nproc_per_node or --nnodes.

SFT

DPO

Reward

GRPO

RLOO

KTO

DeepSpeed and FSDP

DeepSpeed ZeRO and FSDP are configured through the training arguments, like for any Trainer: --deepspeed takes a DeepSpeed JSON config, and --fsdp turns on FSDP, tuned with --fsdp_config. A bare --fsdp on the command line needs transformers 5.10 or later; with older versions, set fsdp: true in a config file. Ready-to-use configs live in examples/deepspeed_configs and examples/fsdp_configs. If you installed TRL with pip, save this ZeRO-2 config as zero2.json and pass --deepspeed zero2.json:

{
  "bf16": {"enabled": "auto"},
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {"device": "none"}
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto"
}

SFT

DPO

Reward

GRPO

RLOO

KTO

Until TRL v2.0.0, trl still launches through accelerate launch when it gets arguments that only accelerate launch knows, such as --num_processes or --accelerate_config, or when an accelerate config default file exists. This path is deprecated and warns.

Using dataset mixtures

You can use dataset mixtures to combine multiple datasets into a single training dataset. This is useful for training on diverse data sources or when you want to mix different types of data.

SFT

DPO

Reward

GRPO

RLOO

KTO

To see all the available keywords for defining dataset mixtures, refer to the scripts.utils.DatasetConfig and DatasetMixtureConfig classes.

Getting the System Information

You can get the system information by running the following command:

trl env

This will print out the system information, including the GPU information, the CUDA version, the PyTorch version, the transformers version, the TRL version, and any optional dependencies that are installed.

Copy-paste the following information when reporting an issue:

- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.11.9
- PyTorch version: 2.4.1
- accelerator(s): NVIDIA H100 80GB HBM3
- Transformers version: 4.45.0.dev0
- Accelerate version: 0.34.2
- Accelerate config: 
  - compute_environment: LOCAL_MACHINE
  - distributed_type: DEEPSPEED
  - mixed_precision: no
  - use_cpu: False
  - debug: False
  - num_processes: 4
  - machine_rank: 0
  - num_machines: 1
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - deepspeed_config: {'gradient_accumulation_steps': 4, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': False, 'zero_stage': 2}
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []
- Datasets version: 3.0.0
- HF Hub version: 0.24.7
- TRL version: 0.12.0.dev0+acb4d70
- bitsandbytes version: 0.41.1
- DeepSpeed version: 0.15.1
- Diffusers version: 0.30.3
- Liger-Kernel version: 0.3.0
- LLM-Blender version: 0.0.2
- OpenAI version: 1.46.0
- PEFT version: 0.12.0
- vLLM version: not installed

This information is required when reporting an issue.

Update on GitHub