Modal Dojo

Modal Dojo is an open-source Python SDK for RL post-training on Modal.

The library takes care of infrastructure concerns such as cluster topology, Ray/NCCL bring-up, volume mounts, checkpointing, and serving for eval and rollouts, so you (or your agent!) can focus on training models.

To get started, you can go through the Quickstart section below, or paste the prompt below into your favorite agent:

Install the `modal-dojo` library. Then, install the skill bundle into the local project with `modal-dojo skills install`. Finally, tell the user that they can ask you to do something like: "Walk through tutorials/rl_basics.py and train Qwen3.5-4B to write 5-7-5 haikus from statworx/haiku."

Quickstart

Requirements:

• Python 3.12

Install the package directly:

uv pip install -q git+https://github.com/modal-projects/modal-dojo.git@main

Or pin it in pyproject.toml:

modal-dojo = { git = "https://github.com/modal-projects/modal-dojo.git", branch = "main" }

Authenticate with Modal:

modal setup

Set up the dashboard:

modal-dojo setup

And empower your agents with the Modal Dojo’s skill bundle:

modal-dojo skills install

Then, it’s as easy as:

import re

from modal_dojo import (
    HuggingFaceDataset,
    Qwen3_5_4B,
    Qwen3_5_4B_Recipe,
    TrainConfig,
)

model = Qwen3_5_4B()


async def gsm8k_rm(args, sample, **kwargs) -> float:
    text = model.parse_response(sample.response or "").content
    boxed = re.findall(r"\\boxed\{([^}]+)\}", text)
    pred = boxed[-1] if boxed else (re.findall(r"-?[\d,]+(?:\.\d+)?", text) or [""])[-1]
    try:
        return float(float(pred.replace(",", "")) == float(sample.label))
    except ValueError:
        return 0.0


config = TrainConfig(
    model=model,
    dataset=HuggingFaceDataset(
        hf_repo="skrishna/gsm8k_only_answer",
        hf_split="train[:120]",
        input_column="text",
        output_column="label",
        input_format="text",
    ),
    recipe=Qwen3_5_4B_Recipe(
        custom_rm_function=gsm8k_rm,
    ),
)

if __name__ == "__main__":
    run = config.launch()
    print(run.training_run_id)

For a step-by-step walkthrough, see the Getting started tutorial.

Supported models

Family	Models
DeepSeek	DeepSeek-V4.1-Flash
GLM	GLM-4.7
Gemma	gemma-4-26B-A4B-it
Inkling	Inkling-Small
Kimi	Kimi-K3
Moonlight	Moonlight-16B-A3B-Instruct
Qwen3	Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-ASR-1.7B Qwen3-VL-8B-Instruct
Qwen3.5	Qwen3.5-0.8B Qwen3.5-2B Qwen3.5-4B Qwen3.5-9B
Qwen3.6	Qwen3.6-27B Qwen3.6-35B-A3B
Qwen3.8	Qwen3.8-27B