Retool: from SFT to RL#

This example (Retool) demonstrates how to use the retool functionality for tool-enabled language model generation.

Overview#

The retool example provides:

Safe Python code execution in a sandbox environment
Tool registry for managing available tools
Integration with language model generation
Reward calculation for tool usage

Files#

generate_with_retool.py: Main generation function with tool support
tool_sandbox.py: Tool execution and safety management
sft_data_processing.py: Process SFT dataset

Reward Design#

The RL reward function (generate_with_retool.reward_func) uses a tool-aware reward shaping strategy on top of the math accuracy reward:

Answer	Tool Used	Reward	Rationale
✅ Correct	No	1.0	Pure reasoning — the ideal case
✅ Correct	Yes	1.0 + min(0.2, turns × 0.05)	Bonus for effective tool use (capped at 0.2)
❌ Wrong	No	0.0	Neutral — model didn’t attempt tools
❌ Wrong	Yes	min(0.1, turns × 0.02)	Small positive to encourage exploration

This design encourages the model to explore tool calling during early RL training without letting it reward-hack by preferring tool calls over correct answers. As training progresses and accuracy improves, the accuracy reward dominates and the tool bonus becomes a tiebreaker.

Usage#

1. Setup#

git clone -b ascend https://github.com/vllm-project/vime.git
cd vime
docker build -f docker/Dockerfile.npu -t vime-ascend:latest .

# Update the vime image
export IMAGE=vime-ascend:latest

docker run -d --name vime-npu -it --net=host --shm-size=1024g \
    --privileged=true \
    --cap-add=SYS_PTRACE \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/sbin:/usr/local/sbin \
    -v /home:/home \
    -v /mnt:/mnt \
    -v /tmp:/tmp \
    -v /data:/data \
    -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime \
    $IMAGE

docker exec -it vime-npu bash

2. Download#

# For SFT part, you can use later model to RL directly and skip SFT.
hf download --repo-type dataset JoeYing/ReTool-SFT  --local-dir /path/to/ReTool-SFT
hf download Qwen/Qwen3-4B-Instruct-2507 --local-dir /path/to/Qwen3-4B-Instruct-2507

# For RL part
hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /path/to/dapo-math-17k
hf download --repo-type dataset zhuzilin/aime-2024  --local-dir /path/to/aime-2024
# download our SFT model if you want to skip SFT
hf download font-info/qwen3-4b-sft-SGLang-RL --local-dir /path/to/qwen3-4b-sft

3. Preprocessing#

cd /root/vime
# Replace the save path with /path/to/ReTool-SFT.parquet
python examples/retool/sft_data_processing.py

4. SFT#

cd /root/vime
# Replace the model and data loading/saving paths
python examples/retool/retool_qwen3_4b_sft.sh

5. RL#

cd /root/vime
# Replace the model and data loading/saving paths
python examples/retool/retool_qwen3_4b_rl.sh

Retool: from SFT to RL

Contents