SDPO training of Qwen3.6-35B-A3B using Modal
A five-step SDPO bridge run using the cached Qwen/Qwen3.6-35B-A3B model, ETO agent trajectory data, a one-GPU self-teacher, and four H200s for the student. Despite the shorthand "Qwen3.5-35B-A3B" in planning notes, the artifact-backed run used the Qwen3.6 35B-A3B checkpoint.
Tensor parallel 4, expert parallel 4, bf16 model dtype, CPU parameter and optimizer offload enabled.
One forward-pass teacher with TP 1, EP 1, and 0.9 vLLM memory utilization.
What actually completed
The Modal volume log was downloaded into this repository and inspected with the local SDPO log classifier. The classifier found all five configured steps and marked the run as completed with a teardown warning.
Final metric line includes training/global_step:5.
Prompt cap 2048, response cap 128, with observed prompts below cap.
Max allocated per actor worker; reserved memory reached 87.7 GB.
Step 5 throughput after warmup; first step paid initialization cost.
ETO traces as Apex-style signal
Instead of GSM8K, the run used open agent trajectories from
agent-eto/eto-sft-trajectory. The prepared tiny smoke
split used WebShop-style rows for training and SciWorld-style rows
for validation, converted into Verl parquet with feedback-aware
prompt fields.
Load cached model
Download first, then train from
/cache/models/Qwen_Qwen3_6-35B-A3B on the Modal volume.
Prepare trajectory data
Reuse /cache/data/eto_sdpo/train.parquet and
/cache/data/eto_sdpo/test.parquet.
Sample with vLLM
Student rollout uses TP 4 and EP 4 with one generated response per prompt and tight sequence limits.
Score with self-teacher
Teacher compute stays small: one H200, one forward pass path, and the same local checkpoint as the student.
Update full model
Full FSDP update with AdamW, bf16, activation offload, parameter offload, and optimizer offload.
The knobs that mattered
The successful run used the OPD-style SDPO shell wrapper so batch, topology, model, teacher, and token settings are environment variables rather than hard-coded Python values.
Learning checklist
- Two H200s were not enough for full Qwen 35B-A3B Adam/FSDP training; four student H200s got the smoke through.
- The teacher can stay at one GPU for this SDPO bridge because it only supplies log-probability guidance for the sampled batch.
- MoE rollout needed expert parallelism aligned with the student topology: TP 4 and EP 4 for the four-H200 student.
- Batch size 1 failed with the four-worker old-log-prob path; the completed run used train batch size 4 and mini-batch size 4.
Core launch settings
STUDENT_MODEL=/cache/models/Qwen_Qwen3_6-35B-A3B
TEACHER_MODEL=/cache/models/Qwen_Qwen3_6-35B-A3B
TRAIN_FILE=/cache/data/eto_sdpo/train.parquet
VAL_FILE=/cache/data/eto_sdpo/test.parquet
NGPUS_PER_NODE=4
TEACHER_WORLD_SIZE=1
ROLLOUT_TP=4
ROLLOUT_EP=4
TRAIN_SIZE_BSZ=4
PPO_MINI_BATCH_SIZE=4
MAX_PROMPT_LENGTH=2048
MAX_RESPONSE_LENGTH=128
TOTAL_EPOCHS=5
TOTAL_TRAINING_STEPS=5
PARAM_OFFLOAD=True
OPTIMIZER_OFFLOAD=True
ENABLE_ACTIVATION_OFFLOAD=True
FSDP_MODEL_DTYPE=bfloat16What to inspect next
The run is a smoke proof, not a quality claim. Scores stayed at zero on the tiny ETO slice, responses clipped at 128 tokens, and the log includes a DataLoader worker killed during teardown after step 5.
Saved artifacts
- Published log:
docs/artifacts/eto_sdpo-sdpo-shell-20260611T055738Z.log - Modal volume log:
/cache/logs/eto_sdpo-sdpo-shell-20260611T055738Z.log - SDPO launcher:
SDPO/modal_verl_sdpo.py - Published shell wrapper:
docs/artifacts/run_qwen_sdpo_mopd_fsdp.sh
Recommended next run
Keep the same cached model and dataset path, raise response length only after memory remains stable, and save a checkpoint once the teardown warning is cleaned up or shown to be harmless across repeated runs.