ContinuumAI

OPD vs SDPO for continual learning.

OPD uses a teacher to score what the policy actually sampled. SDPO turns the model's own attempts and feedback into dense self-teaching signals, making continual learning less brittle.

OPD SDPO Verl Modal GPUs Qwen / Kimi
Current policy Samples answers, tool calls, code, or reasoning traces.
→
Dense guidance External teacher for OPD; feedback-conditioned self-teacher for SDPO.
→
Policy update Combines RL reward with dense token-level distillation losses.
Core Difference

OPD imports a teacher. SDPO grows one from feedback.

Both methods stay on-policy: the learning signal is attached to trajectories the current model actually sampled. They differ in where dense supervision comes from and how useful feedback becomes.

OPD

Teacher

On-policy distillation asks a stronger or frozen teacher to score the student's sampled trajectory with token-level probabilities.

  • Best when a reliable teacher is available.
  • Useful for recovery distillation after domain updates.
  • Can preserve broad behavior while adding new skills.

SDPO

Self-teacher

Self-Distilled Policy Optimization conditions the model on its own attempts, successes, and rich feedback to create dense targets.

  • Best when environments produce useful feedback.
  • Turns test errors and critiques into token-level guidance.
  • Can keep improving as the policy and self-teacher improve.
Loss Anatomy

SDPO turns feedback into reusable learning signal.

Sparse rewards say whether an attempt worked. OPD and SDPO add dense token guidance; SDPO is especially useful when failed attempts come with explanations, traces, or test errors.

Policy Current samples

All learning data starts from the latest policy, not a stale offline distribution.

Reward Outcome signal

Math checks, unit tests, judges, and tools decide what actually succeeded.

OPD External guidance

A teacher or frozen checkpoint scores sampled trajectories.

SDPO Feedback guidance

The model uses attempts and feedback to teach its future policy.

Continual Learning

SDPO helps the model learn without waiting for a perfect teacher.

Continual learning needs a repeatable way to absorb new domains, learn from mistakes, and preserve old skills. OPD supplies recovery; SDPO supplies feedback-aware adaptation.

1

Attempt

The current policy samples solutions on new tasks and domains.

2

Collect Feedback

Tests, tools, verifiers, or judges explain what worked and what failed.

3

Self-Distill

SDPO converts successes and feedback into dense corrective targets.

4

Recover With OPD

A stable teacher or checkpoint reduces regression on older behavior.

5

Promote

Advance only when task gains, drift checks, and cost all pass.

Infrastructure

The hard part is coordinating fast loops.

Trainer, rollout engine, teacher, environments, and checkpoints have to move together. Modal supplies elastic compute; Verl and Slime supply the algorithm and distributed training machinery.

Control plane

ContinuumAI should stay thin: experiment specs, launch adapters, run manifests, eval comparison, and operational notes. Framework code remains upstream.

View Kimi K2.6 GPU layout View Qwen 35B-A3B SDPO run View Harvey LAB SDPO run
Trainer Verl, Slime, or Training Gym performs policy updates and checkpointing.
Rollout engine vLLM or SGLang samples current-policy trajectories at high throughput.
Teacher and feedback External teachers, frozen checkpoints, tools, tests, and sandboxes produce dense and scalar signals.
Modal substrate GPU functions, volumes, logs, secrets, and sandboxes keep experiments repeatable.
Guardrails

Measure the learning, not just the loss.

Distillation can preserve behavior or amplify a bad target. Keep the teacher signal, reward signal, and eval signal separately visible.

Separate metrics

Log reward, distillation loss, KL, length, pass rates, and eval deltas independently.

Teacher hygiene

Start with same-family models before cross-family teachers to avoid tokenizer alignment failures.

Regression gates

Compare against base, previous checkpoint, and frozen recovery teacher before promotion.