Harvey LAB + SDPO smoke

Legal-agent SDPO on Harvey data.

A practical ContinuumAI run that adapts the Harvey LAB benchmark into feedback-conditioned Verl parquet, trains Qwen3.6-35B-A3B with the current SDPO bridge, and keeps the unused Harvey tasks held out for later evaluation.

Qwen3.6-35B-A3B Harvey LAB 8 train tasks 2 eval preview tasks 5x H200 Modal run
Student policy FSDP
H200 1
H200 2
H200 3
H200 4

Full model update with four actor ranks, TP 4 / EP 4 rollout, bf16 model dtype, activation offload, parameter offload, and optimizer offload.

Self-teacher vLLM
H200 5

Same cached Qwen3.6 checkpoint, TP 1 / EP 1, used for dense self-distillation guidance keyed by harvey/lab.

Run evidence

What actually completed

The successful Modal run trained for one configured step, emitted final metrics, and persisted both FSDP and Hugging Face checkpoint artifacts under global_step_1. The app stopped after a post-completion DataLoader worker warning.

Status 1 / 1 step

Final line includes training/global_step:1.

Reward mean 0.4708

Harvey rubric-term reward branch over generated legal outputs.

Checkpoint global_step_1

Four FSDP rank shards plus a two-shard HF safetensors export.

Step time 495.1 s

353.1 s was checkpoint save time for the 35B model.

Data split

Seeded, practice-area stratified split

The pipeline discovers every tasks/**/task.json, buckets tasks by top-level Harvey practice area, shuffles each bucket with seed 17, then round-robins across practice areas. The first 8 tasks become feedback-conditioned train rows; every remaining task is held out, with 2 heldout tasks materialized into the eval-preview parquet.

1

Discover tasks

Read Harvey LAB task metadata, instructions, deliverables, work type, practice area, and rubric criteria.

2

Build prompts

Combine instructions with trimmed excerpts from DOCX, XLSX, EML, text, CSV, JSON, and Markdown source files.

3

Generate attempts

Use Qwen3.6-35B-A3B on Modal to draft the requested legal work products before feedback conditioning.

4

Attach feedback

With no judge secret configured, the run used rubric-derived feedback and reward terms rather than external Harvey judge scores.

5

Write parquet

Emit Verl RLHF parquet with data_source=harvey/lab, previous attempt, feedback, reward terms, and task metadata.

Training rows

The 8 Harvey tasks used for SDPO

These rows are the feedback-conditioned training set in /cache/data/harvey_lab_sdpo/train.parquet. The task names below are the exact Harvey task ids written into extra_info.task_id.

Heldout preview

Eval data was kept separate

The rest of the Harvey LAB tasks remain held out in the manifest. Two heldout tasks were also converted into /cache/data/harvey_lab_sdpo/test.parquet as an eval preview. They were not used as feedback-conditioned train rows.

Eval-preview tasks

  • Emerging Companies / Venture Capital
    analyze-counterparty-markup-of-bridge-loan-agreement
  • Employment / Labor
    compare-employment-discrimination-complaint-against-personnel-file-records

Why only a preview?

This first Modal run was a smoke gate for the SDPO mechanics: data conversion, prompt filtering, Qwen generation, the custom harvey/lab reward branch, FSDP update, and checkpoint save. Full Harvey judge evaluation should be enabled only after a Modal API secret is configured for the selected judge provider.

Loss curve

One-step loss snapshot

Because this was a one-step smoke, the chart is a single measured point rather than a claimed trend. It is still useful: it proves the rollout, reward, old-log-prob, actor update, distillation term, and checkpoint path all executed together.

Step 1 measured values

Harvey SDPO one-step loss and reward snapshot A single measured step with actor loss -0.4708, distillation absolute loss 0.0254, and reward mean 0.4708. 0.6 0 -0.6 global step 1 actor/loss -0.4708 distill abs 0.0254 reward mean 0.4708
Actor loss Distillation absolute loss Reward mean

Step metrics

  • actor/loss
    -0.4707865715
  • actor/distillation/abs_loss
    0.0253955205
  • actor/distillation/loss
    0.0000463274
  • critic/rewards/mean
    0.4708333313
  • response length
    512 / 512 clipped on all sampled rows
  • prompt length mean
    2157.75 tokens, below the 4096 cap
Launch config

The settings that made it run

The first batch-size-1 attempt reached training and failed because a one-row tensor batch could not be split over four actor ranks. The completed checkpoint run used batch size 4 and mini-batch size 4.

Core shell environment

STUDENT_MODEL=/cache/models/Qwen_Qwen3_6-35B-A3B
TEACHER_MODEL=/cache/models/Qwen_Qwen3_6-35B-A3B
TEACHER_KEY=harvey/lab
TRAIN_FILE=/cache/data/harvey_lab_sdpo/train.parquet
VAL_FILE=/cache/data/harvey_lab_sdpo/test.parquet
NGPUS_PER_NODE=4
TRAIN_SIZE_BSZ=4
PPO_MINI_BATCH_SIZE=4
MAX_PROMPT_LENGTH=4096
MAX_RESPONSE_LENGTH=512
ROLLOUT_TP=4
ROLLOUT_EP=4
TEACHER_WORLD_SIZE=1
SAVE_FREQ=1
TOTAL_TRAINING_STEPS=1
TOTAL_EPOCHS=1

Durable artifacts

  • Modal app: ap-0jiiNIjVSxMfL4U6p8TYrH
  • Function call: fc-01KTWWF4W6KCWN526726B38ZXN
  • Log: /cache/logs/harvey_lab_sdpo-sdpo-shell-20260612T030212Z.log
  • Checkpoint: /cache/checkpoints/harvey_lab_sdpo-sdpo-shell-20260612T030212Z/global_step_1
  • HF export: model-00001-of-00002.safetensors and model-00002-of-00002.safetensors
Practical readout

What this smoke run proves

This is not a claim that Qwen improved on Harvey LAB yet. It is a proof that Harvey LAB data can feed the current ContinuumAI Verl SDPO bridge and produce a persisted Qwen3.6-35B-A3B checkpoint.

Proven

  • Harvey tasks can be transformed into SDPO-style train and eval parquet.
  • All 8 train prompts survived the 4096-token Verl prompt filter.
  • The harvey/lab reward branch returned nonzero rewards.
  • Full-model FSDP Qwen3.6-35B-A3B training completed one step on Modal.
  • The checkpoint was saved as both FSDP rank shards and Hugging Face safetensors.

Still open

  • Enable a Modal judge secret and run Harvey's evaluator instead of rubric fallback feedback.
  • Extend beyond one step and plot a real loss curve across multiple checkpoints.
  • Reduce 512-token response clipping before judging quality.
  • Clean up the post-completion DataLoader worker killed warning.
  • Run the heldout Harvey eval set against base, step 1, and later checkpoints.