Kimi K2.6 Layout

Distributed SDPO across H200 and B200 clusters.

Kimi K2.6 is a trillion-parameter MoE with 32B active parameters per token. Training it is less about one huge GPU and more about balancing expert sharding, rollout throughput, RDMA weight sync, and adapter size.

Back to OPD vs SDPO Read runbook
128 H200 full pilot 64 B200 probe 16 H200 LoRA pilot RDMA + NCCL
H200 full 16 nodes x 8 GPUs. Conservative full-fine-tune pilot using Hopper software maturity.
B200 full 8 nodes x 8 GPUs. Higher memory bandwidth, newer kernels, tighter compatibility checks.
GPU 0
GPU 1
GPU 2
GPU 3
GPU 4
GPU 5
GPU 6
GPU 7
One Modal node Request `H200:8` or `B200:8`, then use clustered functions with `rdma=True` for multi-node Verl workers.
Topology Choices

Start with the mode that matches the risk.

The full modes are expensive cluster experiments. The LoRA mode is the fastest way to validate SDPO rewards, feedback templates, rollout quality, and NCCL/RDMA behavior.

h200-full

Full Kimi pilot on H200

Best first full-training target when Hopper kernels and framework support matter most.

  • GPUs16 x H200:8 = 128
  • ActorTP 8, PP 2, EP 8, ETP 1
  • RolloutTP 8, EP 8, rollout_n 1
  • UseFull fine-tuning pilot after conversion is validated
b200-full

Full Kimi probe on B200

Promising for memory-bound rollout and MoE serving, but image and kernel compatibility matter.

  • GPUs8 x B200:8 = 64
  • ActorTP 8, PP 1, EP 8, ETP 1
  • RolloutTP 8, EP 8, chunked prefill
  • UseThroughput probe with newer compatible B200 image
h200-lora

Rank-32 LoRA on H200

Best first Kimi experiment because adapter updates are small enough for fast iteration.

  • GPUs2 x H200:8 = 16
  • ActorTP 4, PP 1, EP 4, LoRA rank 32
  • RolloutTP 8, EP 8, base model mostly frozen
  • UseReward, feedback, and continual-learning validation
Distributed Loop

SDPO needs five systems to move together.

The expensive part is not only training. Rollout workers, verifier rewards, dense self-teacher probabilities, and checkpoint sync must keep the trainer fed.

1

Actor trainer

Megatron shards the Kimi MoE actor across TP, PP, EP, ETP, and optional CP.

2

Rollout engine

vLLM samples current-policy trajectories with rollout TP and expert parallelism.

3

Feedback

Tests, tools, judges, or verifiers produce rewards and correction text.

4

Self-teacher

Kimi or a frozen checkpoint scores feedback-conditioned trajectories with k1 OPD.

5

RDMA sync

NCCL over RDMA moves full or LoRA weight deltas back to rollout workers quickly.

Parallelism Map

TP, PP, EP, ETP, and CP solve different bottlenecks.

Kimi K2.6 is an MoE model, so expert placement is a first-class sizing problem. Keep the knobs explicit and change one at a time.

How to read the knobs

Tensor parallelism splits matrix math. Pipeline parallelism splits layers. Expert parallelism spreads MoE experts. Expert tensor parallelism splits individual experts. Context parallelism helps when sequence length dominates memory.

For rollout serving in the current Verl/vLLM path, keep expert parallelism consistent with tensor and data parallelism: `rollout_ep == rollout_tp * rollout_dp`.

TP

Raise when attention and dense blocks cannot fit per rank.

PP

Raise when layer stacks need to be spread across more GPUs.

EP

Main Kimi MoE lever for distributing the expert pool.

ETP

Start at 1; raise only if individual experts are too large or slow.

CP

Useful when long context is the memory pressure point.

LoRA

Freeze base weights and sync adapter deltas for faster SDPO iteration.
RDMA Impact

Weight sync decides whether GPUs wait or learn.

Modal's RL infrastructure blog shows why clustered functions with RDMA matter. Kimi-scale updates are too large to treat network transfer as background noise.

Full update

595.2 GB INT4 MoE plus BF16 attention.

TCP 95.23 s
RDMA 1.49 s

Shared-outer LoRA

Rank-32 adapter update with 9.4 GB transferred.

TCP 1.50 s
RDMA 23.5 ms

Per-expert LoRA

Rank-32 adapter update with 41.0 GB transferred.

TCP 6.56 s
RDMA 102.5 ms
LoRA Layout

Use H200 LoRA to prove the loop before full training.

LoRA turns the first Kimi SDPO run from a massive full-weight synchronization problem into a smaller adapter synchronization problem while preserving the same reward and rollout plumbing.

Frozen Kimi base

Native INT4 MoE and attention stack stay mostly fixed during the pilot.
→

Rank-32 adapters

Train a compact behavior delta from feedback-conditioned SDPO examples.
→

Fast promotion loop

Sync adapters, evaluate regressions, then decide whether to merge or keep modular.