plan
A staged pipeline for training small language models via warm-start weight transmutation and token-level logit distillation from large teachers. 3B-5B parameters first: $15k budget yields ~10 full training sweeps at $1.5k per run, fitting on local 32GB VRAM (RTX 5090) with an optimal teacher-student capacity ratio.
Core philosophy
Scaling to 3B-5B parameters first is the optimal path. At roughly $1.5k per full training run and a $15k budget, you get ~10 complete sweeps to iterate on architecture, data, and hyperparameters. A 3B model fits comfortably on 32GB VRAM (RTX 5090), enabling local development without cloud GPU dependencies.
The teacher-student capacity ratio is critical. A 70B teacher distilling into a 3B student hits roughly 23:1 - within the proven effective range. Larger teachers (405B) push beyond 100:1, where distillation efficiency degrades. The 3B-5B sweet spot maximises knowledge transfer per parameter.
$15k
Budget
~10
Full training sweeps
$1.5k
Per run cost
Pipeline architecture & strategy
Warm-start weight transmutation
Extract MLPs, embeddings, and layernorms from a pretrained 3B Transformer (e.g. Qwen 2.5 or Llama 3.2). Map these weights into DIMBA Mamba-2 SSM blocks to initialise the student model, avoiding training from Gaussian noise. This preserves the teacher's learned representations in a different architectural family, giving the student a head start before distillation begins.
Token-level logit distillation
Apply KL divergence loss to match the student's softmax output distributions against those of the 70B/72B teacher model. Token-level distillation captures deep representational knowledge and the teacher's world model - not just hard-label accuracy - enabling the student to inherit the teacher's reasoning patterns, uncertainty calibration, and edge-case handling.
Staged execution matrix
Three-stage rollout with clearly defined scope and cost thresholds per phase.
1B-1.5B Pilot
$200-$300
Dry run for Triton kernel throughput, gradient stability, and loss-curve analysis. Validates the warm-start mapping and distillation loop before committing to the full model.
3B-5B Main Release
Core budget
Primary deliverable. The benchmark and research model - trains on the full distillation pipeline, evaluated against standard LM benchmarks.
10B+ Scale-Up
Post-grant
Follow-on stage funded by grants. Extends the architecture and distillation pipeline to larger parameter counts with refined hyperparameters.
SFT & preference optimisation
Apply GRPO (Group Relative Policy Optimisation) or DPO (Direct Preference Optimisation) using a large model as the judge. This final phase aligns the distilled model with human preferences, improving helpfulness, safety, and instruction following without expensive human annotation pipelines.