CodeK LoRA v1 -- Qwen2.5-Coder-7B-Instruct
A LoRA adapter fine-tuned on the CodeK v2 dataset: a reasoning-first, pedagogical coding dataset with ~2x the seeds of v0. Teaches decomposition, bug diagnosis, contrast reasoning, and hypothesis-driven thinking about code.
Eval Results (Pass 2 ground-truth, 50 seeds)
| Model | Pass@1 | vs v0 |
|---|---|---|
| Base (Qwen2.5-Coder-7B-Instruct) | 62% | -2% |
| LoRA v1 (checkpoint-1300) | 60% | +2% |
The regression gap vs base model closed from -6% (v0) to -2% (v1). Evaluated on the same 50 seeds as v0 for direct comparison.
Note: best checkpoint (step 900, eval loss 0.0664) was rotated out during training (save_total_limit=3). checkpoint-1300 (eval loss 0.077) used instead. True best checkpoint would likely score 62–64%.
v0 → v1 Comparison
| v0 | v1 | |
|---|---|---|
| Dataset | codek-v1 (201 seeds) | codek-v2 (398 seeds) |
| Train pairs | 2,351 | 4,567 |
| Best eval loss | 0.0583 | 0.0664 (best surviving: 0.077) |
| LoRA Pass@1 | 58% | 60% |
| Base Pass@1 | 64% | 62% |
| Gap (LoRA vs base) | -6% | -2% |
Training
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-7B-Instruct |
| Method | LoRA (RS-LoRA) |
| Rank / Alpha | 16 / 32 |
| Dropout | 0.0 (Unsloth fast patching) |
| Epochs | 3 (stopped at step 1500 / epoch 2.6) |
| Batch (effective) | 8 |
| Learning rate | 2e-4 linear |
| Train pairs | 4,567 |
| Val pairs | 508 |
| Best eval loss | 0.0664 (step 900, epoch 1.93) |
| Checkpoint used | checkpoint-1300 (eval loss 0.077, best surviving) |
| Hardware | RunPod A100 80GB, ~2.5 hrs |
Eval Loss Curve
| Step | Eval Loss |
|---|---|
| 100 | 0.2323 |
| 200 | 0.1354 |
| 300 | 0.1028 |
| 400 | 0.0854 |
| 500 | 0.0750 |
| 600 | 0.0747 |
| 700 | 0.0747 |
| 800 | 0.0689 |
| 900 | 0.0664 ← best (rotated out) |
| 1000 | 0.0755 |
| 1100 | 0.0765 |
| 1200 | 0.0761 |
| 1300 | 0.0767 ← used for eval |
Dataset
mechramc/codek-v2 (coming soon) -- 398 seeds, 4 augmentation passes, 5,075 ShareGPT pairs. Categories: data structures, algorithms, ML fundamentals, NN components, training infra, utilities, numerical, parsing.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "mechramc/codek-qwen2.5-coder-7b-lora-v2")
tokenizer = AutoTokenizer.from_pretrained("mechramc/codek-qwen2.5-coder-7b-lora-v2")
Links
- Downloads last month
- 34