AgentAlley

Skills · Data & AI

LoRA & QLoRA Recipes

Unverified●31/40

Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add lora-qlora-recipes

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.

The whole source

No sign-in, no blur, nothing truncated
lora-qlora-recipes/SKILL.md223 lines7.5 KBRawView on GitHub
Frontmatter — 2 properties
namelora-qlora-recipes
descriptionConfigure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.
1---
2name: lora-qlora-recipes
3description: Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# LoRA & QLoRA Recipes
7 
8This skill assumes the routing decision already
9happened — `finetuning-method-selection` should
10have already pointed here because the data shape
11is demonstrations (SFT), not preference pairs or
12a verifiable reward signal. What follows is the
13current best-practice recipe for configuring the
14adapter itself: which modules to target, how to
15size rank and alpha, what learning rate to use,
16and when QLoRA buys real headroom versus when it
17just adds risk. Dataset preparation and quality
18checks are a separate concern — see
19`dataset-curation`.
20 
21**Input:** a routing decision (SFT via LoRA/
22QLoRA) plus a target size class.
23**Output format:** a validated adapter config —
24the kwarg values below, not free-form advice —
25that `llm-finetuning-training-engineer` consumes
26directly when it generates a runnable script.
27 
28## The Reference Recipe
29 
30The reference recipe is "LoRA Without Regret"
31(Thinking Machines/Schulman, 2025-09), now the
32settled convention for LoRA/QLoRA SFT.
33 
34### Target Modules
35 
36Target **all-linear** modules, not just
37attention:
38 
39```python
40target_modules = [
41 "q_proj", "k_proj", "v_proj", "o_proj", # attention
42 "gate_proj", "up_proj", "down_proj", # MLP — matters most
43]
44```
45 
46The MLP layers (`gate_proj`, `up_proj`,
47`down_proj`) matter most — attention-only
48targeting was the older, weaker convention.
49Dropping modules to save memory is a Failure
50Mode below, not a valid optimization.
51 
52### Alpha and Learning Rate
53 
54- **`lora_alpha = 2 * r`** is the settled
55 convention (NeurIPS 2025 "intruder dimensions"
56 result). Don't hand-tune alpha independently of
57 rank — derive it from rank every time.
58- **LoRA learning rate ≈ 10x the equivalent
59 full-fine-tune LR.** For QLoRA specifically,
60 **2e-4** is the standard starting point. Full
61 hyperparameter tables and worked examples:
62 `references/hyperparameters.md`.
63 
64### Rank by Task
65 
66Rank is task-shaped, not a single global default:
67 
68| Task | Rank |
69|---|---|
70| RL (GRPO/RLVR adapters) | 1–32 |
71| General default | 16–32 |
72| SFT at scale | up to ~256 |
73 
74Higher rank isn't automatically better — it
75raises capacity to memorize as fast as it raises
76capacity to generalize. Start at the row matching
77the task, and only move up a row if the lower
78rank measurably underfits on held-out eval, not
79as a default hedge.
80 
81### Effective Batch Size
82 
83Keep **effective batch size under 32**. This
84recipe was validated at that scale — pushing
85effective batch higher is an untested
86extrapolation, not a free throughput win.
87 
88## Unsloth Defaults
89 
90Unsloth is the reference implementation this
91plugin assumes as the default fast path — except
92for messages-shaped conversational SFT with
93`assistant_only_loss=True`, where Unsloth
942026.7.x's compiled trainer has no messages-shaped
95path at all and the plain-TRL escape hatch
96(`references/unsloth-trl-mapping.md`) is the
97default for that combination, not a rare-regression
98fallback. Its out-of-the-box defaults, and why
99each one is set that way:
100 
101- **`lora_dropout=0`** — the optimized kernel
102 path assumes zero dropout; setting a nonzero
103 value forfeits the fused-kernel speedup.
104- **`bias="none"`** — bias terms add adapter
105 parameters for negligible quality gain at this
106 rank range.
107- **`use_gradient_checkpointing="unsloth"`** —
108 Unsloth's checkpointing variant, not vanilla HF
109 checkpointing; saves roughly **30% VRAM** over
110 no checkpointing.
111- **`optim="adamw_8bit"`** — 8-bit AdamW cuts
112 optimizer-state memory with negligible quality
113 impact at LoRA/QLoRA adapter scale.
114- **`random_state`** fixed — pins LoRA
115 initialization for reproducibility across runs;
116 treat it like any other seed, not a tunable.
117 
118These show up together on the `get_peft_model`
119call:
120 
121```python
122model = FastLanguageModel.get_peft_model(
123 model,
124 r=32,
125 target_modules=target_modules,
126 lora_alpha=64, # 2 * r
127 lora_dropout=0,
128 bias="none",
129 use_gradient_checkpointing="unsloth",
130 random_state=3407,
131)
132```
133 
134Exact kwarg names and their plain-TRL/PEFT
135equivalents, plus a full worked config including
136`SFTConfig`: `references/unsloth-trl-mapping.md`
137and `references/hyperparameters.md`.
138 
139## LoRA vs QLoRA vs Full FT
140 
141| Situation | Default choice |
142|---|---|
143| Adapting behavior on demonstrations | LoRA |
144| Base model doesn't fit in bf16 at target rank | QLoRA |
145| Injecting dense new domain knowledge | Full FT (see `finetuning-method-selection`) |
146| Unsure which one | LoRA — upgrade to QLoRA only if memory forces it |
147 
148- **QLoRA** = NF4-quantized frozen base weights +
149 BF16 adapters. This is what makes a 65B-class
150 model trainable on 48GB — the quantized base
151 is the memory win, not the adapter itself.
152- **Full fine-tuning is not a default.** Reserve
153 it for dense knowledge injection where the goal
154 is changing what the model knows at the weight
155 level, not adapting a behavior. For everything
156 else in this skill's scope, LoRA or QLoRA is
157 the starting assumption.
158- **On DGX Spark, QLoRA can OOM before an
159 equivalent bf16 LoRA run would**, even though
160 QLoRA's steady-state footprint is smaller —
161 bitsandbytes dequantization buffers are
162 transient CUDA-side allocations that spike
163 during load. A QLoRA OOM is not proof the model
164 doesn't fit; the `dgx-spark-ops` plugin's
165 `spark-memory-thermal-ops` skill covers the
166 full OOM remediation ladder (bf16 LoRA is the
167 next thing to try, not a further QLoRA
168 shrink).
169 
170## Failure Modes
171 
172- **fp16 divergence on non-BF16 GPUs.** Training
173 in fp16 on hardware that doesn't have solid
174 BF16 support is a known source of loss spikes
175 and silent divergence. Force `bf16=True`
176 wherever the hardware supports it; don't fall
177 back to fp16 as if it were equivalent. Check
178 hardware support before picking a dtype:
179 
180 ```bash
181 python -c "import torch; print(torch.cuda.is_bf16_supported())"
182 ```
183 
184- **Rank too high on a small dataset overfits.**
185 A rank picked for "SFT at scale" (up to ~256)
186 on a dataset that doesn't have scale behind it
187 memorizes rather than generalizes. Match rank
188 to the Rank by Task table above, not to the
189 largest number available.
190- **Removing target modules to save memory costs
191 quality for negligible savings.** The adapter
192 parameters on `gate_proj`/`up_proj`/`down_proj`
193 are a small fraction of total model size — cutting
194 them barely moves memory but measurably hurts
195 quality. If memory is tight, move to QLoRA or
196 reduce rank/batch/pack length before trimming
197 target modules.
198 
199All three failure modes share a pattern: they
200look like a training-loop bug (loss spikes,
201plateaus, memorization) but are actually a
202config choice that contradicts the reference
203recipe above. Check configuration against this
204skill before debugging the training loop itself.
205 
206## References
207 
208- `references/hyperparameters.md` — full rank/
209 alpha/LR tables by task type, rsLoRA notes,
210 batch/packing interactions, and a complete
211 worked Unsloth config block.
212- `references/unsloth-trl-mapping.md` — every
213 Unsloth kwarg mapped to its TRL/PEFT
214 equivalent, current TRL API notes, and the
215 escape-hatch rule for when to drop back to
216 plain TRL.
217 
218Related skills: `finetuning-method-selection`
219routes here; `dataset-curation` covers the data
220side this skill doesn't; `llm-finetuning-training-engineer`
221is the downstream consumer of the config this
222skill produces.
223 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40