Skills · Data & AI

Preference Optimization

Unverified●31/40

Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time

Cursor◐PartialPlain prose you can paste in — but no Cursor rules file

Codex◐PartialPlain prose you can paste in — but no AGENTS.md

Gemini CLI◐PartialPlain prose you can paste in

Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file

npx agentalley add preference-optimization

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

The whole source

No sign-in, no blur, nothing truncated

preference-optimization/SKILL.md203 lines7.7 KBRawView on GitHub

Frontmatter — 2 properties

name	preference-optimization
description	Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.

1	---
2	name: preference-optimization
3	description: Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
4	---A5 — No allowed-tools declared — no way to tell what this skill may touch
5
6	# Preference Optimization
7
8	This skill assumes `finetuning-method-selection`
9	already routed here because the data shape is
10	preference pairs or unpaired thumbs-up/down
11	feedback, not demonstrations (that's
12	`lora-qlora-recipes`) or a verifiable reward
13	signal (that's `grpo-rlvr-training`). What
14	follows is method selection among the DPO family,
15	the evidence for how much that selection actually
16	matters, the production training pattern, and how
17	to build the pairs in the first place.
18
19	Input: a routing decision (preference
20	optimization) plus preference pairs or unpaired
21	feedback, usually from an SFT checkpoint.
22	Output format: a validated method choice plus
23	a config — the kwarg values in
24	`references/method-configs.md`, not free-form
25	advice — that `llm-finetuning-training-engineer`
26	consumes directly.
27
28	## Method Selection
29
30	\| Data shape \| Method \| Key parameters \|
31	\|---\|---\|---\|
32	\| Preference pairs, default case \| DPO \| β=0.1, LR 5e-7–1e-6, 1–2 epochs \|
33	\| Memory-bound or no SFT checkpoint \| ORPO \| reference-free, fused SFT+preference in one loss \|
34	\| Unpaired thumbs-up/down \| KTO \| binary label per example, no pairing needed \|
35	\| Length bias observed, sweep budget available \| SimPO \| reference-free; see sweep grid below \|
36
37	- DPO is the safe default. Use β=0.1 and a
38	learning rate of 5e-7 to 1e-6 for 1–2 epochs.
39	This LR is lower than the SFT LR that produced
40	the checkpoint being aligned — porting an SFT-
41	scale LR into a DPO run is the most common
42	misconfiguration here, not an edge case.
43	- ORPO routes in when memory is the
44	constraint, or when there's no separate SFT
45	checkpoint to start from — it's reference-free
46	and fuses the SFT and preference objectives into
47	one loss, skipping the separate SFT pass and the
48	reference-model memory cost DPO carries.
49	- KTO routes in when feedback is unpaired
50	binary signal (thumbs-up/down) rather than
51	matched preference pairs — don't force unpaired
52	feedback into synthetic pairs to use DPO instead.
53	- SimPO fixes DPO's length bias but only pays
54	off with disciplined sweeping — its published
55	gains are a ceiling reported under a tuned sweep,
56	not a baseline any single config will reproduce.
57	Route here only when there's sweep budget; use
58	DPO instead if there isn't.
59	- Classic RLHF (reward model + PPO) is retired
60	outside frontier labs. Don't reach for it in a
61	production pipeline — every method above is
62	cheaper and better-supported for the same data
63	shapes.
64
65	### Worked Examples
66
67	- *"We have an SFT checkpoint and clean paired
68	preference data, no length-bias complaints yet."*
69	→ default case → DPO at β=0.1.
70	- *"Reviewers click thumbs-up/down per response;
71	nothing is paired."* → unpaired signal →
72	KTO, not DPO — don't synthesize pairs to
73	force DPO onto unpaired data.
74	- *"GPU budget doesn't cover a separate SFT pass
75	plus a DPO reference model."* → memory-bound,
76	no separate checkpoint → ORPO.
77	- *"DPO output favors longer answers regardless of
78	quality, and there's time to run a sweep."* →
79	length bias plus sweep budget → SimPO. Skip
80	it if the sweep budget isn't actually there.
81
82	## The Low-Leverage Truth
83
84	A 2026 240-H100-run study (arXiv 2603.19335) is
85	the load-bearing evidence behind the table above:
86	**loss-function choice is worth roughly 1
87	percentage point of leverage, model scale is
88	worth roughly 50.** Zero of 20 DPO variants tested
89	beat vanilla DPO. Rankings also **invert with
90	scale** — a variant that wins in a small pilot can
91	lose at deployment size.
92
93	Two practical consequences:
94
95	- Don't spend a routing decision agonizing over
96	DPO-variant bake-offs. The table above is
97	sufficient; deeper variant selection is
98	low-leverage compared to data quality and scale.
99	- **Validate at deployment scale before trusting a
100	ranking.** A method comparison run on a small
101	pilot model doesn't transfer to the production
102	size class — re-check the winner once scale
103	changes.
104
105	This is also why the Method Selection table above
106	is deliberately short: it encodes the ~1pp lever,
107	not a ranking of DPO variants that the same study
108	shows doesn't hold up across scale. Treat any
109	variant-selection advice that isn't in that table
110	— including advice that claims a specific variant
111	"wins" — as unproven until it's been validated at
112	the target deployment size.
113
114	## Production Pattern: Iterative On-Policy DPO
115
116	A single offline DPO pass on a static preference
117	dataset is a starting point, not the production
118	pattern. The policy drifts away from the
119	distribution the pairs were sampled from as
120	training proceeds, and a static dataset goes stale
121	against that drift. Production pipelines run DPO
122	iteratively and on-policy instead:
123
124	1. Sample completions from the current policy
125	checkpoint.
126	2. Score or rank the completions (reward model,
127	judge, or task grader).
128	3. Run a DPO pass using the current checkpoint as
129	the reference model.
130	4. The resulting checkpoint becomes both the new
131	policy and the new reference for the next
132	round.
133
134	Repeat. Each round's reference model is the prior
135	round's output, not a fixed initial checkpoint —
136	that's what keeps the preference signal on-policy
137	instead of scoring against an increasingly stale
138	distribution.
139
140	A single-pass DPO run is still a reasonable first
141	iteration — it just isn't the whole pipeline. Plan
142	for at least one more round once the first
143	checkpoint exists, rather than treating pass one
144	as the finished artifact.
145
146	## Pair Construction
147
148	Build DPO/ORPO pairs from **same-task
149	passing-vs-failing trajectories** — two attempts
150	at the same underlying task, not unrelated
151	best-and-worst examples pulled from different
152	tasks. Within that trajectory set, select the
153	rejected member at **μ−2σ of the reward
154	distribution, never the minimum**. Naive
155	best-vs-worst pair construction (max reward vs.
156	absolute minimum) degrades as scale increases; the
157	μ−2σ selection is more robust to the same scale
158	sensitivity the low-leverage study surfaced above.
159
160	```
161	sorted_by_reward = sort(trajectories, key=reward)
162	chosen = sorted_by_reward[-1] # highest reward
163	mu, sigma = mean(rewards), stdev(rewards)
164	rejected = closest(sorted_by_reward, mu - 2 * sigma)
165	# NOT sorted_by_reward[0] — the absolute minimum
166	# is the naive best-vs-worst construction that
167	# degrades as scale increases.
168	```
169
170	For the mechanics of turning graded traces into
171	these pairs — including rejection sampling and
172	judge-scored delta selection — see
173	`trace-to-training-data`.
174
175	## References
176
177	Complete TRL config blocks per method —
178	`DPOConfig`, `ORPOConfig`, `KTOConfig`, and the
179	SimPO sweep grid — plus Unsloth wrappers and a
180	catastrophic-forgetting note live in
181	`references/method-configs.md`. Those configs use
182	the same current-TRL API conventions established
183	in `lora-qlora-recipes`'s
184	`references/unsloth-trl-mapping.md`
185	(`processing_class`, not `tokenizer=`).
186
187	`references/method-configs.md` also carries the
188	catastrophic-forgetting note: a too-high learning
189	rate is the usual cause when a preference-tuned
190	checkpoint loses general capability, and the fix
191	is almost always to drop the LR toward the low end
192	of the range in the Method Selection table above
193	before reaching for any other remediation.
194
195	Related skills: `finetuning-method-selection`
196	routes here once preference pairs or unpaired
197	feedback exist; `lora-qlora-recipes` produces the
198	SFT checkpoint DPO/KTO/SimPO align (ORPO's
199	fused path can skip it); `trace-to-training-data`
200	converts passing/failing trajectories into the
201	pairs this skill's Pair Construction section
202	consumes.
203

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI

Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40 Ebay Seller ToolsData & AI · MIT·····●34/40 Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40 DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40

Verification score

Safety23/25

A1Destructive commands6/6
A2Data exfiltration6/6
A3Hardcoded secrets4/4
A4Prompt-injection guard5/5
A5Permission footprint · line 42/4

Transparency8.2/15

B1No obfuscation5/5
B2Declares dependencies0/4
B3Documented3.2/4
B4Pinned version0/2

Effectivenessnot scored yet

Needs real runs and real usage before it can be measured.

Costnot scored yet

Needs real runs and real usage before it can be measured.

Real-world usenot scored yet

Needs real runs and real usage before it can be measured.

31.2/40 on the part a machine can score. The badge only reaches Verified once Effectiveness and Cost have actually run.

wshobson

Original author

LicenceMIT

Repowshobson/agents

Repo stars39,748

Licence found inrepo

View on GitHub This is mine

README badge

The SVG endpoint is still being built. This is the final markdown shape.

[![AgentAlley](https://agentalley.ai/badge/preference-optimization.svg)](https://agentalley.ai/s/preference-optimization)

Submit your own