Quantized Export
Unverified●32/40Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add quantized-exportWho is stuck, and on what
Export a promoted fine-tuned model in the right deployment format — merged safetensors, LoRA-only, GGUF with imatrix, or FP8. Use after a checkpoint passes promotion, when choosing a quantization format for a target device, or when an exported model fails its smoke test.
The whole source
Frontmatter — 2 properties
| name | quantized-export |
|---|---|
| description | Export a promoted fine-tuned model in the right deployment format — merged safetensors, LoRA-only, GGUF with imatrix, or FP8. Use after a checkpoint passes promotion, when choosing a quantization format for a target device, or when an exported model fails its smoke test. |
| 1 | --- |
| 2 | name: quantized-export |
| 3 | description: Export a promoted fine-tuned model in the right deployment format — merged safetensors, LoRA-only, GGUF with imatrix, or FP8. Use after a checkpoint passes promotion, when choosing a quantization format for a target device, or when an exported model fails its smoke test. |
| 4 | ---A5 — No allowed-tools declared — no way to tell what this skill may touch |
| 5 | |
| 6 | # Quantized Export |
| 7 | |
| 8 | The last stop after `checkpoint-promotion` |
| 9 | hands off a `PROMOTE` verdict: a checkpoint |
| 10 | that cleared the four-stage gate still isn't |
| 11 | deployed until it's exported in the right |
| 12 | format for its target runtime and proven to |
| 13 | still work post-export. A `REJECT` verdict |
| 14 | never reaches this skill — export starts only |
| 15 | from a promoted checkpoint. |
| 16 | |
| 17 | **Input:** a promoted checkpoint (or LoRA |
| 18 | adapter) plus the target deployment surface — |
| 19 | GPU class, serving stack, and whether |
| 20 | long-context/code/math workloads are in |
| 21 | scope. |
| 22 | **Output format:** an exported artifact in |
| 23 | the chosen format plus a smoke-test diff |
| 24 | report comparing 3–5 golden outputs |
| 25 | pre-export and post-export. |
| 26 | |
| 27 | ## Format Map |
| 28 | |
| 29 | Pick format by hardware and deployment shape, |
| 30 | not by habit — the wrong pick either wastes |
| 31 | throughput headroom or breaks silently on |
| 32 | specific workloads (see Workload Overrides). |
| 33 | |
| 34 | - **FP8 is the default on Hopper-class GPUs |
| 35 | and newer.** It preserves near-bf16 quality |
| 36 | at roughly half the memory, and it's the |
| 37 | safe first choice whenever the target GPU |
| 38 | supports it and no edge-device constraint |
| 39 | applies. |
| 40 | - **AWQ INT4 targets older GPUs** that predate |
| 41 | FP8 hardware support. **GPTQ is superseded |
| 42 | for new deployments** — don't reach for it |
| 43 | on a fresh export; AWQ has better accuracy |
| 44 | retention at the same bit width and wider |
| 45 | current tooling support. |
| 46 | - **GGUF with Q4_K_M quantization, built from |
| 47 | an imatrix, is the edge/llama.cpp format.** |
| 48 | Use it for local or CPU-adjacent |
| 49 | deployment, not for GPU-serving |
| 50 | throughput — it optimizes for footprint, |
| 51 | not tokens/sec on a datacenter GPU. |
| 52 | - **NVFP4 is for Blackwell-at-scale |
| 53 | deployments only — and explicitly NOT on |
| 54 | GB10.** NVFP4 on SM121 (GB10) runs **~32% |
| 55 | slower than FP8** because the hardware |
| 56 | lacks a native `cvt.e2m1x2` path unless the |
| 57 | kernel is compiled `sm_121a`. Choosing |
| 58 | NVFP4 on a GB10 target is a regression, not |
| 59 | an upgrade — pick FP8 there instead. |
| 60 | - **Merged vs. LoRA-only is a separate axis |
| 61 | from quant format.** A merged export folds |
| 62 | the adapter into the base weights: larger |
| 63 | artifact, no base-model dependency at serve |
| 64 | time. LoRA-only keeps the adapter separate: |
| 65 | much smaller artifact, but the serving stack |
| 66 | must load the exact same base model |
| 67 | alongside it — a mismatched or |
| 68 | wrong-revision base silently changes |
| 69 | outputs. Pick merged when artifact |
| 70 | portability matters more than storage; pick |
| 71 | LoRA-only when disk footprint or multi-adapter |
| 72 | serving matters more. |
| 73 | |
| 74 | ### Worked Picks |
| 75 | |
| 76 | The core format-selection tradeoff, read as a |
| 77 | lookup table for common scenarios: |
| 78 | |
| 79 | | Target | Workload | Format | |
| 80 | |---|---|---| |
| 81 | | Datacenter GPU | generic chat | FP8 | |
| 82 | | Datacenter GPU | long-context/code/math | FP8 or W8A8 — never INT4 | |
| 83 | | Older GPU generation | generic | AWQ INT4 | |
| 84 | | Edge device / laptop | llama.cpp serving | GGUF Q4_K_M + imatrix | |
| 85 | | GB10 | any workload | FP8 via vLLM nightly, or GGUF via llama.cpp locally — skip NVFP4 | |
| 86 | |
| 87 | ```yaml |
| 88 | # quick decision snippet — see the table above for the full map |
| 89 | hopper_or_newer: fp8 |
| 90 | older_gpu: awq-int4 |
| 91 | edge_llama_cpp: gguf-q4_k_m+imatrix |
| 92 | gb10_any_workload: fp8-vllm-nightly # never nvfp4 on GB10 |
| 93 | ``` |
| 94 | |
| 95 | ## Workload Overrides |
| 96 | |
| 97 | The Format Map above is a default, not a rule |
| 98 | that survives every workload. **Long-context, |
| 99 | code, and math workloads break at INT4** — |
| 100 | quantization error compounds across long |
| 101 | sequences and precise token-level reasoning in |
| 102 | ways that don't show up on short, generic |
| 103 | prompts. For any of these three workload |
| 104 | classes, **stay on FP8 or W8A8** even if the |
| 105 | target hardware would otherwise justify INT4 |
| 106 | on cost grounds. |
| 107 | |
| 108 | - Don't validate this override with MMLU or |
| 109 | similar broad-knowledge benchmarks — they |
| 110 | don't stress the failure mode. **Measure |
| 111 | with the actual task evals** — the goldens |
| 112 | and graders from `eval-harness-first`, run |
| 113 | through the exported artifact — because |
| 114 | INT4 degradation on long-context, code, or |
| 115 | math shows up as task-specific failures |
| 116 | (dropped context, broken syntax, arithmetic |
| 117 | errors) well before it moves a knowledge |
| 118 | benchmark. |
| 119 | - If a task eval regresses after an INT4 |
| 120 | export on one of these three workload |
| 121 | classes, the fix is switching format, not |
| 122 | re-tuning the quantization recipe — AWQ |
| 123 | and GPTQ variants at the same bit width |
| 124 | share the same compounding-error failure |
| 125 | mode on these workloads. |
| 126 | |
| 127 | ## The Smoke Test |
| 128 | |
| 129 | Export bugs are silent at the file level — a |
| 130 | malformed export still produces a |
| 131 | loadable artifact, so file-existence checks |
| 132 | prove nothing. **The smoke test is |
| 133 | mandatory for every export, with no |
| 134 | exception for a format that "should just |
| 135 | work":** |
| 136 | |
| 137 | 1. **Load the exported artifact in its actual |
| 138 | target runtime** — vLLM for FP8/AWQ, |
| 139 | llama.cpp for GGUF, not a quick |
| 140 | sanity load in a different framework than |
| 141 | the one that will serve it in production. |
| 142 | 2. **Run 3–5 golden prompts through it** — |
| 143 | pull these from the same `eval/goldens.jsonl` |
| 144 | `eval-harness-first` maintains, not a fresh |
| 145 | ad hoc set. |
| 146 | 3. **Compare each output against the |
| 147 | pre-export generation** for the same |
| 148 | prompt, same deterministic sampling |
| 149 | settings — greedy decoding (temperature 0) |
| 150 | and a fixed seed, persisted and reused |
| 151 | between the pre- and post-export runs, not |
| 152 | just nominally identical config. **For a |
| 153 | lossless export, byte match is the gate — |
| 154 | any diff is a bug.** For a **lossy** |
| 155 | (quantized) export, byte match is expected |
| 156 | to fail; the gate is task-grader verdict |
| 157 | agreement instead — see |
| 158 | `references/export-commands.md`'s |
| 159 | Smoke-Test Script Skeleton. |
| 160 | |
| 161 | Run this as a gate, not a manual check: |
| 162 | |
| 163 | ```bash |
| 164 | python smoke_test.py "$EXPORT_PATH" \ |
| 165 | eval/goldens.jsonl pre-export-outputs.jsonl |
| 166 | # non-zero exit on any pre/post mismatch |
| 167 | ``` |
| 168 | |
| 169 | ### Failure Signatures |
| 170 | |
| 171 | What export bugs actually look like, not a |
| 172 | clean pass/fail flag: |
| 173 | |
| 174 | - **Template mismatch** presents as garbled or |
| 175 | run-on output — the chat template baked |
| 176 | into the export doesn't match the one the |
| 177 | checkpoint was trained and evaluated |
| 178 | against, so turn boundaries or special |
| 179 | tokens land in the wrong place. |
| 180 | - **Wrong quantization applied to `lm_head`** |
| 181 | presents as off-template or semantically |
| 182 | nonsensical output that still looks |
| 183 | fluent — the output head lost precision it |
| 184 | needed even though the rest of the network |
| 185 | quantized cleanly. |
| 186 | |
| 187 | Never ship an export that skipped this step — |
| 188 | a checkpoint's `PROMOTE` verdict says the |
| 189 | un-exported checkpoint is good; it says |
| 190 | nothing about the export pipeline. Re-run on |
| 191 | any quant-method or runtime version bump, not |
| 192 | only after the first export. Runnable command |
| 193 | sequences for every format plus the |
| 194 | smoke-test script skeleton: |
| 195 | `references/export-commands.md`. |
| 196 | |
| 197 | ## Related Skills |
| 198 | |
| 199 | - `checkpoint-promotion` — the only valid |
| 200 | upstream source for this skill. A checkpoint |
| 201 | without a `PROMOTE` verdict doesn't reach |
| 202 | export. |
| 203 | - `eval-harness-first` — owns the |
| 204 | `eval/goldens.jsonl` this skill's smoke test |
| 205 | draws its 3–5 prompts from, and the task |
| 206 | evals the Workload Overrides section |
| 207 | requires for long-context/code/math |
| 208 | validation. |
| 209 | - `finetuning-method-selection` — its |
| 210 | `references/model-catalog.md` is the place |
| 211 | to check hardware-class assumptions (which |
| 212 | GPU generations a base model targets) before |
| 213 | picking a format off the Format Map above. |
| 214 | |
| 215 | **Spark users:** on GB10, GGUF via llama.cpp |
| 216 | works well for local serving, and FP8 serving |
| 217 | via vLLM nightly builds is the other proven |
| 218 | path — NVFP4 is the one format to avoid there |
| 219 | (see the Format Map exception above). Once the |
| 220 | `dgx-spark-ops` plugin is installed, defer |
| 221 | Spark-specific serving and thermal questions to |
| 222 | its skills rather than re-deriving them here. |
| 223 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.◐◐◐◐◐●35/40Ebay Seller Tools·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.●····●32/40