AgentAlley

Skills · Infrastructure & ops

Spark Memory & Thermal Ops

Unverified●32/40

Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add spark-memory-thermal-ops

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.

The whole source

No sign-in, no blur, nothing truncated
spark-memory-thermal-ops/SKILL.md231 lines7.8 KBRawView on GitHub
Frontmatter — 2 properties
namespark-memory-thermal-ops
descriptionManage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
1---
2name: spark-memory-thermal-ops
3description: Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Spark Memory & Thermal Ops
7 
8DGX Spark's GB10 chip has one 128GB unified
9memory (UMA) pool shared by CPU and GPU, and a
10sustained power ceiling well below its rated
11figure. Both break discrete-GPU assumptions:
12headroom isn't what `nvidia-smi` reports, and a
13run that starts fast will slow down mid-job
14with nothing misconfigured. This skill covers
15planning memory headroom, working an actual
16OOM, and watching thermals across a long job.
17For launch-time failure modes (ABI mismatches,
18flash-attn, playbook breakage), see
19`spark-training-gotchas` — this skill assumes
20the job starts.
21 
22## Common Issues Quick Reference
23 
24| Situation | Do this |
25|---|---|
26| Planning headroom before launch | Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model |
27| Job OOMs on unified memory | Work the OOM Ladder in order: flush, then batch/pack, then method downgrade |
28| Throughput drops mid-run | Check the power/temp log before assuming a config bug — see Thermal Monitoring |
29| Trainer + inference server both wanted | Run one at a time — see Concurrent Workloads |
30 
31## When to Use This Skill
32 
33- Sizing a training run against the 128GB pool
34 before launch — will this model, method, and
35 batch/pack combination fit.
36- A run OOMs mid-load or mid-step and the
37 remediation order matters — what to try first,
38 second, third.
39- Watching temperature and power during a
40 multi-hour job, deciding whether a slowdown is
41 thermal throttling or something else.
42- Planning to run a trainer alongside an
43 inference server (vLLM, Ollama) on the same box.
44 
45## UMA Memory Model
46 
47Spark has no separate GPU VRAM — the GPU and
48CPU share one 128GB pool. Two consequences:
49 
50- **`nvidia-smi` and `cudaMemGetInfo`
51 underreport pressure — or report nothing at
52 all.** Both report CUDA-allocator-visible
53 memory, not the pool's actual state — a box can
54 show headroom in `nvidia-smi` and still OOM,
55 because page-cache and mmap'd pages the
56 allocator doesn't see consume the same pool. On
57 some driver/setups, the memory query returns
58 `[N/A], [N/A]` outright instead of a number — a
59 script grepping for a numeric value there gets
60 nothing, not a misleading undercount (see
61 `spark-training-gotchas` gotcha G3).
62 
63- **Model load is a transient peak, not the
64 steady state.** Loading safetensors weights
65 mmaps the file, then copies into CUDA
66 tensors — for a window during load, both the
67 mmap'd pages and the CUDA copy count against
68 the pool at once. A model that fits while
69 training can still OOM during load if headroom
70 was sized for the post-load footprint instead
71 of this doubled transient.
72 
73Plan and diagnose with `free -g`, not
74`nvidia-smi`:
75 
76```bash
77free -g | awk 'NR==2 {print "free:", $4, "GB"}'
78```
79 
80Rule of thumb: take that free figure, subtract a
81few GB for OS/driver overhead, and budget against
82the result — not the 128GB spec number.
83The worksheet in `references/uma-accounting.md`
84accepts parameter count, dtype, and method as
85input, and returns a memory estimate to compare
86against known anchors.
87 
88### Planning Sequence
89 
90Before launch, work through these in order:
91 
921. Read `free -g`; subtract OS/driver overhead
93 for the budget.
942. Estimate weights + optimizer + gradients +
95 activations from `references/uma-accounting.md`.
963. Compare against the closest anchor (70B
97 QLoRA, 27B LoRA, 9B full FT), not the
98 estimate alone.
994. If the estimate is close to the budget, start
100 with shorter packing or a smaller batch —
101 cheaper than hitting the OOM Ladder mid-run.
102 
103### Example: Sizing a 70B QLoRA Run
104 
105A sanity check of the worksheet formula against
106the ≈40GB anchor:
107 
108```python
109params = 70e9
110weights_gb = params * 0.5 / 1e9 # NF4, step 1
111adapter_gb = 0.5 # step 5, negligible
112total_gb = weights_gb + adapter_gb # + activations
113print(f"{total_gb:.0f}GB before activations")
114```
115 
116Weights alone land near the ≈40GB anchor — a plan
117estimating far above that for the same model
118class is a signal to recheck dtype and method.
119 
120## The OOM Ladder
121 
122When a job OOMs on unified memory, work this
123ladder in order. Each step is more disruptive
124than the last — don't skip ahead:
125**reducing batch size is never step 1.**
126 
1271. **Flush the buffer cache.** Page cache from a
128 previous run or a large dataset read often
129 accounts for GB of the "missing" headroom.
130 This costs nothing but a rerun and doesn't
131 touch the job's configuration:
132 
133 ```bash
134 sync; echo 3 > /proc/sys/vm/drop_caches
135 ```
136 
137 Needs root; a between-run reset, not a
138 mid-training step. See
139 `spark-training-gotchas` (gotcha G3) for the
140 full diagnostic behind this step.
141 
1422. **Reduce batch size or packing length.** Only
143 after a flush fails to free enough headroom,
144 cut batch size or packing length — the first
145 step that changes what the run does. Prefer
146 packing length first; it drives activation
147 footprint more directly at long context.
148 
1493. **Downgrade the method: bf16 LoRA before
150 QLoRA.** If flushing and shrinking batch/pack
151 still OOM, drop the method a tier — bf16 LoRA
152 is next, not the reverse. QLoRA's bitsandbytes
153 dequantization buffers are transient CUDA-side
154 allocations that can OOM before an equivalent
155 bf16 LoRA run would, even though QLoRA's
156 steady-state footprint is smaller. A QLoRA OOM
157 is not proof the model doesn't fit.
158 
159Fall back further (smaller model, multi-Spark)
160only after all three steps and the job still
161won't fit.
162 
163## Thermal Monitoring
164 
165Multi-hour runs push into Spark's sustained
166power ceiling, well under the rated figure —
167expected platform behavior, not a symptom to
168explain away:
169 
170- Sample temperature and power alongside the
171 training logs, not after a slowdown is
172 noticed — every 30-60 seconds correlates a
173 throughput drop with a thermal event. Keep
174 the CSV output format `assets/thermal-sample.sh`
175 writes, so timestamps line up against the log:
176 
177 ```bash
178 bash assets/thermal-sample.sh 30 thermal.log
179 ```
180 
181- **A sustained ~100W power draw is the platform
182 cap, not a configuration bug.** Don't re-tune
183 batch size or precision to "fix" a plateau
184 that's the box behaving normally under load.
185 If temperature climbs while power stays flat
186 under the rated 240W figure, that's the
187 signature to recognize.
188 
189- Log throttle events explicitly instead of
190 letting a run silently slow down unrecorded. A
191 run whose per-step time doubles two hours in
192 should show that in the log, correlated against
193 the thermal sample at that timestamp. Full
194 throttling diagnostics: `spark-training-gotchas`
195 (gotcha G4).
196 
197## Concurrent Workloads
198 
199Because the 128GB pool is global, eviction
200happens without either process's logs showing
201an OOM:
202 
203- The one-heavy-job rule applies to **uncapped or
204 near-capacity** workloads — an uncapped trainer
205 and inference server (vLLM, Ollama) compete for
206 the same pool. A small, capped workload doesn't:
207 a <4GB LoRA fine-tune coexists fine alongside
208 vLLM capped at `gpu-memory-utilization<=0.5` —
209 check the other process's cap, not just its
210 presence, before stopping it.
211 
212- Inference servers evict trainer pages silently
213 under uncapped/near-capacity contention, and
214 vice versa — neither logs an error, so a slow
215 run or lost KV cache is a contention symptom to
216 check for. Stop unrelated *uncapped* servers
217 before a long or full-pool run.
218 
219Check for GPU-resident processes first:
220 
221```bash
222ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep
223```
224 
225This procedure complements `spark-training-gotchas`
226(gotchas G3, G4, G6) — that skill covers launch-time
227failures; this one, the running job.
228 
229Memory math worksheets:
230`references/uma-accounting.md`.
231 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Infrastructure & ops
Paper Poster (HTML): measurement-gated poster generationDEFAULT poster pipeline — build an academic conference poster (ICML/NeurIPS/ICLR/CVPR/...) as a single HTML/CSS file with measurement-driven hard gates, real paper figures, a two-hue design-token system, and print-ready PDF via headless Chromium. Use when theInfrastructure & ops · MIT●····●36/40Brand Monitoring 📡Brand monitoring tool for tracking mentions across social media platforms. Monitor Reddit, Google News, YouTube, and DuckDuckGo for brand mentions. Includes sentiment analysis, trend tracking, crisis detection, and competitor comparison. No API key required foInfrastructure & ops · MIT◐····●34/40Secrets ManagementImplement secure secrets management for CI/CD pipelines using Vault, AWS Secrets Manager, or native platform solutions. Use when handling sensitive credentials, rotating secrets, or securing CI/CD environments.Infrastructure & ops · MIT◐····●32/40Deployment Pipeline DesignDesign multi-stage CI/CD pipelines with approval gates, security checks, and deployment orchestration. Use this skill when designing zero-downtime deployment pipelines, implementing canary rollout strategies, setting up multi-environment promotion workflows, or debugging failed deployment gates in CI/CD.Infrastructure & ops · MIT◐····●31/40