A Desktop Grace Blackwell Machine as Research Infrastructure: Serving and Training Benchmarks for the NVIDIA DGX Spark

Generation speed

The headline figure is 106.3 tokens per second for a single stream with greedy decoding. Under sampling, which is what the application actually uses, the rate is slightly lower and declines gently as the answer grows longer.

Figure 2: Token generation rate against the length of the requested output, for the same model on the Spark at four-bit precision and on an H200 at sixteen-bit precision. Direct labels give the rate in tokens per second. The Spark’s decline across the range is 11 per cent; the H200’s rate is flat.

Forty per cent is a much better showing than the 7.0 per cent bandwidth ratio predicts, and the reason is precision. Generating a token requires reading the active weights out of memory, and four-bit weights are a quarter of the bytes of sixteen-bit ones. A card starved of bandwidth therefore gains far more from quantisation than a card with bandwidth to spare. Seven per cent of the bandwidth, multiplied by four times fewer bytes per token, lands close to the observed ratio.

The number that matters to a reader, and it is not the first one

Nemotron-3.5-Lightning is a reasoning model. Before it answers, it writes out a chain of thought, and the serving stack streams that trace separately from the answer. This makes “time to first token” ambiguous in a way that matters.

Figure 3: Two latencies on the same request. Time to first token is what a monitoring dashboard records; time to the first token of the answer is what a reader waits for. The right panel gives the number of answer tokens produced under four output budgets, three runs each.

A monitoring dashboard watching this deployment would report a 97-millisecond time to first token and conclude the service is fast. A reader waits eighteen seconds. On one measured 2,465-token answer the first visible word arrived at 23.29 seconds of a 25.09-second turn: ninety-three per cent of the wall clock was deliberation.

The consequence for anyone deploying this class of model is sharper than a latency figure. At output budgets of 64, 256 and 1,024 tokens the model produced no answer tokens at all, three runs each. The reasoning consumed the entire allowance and the stream was cut off mid-thought. The deployment sets a budget of twelve thousand tokens, which had looked generous and is in fact close to the minimum that works. A reasonable-seeming cap of 512 or 1,024 tokens would make this model appear broken.

The application conceals the wait by streaming the reasoning trace to the reader, which is why it feels responsive despite the underlying number.

Smoothness

Average generation rate says nothing about whether a stream stutters. The distribution of gaps between tokens does.

Figure 4: Distribution of the interval between consecutive tokens, pooled over 1,670 gaps on the Spark and 2,046 on the H200. Note the logarithmic vertical axis. The Spark’s ninety-ninth percentile is almost exactly twice its median, which is the cost of one rejected speculation.

The Spark’s stream is smooth in ordinary operation: the ninetieth percentile sits eighteen per cent above the median. The tail is a single phenomenon rather than general jitter, and identifying it required a separate experiment described below. The H200’s ratio of 1.014 between ninety-ninth percentile and median is what a card with three terabytes per second of bandwidth looks like when it never has to guess.

Reading a long document

Prefill is the phase that processes the prompt before generation begins. Its throughput rises with prompt length as the hardware fills, then falls as attention’s quadratic term takes over.

Figure 5: Prefill throughput and time to first token against prompt length, on cold prompts with no cache reuse. The vertical marker is the point of peak throughput. Beyond it the rate declines because attention cost grows with the square of the sequence length while the key-value cache is written through 232 GB/s.

A fifty-eight-thousand-token prompt takes ten seconds before the first token appears. This is the Spark’s weakest phase relative to a datacenter card, at 16.6 per cent of an H200, because prefill is compute-bound and compute is where the gap is widest.

Long context is nearly free after that

Conventional attention keeps a key-value cache that grows with the sequence, and reading it back dominates generation once the context is long. This model is a hybrid: most of its layers carry a fixed-size recurrent state instead.

Figure 6: Token generation rate against how much context is already in the window, from two independent runs using different samplers and different prompt sets. Both are flat. On a conventional attention-only model of this size the rate would visibly decline as the cache grew.

Across a hundred-fold increase in context the greedy run varies by 1.3 per cent and the sampled run by 3.3 per cent. For a retrieval application this is the single most useful property of the architecture: adding more retrieved material to a prompt costs prefill once and does not slow the answer down.

What speculative decoding is actually worth

The engine runs multi-token prediction, a form of speculative decoding in which a small draft head proposes the next token and the main model verifies it. The published acceptance rate is 97.6 per cent and each engine step emits 1.96 tokens, from which it is tempting to conclude that generation is nearly twice as fast as it would otherwise be.

That conclusion is wrong, and establishing so required running a second engine on the same machine with the same weights and the same flags, varying only the speculation depth.

Figure 7: Speculative decoding depth against three measures, from a second engine instance differing from the deployed one only in this setting. Tokens per engine step rise with depth while the time each step takes rises faster, so throughput peaks at a depth of one.

Measured directly, speculation is worth between 1.14 and 1.19 times, not two. The mechanism is visible in the third panel: a speculative step emits 1.7 times the tokens but takes 1.49 times as long, because it must run the draft head and then verify its proposal. Net throughput is the ratio of those two effects, and only one of them appears in an acceptance rate.

Going to a depth of two makes matters worse. The second draft token is accepted only sixty-six per cent of the time against the first token’s eighty-two per cent, and it is paid for on every step regardless. At a thousand-token output, depth two is indistinguishable from no speculation at all. The shipped default of one is correct.

One consolation for the deeper setting is worth noting: it produces by far the most even stream, with a ninety-ninth percentile only 1.12 times its median, because step time comes to be dominated by the fixed cost of drafting and verifying rather than by whether a guess landed.

How many people can use it at once

The deployed engine reserves a quarter of the memory pool and eight concurrent request slots, deliberately, so that other work can share the machine. Both are policy rather than hardware limits, so a second configuration gave the engine ninety per cent of the pool and thirty-two slots.

Figure 8: Aggregate and per-stream generation rate against the number of simultaneous requests, for two Spark configurations and the same model on an H200. The deployed configuration saturates at eight slots: its sixteen-stream point gains four per cent of throughput and its median wait rises to 3.41 seconds because the ninth request queues.

Giving the engine the whole machine raises peak aggregate throughput from 324 to 664 tokens per second, and the deployed configuration only reached its ceiling by queueing: at sixteen streams its median wait was 3.41 seconds, against 0.388 seconds for the larger configuration carrying twice the load. But single-stream generation falls from 107.9 to 84.3 tokens per second, and the ninety-ninth-percentile gap between tokens degrades 4.6 times across the range.

The rule that follows is simple. Choose the small configuration for a handful of users who each want a fast answer, and the large one for many users who will tolerate twenty-two tokens per second each.

Whether the answers are any good

Speed is worthless if the assistant is wrong. The application ships an evaluation harness with a labelled question bank, and it was run against this deployment: 116 tool-driven turns, scored for whether the assistant answered, whether it searched first, whether it cited the correct page, and whether it correctly declined questions the corpus cannot answer.

The same 116 turns were then run against the same checkpoint at sixteen-bit precision on an H200, which isolates what four-bit quantisation costs.

Figure 9: Agentic evaluation over 116 tool-driven turns, comparing the four-bit deployment against the same checkpoint at sixteen-bit precision on identical questions. Higher is better throughout. Tool calls never malformed in either run.

Both configurations answered every question, always searched before answering, and never emitted a malformed tool call in 116 turns. The four-bit deployment cited the correct page 95.7 per cent of the time against 97.0, and refused correctly 82.6 per cent of the time against 87.0: differences of roughly one and two turns respectively. Defects per answer were 0.11 for four-bit against 0.23 for sixteen-bit, which favours the quantised run, but that difference rests on a single small-count category and the two runs also differ in engine version, so the defensible conclusion is the null one. Quantisation does not measurably degrade this assistant.

The assistant’s genuine weakness lies elsewhere. Eight of its thirteen defects are a single class: answering a question it should have declined. That is the direct cost of a prompt rule introduced to fix the opposite failure, in which the model declined without searching. One rule, two failure modes, and it trades between them.

What the agentic machinery costs

Every turn of this application carries tool schemas, which engages the serving engine’s guided decoding path: a grammar compiled for the request and a mask applied to the model’s output at every step. This is overhead that a plain generation benchmark never sees.

Measured on identical prompts, attaching tool schemas costs fifty-six milliseconds of time to first token and nothing measurable afterwards: generation rate and inter-token latency are unchanged within noise. At the 4.6 tool rounds the assistant averages per turn, that is about 0.26 seconds of a 10.4-second turn, under three per cent. A forced JSON output schema costs nothing at all.

Where a turn’s time goes, and what it costs to run

Figure 10: Composition of a median assistant turn, and generation energy for the two machines. The turn is decode-bound, which is why speculative decoding is the lever that matters and additional prefill throughput would buy about a fifth of it.

The energy comparison requires a caveat. The Spark’s thirty-three watts is the whole module, processor and memory included, while the H200’s 693 watts is the accelerator board alone and excludes its host. The true system-level gap is therefore wider than four times, not narrower. Either way, a twenty-one-fold difference in power for a 2.6-fold difference in throughput is the entire argument for putting a model on a desk.

The machine also does not derate under sustained load. Over seventy-eight consecutive load windows, throughput fell 1.5 per cent, peak temperature reached 81 degrees, power averaged 88 watts, and no throttling was reported.

Throughput

Figure 11: Training throughput for an identical 2.06-billion-parameter recipe on one GPU, at each card’s best configuration. Missing bars are not omissions: an L40S with 46 GB cannot hold this recipe at any batch size, and eight-bit training is unavailable before the Hopper generation.

Two observations that a single ratio hides. The forty-six-gigabyte L40S fails at every batch size tried, because thirty-two-bit master weights plus the optimizer’s two moment estimates for 2.06 billion parameters occupy about thirty-three gigabytes before a single activation is stored. The Spark’s 121.7 GiB holds the recipe comfortably. And eight-bit training does not merely underperform on an A100, it does not run: the arithmetic format the recipe requires was introduced with the Hopper generation, and the compiler reports the type as unsupported.

Tuning is worth more than three times

The first measurement taken on the Spark was 1,149.9 tokens per second. The last was 3,608.6, on the same hardware and the same model.

Figure 12: Each successive change to the training configuration on the Spark, cumulative. The largest single contribution is compilation. The fourth rung is not a software change: it is stopping the inference engine that shares the memory pool.

Compilation alone is worth 2.35 times, and the same multiplier appears on the H100, so it is not an artefact of the unusual architecture. Two rungs are worth dwelling on.

Eight-bit precision is the best single knob on this machine, and not primarily for speed. It gives 3,608.6 tokens per second against 3,338.1 for the best sixteen-bit configuration, using sixteen fewer gigabytes at twenty-two per cent less power. In energy terms that is 60.6 tokens per joule against 41.7, an improvement of forty-five per cent. On a machine whose case for existing is energy, that is the number that matters.

Gradient checkpointing turns out not to matter. The recipe enables it by default, which is correct on an eighty-gigabyte card where activation memory binds. Disabling it on the Spark is worth thirteen per cent at a fixed batch size. But checkpointing frees memory, that memory converts into batch size, and at each policy’s own best batch the two are 1.1 per cent apart. The honest advice is that either is fine and this is not where the wins are, which is duller than the fixed-batch comparison suggests and is what a practitioner should act on.

The optimal sequence length belongs to the model, not the card

Figure 13: Training throughput against sequence length at a fixed micro-batch of one, on four cards spanning an eightfold range of arithmetic throughput. Every card peaks between two and four thousand tokens. The vertical marker is the sequence length the project’s own fine-tuning configuration uses.

The peak sits between two and four thousand tokens on every card, across a 7.9-fold range of arithmetic throughput and a 14-fold range of bandwidth. It is therefore a property of the model’s shape rather than of the hardware, which makes it a portable tuning result: choose the sequence length once for the model. Beyond the peak, a fine-tune at 13,568 tokens pays roughly 1.7 times per token what the same model pays at 4,096, on every card, within two per cent.

Does the Spark train the same model?

Every figure so far measures how fast the Spark trains. None of them would notice if its arithmetic diverged, and there is reason to think it might: a different PyTorch on a different processor architecture selects different attention and reduction kernels, so bit-identical results are impossible. If the loss trajectory drifted, the claim that one can pretrain on this machine would be wrong for a reason no throughput benchmark could detect.

Figure 14: Training loss over forty optimizer steps from a fresh initialisation on each card, with identical seed, recipe, optimizer and tokens at identical offsets, compilation disabled and sixteen-bit arithmetic on both sides. The inset value is the difference at the first step, before any accumulated divergence is possible.

The first step agrees to 7.7 parts per million, which is the important result: the weight initialisation, the data pipeline and the forward pass are equivalent across the two software stacks. From there the runs drift as two valid but different floating-point kernel selections should, by 0.37 per cent on average, with total loss reduction matching within 0.6 per cent. The trajectories stay parallel. The Spark trains the same model, 5.5 times more slowly.

This comparison very nearly produced a wrong answer. The first attempt had the two cards reading differently sized data files, and because the batch sampler draws offsets from the file length, the two runs saw entirely different tokens. The resulting trajectories differed by a plausible-looking 1.88 per cent, which measured the data rather than the arithmetic. Pinning the addressable range to an identical size on both sides is what produced the figure above.

The capability no single datacenter card has

The most interesting training result on this machine is not a rate. Consider fine-tuning the thirty-billion-parameter model with low-rank adaptation, which trains a small number of added parameters while the base model stays frozen.

Figure 15: Memory required to fine-tune the thirty-billion-parameter model with low-rank adaptation while simultaneously serving it, against what two accelerators provide. Both components are measured: 76.4 GB for the fine-tuning process and 27 GB for the serving engine. The dashed line is an eighty-gigabyte card’s capacity.

Low-rank adaptation of this model runs at 85.3 tokens per second in 76.4 gigabytes, training 1.9 million parameters, or 0.0059 per cent of the model, at 43.9 watts. Add the twenty-seven gigabytes the serving engine holds and the pair needs 103.4 gigabytes of the available 121.7. An eighty-gigabyte card can host the fine-tuning run or the served copy, but not both.

At 85.3 tokens per second that is about 307,000 tokens an hour, or 7.4 million a day: a few-million-token adaptation pass is an overnight job on a machine that is otherwise idle overnight. It is not a machine for a ten-billion-token fine-tune. But the shape of the work it enables, improving a deployed model on the hardware that deploys it, with no second machine and no maintenance window, is genuinely unavailable on a single datacenter accelerator.

The cost of sharing one pool

Everything on this machine competes for the same 121.7 GiB, and that has a price. A co-resident job costs the serving application seventy-four to seventy-seven per cent of its generation rate, measured twice by different means. In the other direction, the serving engine’s twenty-seven gigabytes cost a training run six per cent of its peak throughput once the batch size is re-tuned around it, and twenty-four per cent if it is not.

That last distinction was expensive to learn. An earlier version of these measurements concluded that filling memory on this machine was actively counterproductive, because throughput collapsed twenty per cent as the batch grew. Repeating the identical configurations with the serving engine stopped, the same batches recovered completely and the largest became the fastest. The collapse was contention with a neighbour, not a property of unified memory, and only running both conditions revealed which.

Two failure modes that do not exist on a discrete card

Unified memory removes the boundary between host and accelerator, and with it two guarantees that ordinary GPU code relies on.

A configuration that would merely fail on a discrete card can stop the whole machine. Requesting an over-large batch on an H100 raises an out-of-memory error, the harness catches it, prints a line, and continues; this happened repeatedly during these measurements and cost nothing. The same request on the Spark draws from the pool the operating system is using. A fine-tuning configuration needing about 151 gigabytes of a 121.7-gigabyte machine did not fail: it took the host off the network for two and a half hours, and the kernel eventually resolved it by killing the serving engine. The machine has sixteen gigabytes of swap, which is precisely what converts an immediate clean failure into a prolonged one. The diagnostic that distinguishes a healthy memory-heavy run from a dying one is the kernel’s own pressure metric rather than free memory: a later run holding eighty-seven gigabytes reported zero stall at ninety-six per cent utilisation, while the incident ran at a load average of fifty-eight.

Capacity planning must count the page cache. A second attempt to fine-tune the thirty-billion-parameter model was killed seventy-three per cent of the way through loading its weights, with room to spare on paper: fifty-seven gigabytes of weights plus a twenty-seven-gigabyte engine is eighty-four of 121.7. What the arithmetic omitted was eighty-two gigabytes of page cache, left over from having written the sixty-two-gigabyte checkpoint to disk minutes earlier. Because those file pages were actively mapped rather than clean and evictable, the kernel killed the loader instead of reclaiming them. On a discrete card the page cache lives in host memory and is irrelevant to what fits in accelerator memory. Here they are the same pool, and a recently touched file is occupied memory.

A related trap sits in the most ordinary line of model-loading code. Reading a checkpoint to host memory and then copying it to the device is two allocations of different memory on a discrete card, and two allocations of the same memory here: a fifty-seven-gigabyte model transiently needs 114 gigabytes. Pinning each shard directly to the device as it is read keeps the peak at one copy.

None of these are arguments against the machine. They are the cost of the property that makes it interesting, and all three are avoidable once known.