slurmpast: Post-Mortem Analysis for Finished HPC Jobs

Three instruments, three moments

The three questions that arise around a submission need three different instruments, because the evidence for each exists at a different time.

Before the run. What should be requested for work that has never been run? slurmate constructs the request.

During the run. Is the job using what it was given? slurmwatch reads the compute node live. It is the only one of the three that can measure a job’s genuine memory requirement, because that quantity exists only while the job is running.

After the run. What happened, and what should the next submission ask for? That is slurmpast. It is the only one of the three that reads every run of a piece of work at once, and several of the most expensive mistakes in this record are invisible within any single run.

The instrument

  58 jobs in 8 workloads · 56.9% completed · 184 GPU-hours total, 91 of them never used
  ordered by compute used (1 GPU-hour = 16 CPU-hours) · "#" stands for a name's digits

  #    JOB NAME               PARTITION  RUNS  FLAGGED CPU / GPU-HOURS LAST RUN
  -------------------------------------------------------------------------------
  1    node-evaluation        test          1        1     336 / 84    2026-07-26
  2    midtrain               test         14        -     105 / 79    2026-07-18
  3    cot-exp                test         20       14      61 / 10    2026-07-20
  4    ddp-pretrain           test          1        1      48 / 6     2026-07-22
  5    att-speed-#            test          7        -      56 / 5     2026-07-12

job 5100044  COMPLETED

  ● TIME   █░░░░░░░░░░░░░░░░░         4.2%   · 00:20:00 of the 08:00:00 limit
  ● CPU    █░░░░░░░░░░░░░░░░░         5.9%   · 0.9 of 16 cores busy
  ● MEM      no percentage        32.5 GiB   · an upper bound, over the 32.0 GiB limit

  findings

  [WARN] Peak memory reads above the limit, yet nothing was OOM-killed
        32.5 GiB against a 32.0 GiB per-node limit, so it is not this job's footprint:
        MaxRSS sums RSS across the process tree under jobacct_gather/linux,
        double-counting shared pages, so treat it as an upper bound.
        -> Measure the cgroup working set live (slurmwatch) before sizing --mem.

  [INFO] Walltime request much larger than needed
        Used 00:20:00 of the 08:00:00 limit (4.2%).
        -> A tighter --time reaches backfill windows a long request cannot enter.

what to request next time
  From how each workload actually ran. Over-requesting narrows which nodes can host it;
  under-requesting kills the run.

  midtrain  test · 14 runs
    --time            raise to 02:30:00   (from 02:00:00)
        longest of 14 completed runs took 01:52:49.
    --mem             already about right
    --cpus-per-task   already about right
    #SBATCH --time=02:30:00

  cot-exp  test · 20 runs
    --time            no advice
        14 of 14 timed-out runs consumed almost no CPU before hitting the wall: blocked,
        not slow -- a longer limit buys a longer hang.
    --mem             lower to 3G   (from 80.0 GiB)
        the most any run used was 1.9 GiB.
        ! MaxRSS sums RSS across the process tree under jobacct_gather/linux,
          double-counting shared pages, so treat it as an upper bound.

cross-run patterns

  [FAIL] Memory request is being hand-searched
        8 OOM kills for rc-tok-github_code with --mem walking 32.0 GiB -> 32.0 GiB ->
        17.0 GiB -> 12.0 GiB -> 12.0 GiB -> 14.0 GiB -> 16.0 GiB -> 18.0 GiB. Job
        5100044 then COMPLETED at 32.0 GiB -- a value that had already OOM'd.
        -> --mem is not the deciding variable: the same request both failed and
           succeeded. Something else changed (worker count, batch size, input shard).
           Find that before tuning memory again.

  [WARN] This work has failed repeatedly in the same way
        14 of 20 runs of cot-exp in test failed; 14 were TIMEOUT. Every one used the
        same --time=00:30:00. 7 GPU-hours consumed by the failures.
        -> 14 of these consumed under 10 CPU-seconds -- they hung rather than ran out of
           time. Raising the limit will not help; fix the blocking call.

The record these measurements come from

The accounting history holds 13,954 usable records for one account, after excluding 357 synthetic sleeper jobs written to exercise slurmwatch, which would otherwise distort every distribution here.

Attribution to Claude Code rests on the job identifier that sbatch prints back into a session transcript. That accounts for 991 submissions and, counting array tasks individually, 6,744 allocations, and those are the jobs used wherever this article discusses requests. The remainder cannot be attributed, for two structural reasons: 5,614 records predate the oldest session transcript still on disk, and a further 1,596 inside the transcript window were launched by a driver script or by hand, so no identifier was ever printed into a transcript. The agent’s own record of its work is partial and expires; the scheduler’s does not.

Figure 1: The 6,744 allocations attributable to Claude Code, by how long each one ran. A live reading has to be taken while the job occupies the node, so the two red bars are out of reach in practice and much of the third is as well. The accounting record covers all five bars equally, and is written whether or not anyone intends to read it.

This is the practical ceiling on the earlier article’s remedy. That article reported live readings for 21 jobs, and 21 was not a failure of diligence: it is approximately the number of runs a person can stand over. The record covers three hundred times as many, and consulting it costs nothing, because the writing was done by the scheduler.

Three failures no single reading can reach

A memory request settled by trial and error

The clearest single case in the record is a tokenizer run over a corpus of source code. It was submitted ten times in one evening, and the memory request changed on almost every attempt.

Figure 2: Ten submissions of one tokenizer, in the order they were made. The grey point is the memory requested; the coloured point is the highest figure the scheduler recorded for that run. Submissions two and three died before any figure was sampled. Where the coloured point lies to the right of the grey one, the run was killed while exceeding its limit.

Eight of the ten runs were killed for memory. The request descended from 48 GiB to 12 across the first six submissions while those kills were occurring, which is the opposite of the direction the evidence supported, then climbed back. The run that finally succeeded held 32 GiB, a value already killed twice at the start of the evening, and reached 24. Something other than the memory request was the deciding variable throughout, which is precisely the conclusion the cross-run check draws and the reason it refuses to recommend a further increment.

The sequence occupied a sixteen-core allocation for one hour of compute and nine hours of wall clock. Every fact required to stop after the third attempt was in the accounting database by then.

An array sized from two unrepresentative tasks

The second case is the more interesting of the two, because it is the failure mode of following the earlier article’s advice correctly.

A campaign scanned a text corpus in 142 shards, one array task per shard. Two single-task probes were run first; they reached 1.00 and 0.72 GiB against an 8 GiB request. Sized against those readings, the full array was submitted at --mem=6G, and 71 of the 142 tasks were killed for memory.

Figure 3: Per-task peak memory for all 142 shards, sorted from smallest to largest, taken from the corrected rerun so that no measurement is truncated by a kill. The two probes the request had been sized from are marked at their position in that order: both fall inside the smallest fifth of the corpus. Fifty-nine shards exceed 6 GiB here, while 71 were actually killed at that request, because a peak sampled every few seconds understates a brief spike.

Thirty-seven hours of task time were destroyed and recovered within forty-one minutes, which is an inexpensive version of this mistake. The general point is that a live reading measures the run that was watched. Where the work is one long job, that is the same thing as measuring the workload, and it is why the earlier article’s remedy succeeded on training runs. Where the work is an array over heterogeneous input, or a pipeline whose phases differ in footprint, one reading is a sample of size one, while the quantity that decides survival is the maximum over the whole population.

Repetition under an unchanged request

Across the whole record, 71 workloads ran at least five times and failed at least three. In 40 of them every failing run used an identical memory request and an identical time limit.

Figure 4: The twelve workloads with the most failures. Each is named by the name its own submission carried, since that is what the record supports: five of the twelve are exploratory runs called some variant of ‘test’, which fail often and are expected to. The faint bar is every run; the solid bar is the runs that failed. Red marks a workload in which every failing run carried an identical memory request and an identical time limit, so the request was never varied in response to the failures.

The reasoning experiment in the third row is the case the check was written for. Its 99 timeouts consumed 300 core-hours and 50 GPU-hours, every one of them at the same thirty-minute limit, and the verdict the tool returns is not “raise the limit”. The timed-out runs consumed almost no processor time before reaching the wall, which means they were blocked rather than slow, and a longer limit purchases a longer hang. That distinction requires both numbers together, and it is the difference between diagnosing the failure and paying for it a hundredth time.

Taken across all 71 workloads, the failures under an unchanged request account for 458 of the 1,206 failed runs.

The verdicts the record produces

Applying the sizing engine to every workload attributable to Claude Code produces a verdict per flag, or an explicit refusal where fewer than three runs support one.

Figure 5: Verdicts over the 470 workloads. The refusals are excluded from the bars and counted in the axis labels instead: most workloads in any real history are run once or twice, and three usable runs are required before a verdict is offered.

The processor row does not follow the pattern: it recommends an increase as often as a reduction. Most of those increases are single-core tasks that kept their one core saturated, and a saturated resource is evidence that more of it might help. On this history the recommendation is nonetheless frequently wrong, for a reason taken up under limitations.

The individual memory verdicts, for the workloads that consumed the most compute, are below.

Figure 6: The twelve most expensive workloads for which the record produces a memory verdict. The grey point is what the most recent run requested; the coloured point is what the record recommends. The vertical tick, and the middle number column, are the largest figure ever recorded for that workload. That is the quantity the recommendation is derived from, with a thirty per cent margin added.

The four increases are the valuable part of that figure and the part no cost-minimising rule would produce. In the gating experiment, the most recent run requested 24 GiB for work that had already reached 34.56 in an earlier run. The time limit produces two more of the same kind, not shown above: a family of experiments whose recent runs requested thirty-five minutes, where the longest completed run of that same work took four hours and ten minutes; and a pretraining slice whose longest completed run finished one minute and sixteen seconds inside an eight-hour limit. Neither had failed at the point the record was read.

The value of acting on them

Figure 7: Every allocation attributable to Claude Code, re-costed under the request its own history recommends. ‘Used’ is what the runs consumed: recorded peak memory held for the duration, processor time actually burned, and wall clock actually elapsed. Memory usage here rests on MaxRSS and is therefore an upper bound, so the true gap in the first panel is wider than shown.

Three readings, and the second is the honest limit of this whole approach.

The time limit collapses, because a completed run’s duration is an exact measurement and the recommendation is that duration plus a quarter. An over-long limit costs queue position rather than capacity, so it is the cheapest of the three to get wrong; it is also the one the record settles outright.

Memory closes about a fifth of its gap. The recommendation is derived from MaxRSS, which reads high, and adds a thirty per cent margin to a figure that is already an over-estimate. It is therefore systematically generous, and deliberately so, because a memory recommendation that is too small destroys a run. The remaining four fifths is exactly the territory a live reading of the working set can reach and the record cannot. The two instruments are not substitutes for one another.

Processor time moves in the wrong direction, by a fifth, because of the single-core increases. Accepting only the recommendations that reduce a request converts that into a saving of 4.9 per cent.

Allocations that held a card and computed nothing

Figure 8: An allocation counts as having computed nothing when it held its resources for the whole of its life while consuming essentially no processor time. A cancelled run that had already performed useful work does not qualify. The two rows are drawn as shares because the record as a whole holds eighteen times the graphics-card hours of the attributable portion; the hours themselves are printed inside the bars.

This is a post-mortem finding by construction. Nothing distinguishes a hung allocation from a working one at the time unless somebody looks; afterwards it is unmistakable, because the record shows hours of wall clock beside a processor total of a few seconds. The attributable portion performs considerably better than the record as a whole, which is the one comparison here that favours the agent: it launches work and moves on, so it rarely leaves an interactive allocation running overnight.

Failures that belong to the machine rather than the job

Figure 9: Hang rate by machine within a single workload of 1,098 placements, so that differences in the work itself cannot explain the differences between machines. Bars are 95 per cent Wilson intervals. Each machine is compared against that same workload on every other machine, computed with the machine under test left out. Eight machines had too few placements to test and are omitted, and one further machine’s interval clears the baseline without surviving the correction for testing eleven rows at once.

Node I received 403 of the 1,098 placements, more than any other machine, and is significantly better than the fleet. That is the confound the workload control exists to defeat: a table pooling the baseline across all placements would have compared Node I against a rate that was itself 37 per cent composed of Node I, dragging the comparison toward whatever that machine happened to do and concealing precisely the machines that ran the most work.

For an automated agent the practical value is separate from the statistics. Forty-five runs of this workload hung on the three machines the table names, holding 689 graphics-card hours between them. An agent reading only its own exit codes cannot reach that conclusion; it will reread its own code instead, which is what makes this the most expensive category of failure to diagnose.

The coverage of each instrument

Collecting the failures documented above, the matrix below records what each available signal reveals about each of them. It is deliberately unflattering to the tool this article introduces.

Figure 10: What each signal reveals about each failure documented in this article. ‘A live reading’ is slurmwatch on the run being watched; ‘the accounting record’ is slurmpast over every run of the work. Two cells deserve comment. A live reading is marked misleading on the array because the two tasks that were read peaked at about a seventh of what the corpus required. The accounting record is marked misleading on an over-large memory request because MaxRSS reads high, so sizing against it leaves capacity reserved that nothing will use.

The log, which is the one artefact almost every workflow actually inspects, reveals none of the seven. The exit status reveals five of them one run at a time, which is precisely the mode of failure that produced ten submissions of one tokenizer in an evening: each kill was informative, and none of them carried the previous nine. The single row on which the accounting record is not the answer is the first, which concerns the level of an individual run rather than a pattern across runs. That row belongs to the live reading, and it is the reason both instruments exist.

Interfaces for automated use

Everything above is available without the dashboard. --plain prints the same content as text, and --json emits all of it, 174 values per job, from analysis modules that import no third-party package.

from slurmpast import History, Sacct
from slurmpast.sizing import recommend

history = History(Sacct().history(user="you", since="now-30days"))
for advice in recommend(history.groups[0].jobs):
    print(advice.flag, advice.verdict, advice.suggestion)

The cost is what allows this to be a habit rather than an investigation. Over the full history used here, the sacct query takes 5.1 seconds; parsing it, grouping the runs into workloads, producing a sizing verdict for every workload and running the cross-run checks together take 0.4 seconds. There is no cache and no database, because at that price neither would justify its complexity.

The corresponding instruction is correspondingly small. Before submitting work that has been run before, read what it did last time; before resubmitting work that has failed, establish whether it has already failed in this way. Both are a single command, and the second is the one that repays the effort, because the strong prior on a failed job is to change the code.

Limitations

The processor recommendation is wrong at the bottom of its range. A job that saturates its single core is advised to request two, because the rule multiplies the observed peak by a fixed margin and rounds up, and one core times 1.2 rounds to two. For a genuinely single-threaded program that is not merely useless but inverted, and on this history it is frequent enough to convert a 4.9 per cent saving into a 19.6 per cent increase. The premise, that a saturated resource may be a bottleneck, is sound; the margin does not belong at a scale where the next available value is double.

The memory recommendation inherits MaxRSS. It is the only memory figure the scheduler retains for every job, and it reads high, so advice built on it errs in the direction that reserves capacity rather than the direction that destroys runs. The tool states this at the point of advice rather than in a footnote, and refers the question to slurmwatch. This is a genuine ceiling on what a post-mortem can achieve alone.

A verdict requires three runs. Most workloads in any real history are run once, and for those the record has nothing to say. The refusal is reported rather than filled in, which is correct, and it also means the coverage in the figures above is a minority of the workloads even though it is most of the compute.

The thresholds are heuristics from one cluster. What is measured is portable; what is judged is calibrated. Each threshold is a named constant, so recalibration means editing a number rather than the logic.

Attribution to Claude Code is partial, for the two structural reasons given earlier. The figures describing requests use only the attributable portion; the figures describing the record as a whole say so.

The record the agent does not keep

The earlier article concluded that Claude Code is not poor at HPC work but blind to one specific quantity, and that the remedy is an instrument rather than an instruction. This history extends that conclusion in a direction live readings alone could not show.

The blindness is not only to a measurement. It is to a history. A submission is written, a job runs, and whatever that job revealed about itself is discarded when the session ends. The transcript expires; the next session begins with no knowledge that this work has been run before and measured every time; and the request is composed again by the same reasoning that produced it originally. The tokenizer settled by trial and error, the array sized from two unrepresentative shards, the experiment that timed out ninety-nine times at an unchanged limit: none of these is a failure of reasoning about the job in front of it. Each is a locally correct decision taken without the previous ones in view.

The scheduler retained all of it. Every request, every outcome, every recorded peak, for every job either party has ever run, in a database that costs five seconds to read. What was absent was neither the data nor the diligence, but a tool that converts the record into the one sentence the next submission requires.

Availability

slurmpast is available on PyPI and GitHub under the MIT licence. It requires Python 3.10 or later and a scheduler with accounting enabled; where accounting is disabled or sacct is absent it prints a single line of explanation rather than a traceback. Continuous integration covers Python 3.10 through 3.13, both ends of the supported Textual range, and a machine with no Slurm installed at all, over a suite of 1,195 tests.

slurmpast --demo runs the entire dashboard against a synthetic history, which is the quickest way to establish whether it answers a question worth asking.