I am a computational scientist at the University of Chicago. I write long-form articles on training language models from scratch, on open-source tools for high-performance computing, and on what tens of millions of job postings and rental listings can and cannot show.
Setting a language model’s temperature to zero is the usual way to ask it for the same answer every time. It does not reliably produce one, and the providers of hosted models say so in their own documentation.
Training a language model across several machines costs something that training on one machine does not. At every optimizer step, each GPU computes gradients, the corrections the model learns from, and every GPU’s copy has to be added together.
The NVIDIA DGX Spark is a desktop computer containing a GB10 Grace Blackwell system on a chip: one GPU with 48 streaming multiprocessors, twenty ARM cores, and 121.7 GiB of LPDDR5X memory that the processor and the accelerator share as a single pool.
Pretraining small models from scratch, teaching one to reason, and the hardware and arithmetic underneath.
8 posts
Open-source command-line tools for shared clusters: watching a job while it runs, reviewing it after it ends, and measuring disk use.
4 posts
What 51.9 million job postings and 12.8 million rental listings can and cannot show, and six degrees of separation at world scale.
6 posts
Shorter analyses in R from 2021 and 2022, most of them on TidyTuesday datasets.
108 posts
New: this site can answer questions about anything published on it, with links to the exact sections.
