An agent that completes a task today may fail the same task tomorrow, so a single run of an evaluation is one draw from a distribution rather than a measurement of the agent. Comparing two versions of an agent therefore needs the tools of an experiment: repeated trials, comparisons made on the same tasks, an interval around every difference, and graders whose own errors are known.
A language model never sees the code behind a tool; it sees a name, a description, a schema for the arguments and whatever text comes back. Those four things are the whole interface, and the reader on the other side takes every word literally, cannot ask a follow-up question and fills any gap with a guess.
A system built on a language model can follow steps that its own code lays down, consulting the model only for particular judgments, or it can let the model choose each next step for itself.
Pretraining small models from scratch, teaching one to reason, and the hardware and arithmetic underneath.
11 posts
Open-source command-line tools for shared clusters: watching a job while it runs, reviewing it after it ends, and measuring disk use.
4 posts
What 51.9 million job postings and 12.8 million rental listings can and cannot show, and six degrees of separation at world scale.
6 posts
Shorter analyses in R from 2021 and 2022, most of them on TidyTuesday datasets.
108 posts
New: this site can answer questions about anything published on it, with links to the exact sections.
