Two earlier articles in this series described the two halves of building a language model from nothing: pretraining five generations of base models, and teaching one of them to reason. Both ended on the same conclusion — that the ceiling on a fine-tuned model is set by the base model beneath it, and that Argonne 3.

A companion article described how Argonne 3.5-base was pretrained: the same 2.88-billion-parameter architecture as Argonne 3.0, unchanged component for component, retrained on 88.84 billion tokens of web text and mathematics with a corrected learning-rate schedule, and finishing with a context window of 13,568 tokens that was trained rather than assumed.

rapiDU is an open-source command-line tool that answers the question du is normally reached for — what is taking up all the space? — and then answers four further questions that du cannot answer at all. It returns the same total as du to the byte, which is verified on every commit, and on a cold walk of a large parallel filesystem it has been measured returning that total five to seven times sooner.

slurmwatch is an open-source command-line tool that shows what a running job is actually doing to the hardware it was given. Point it at a job — or run it with no arguments and let it find the job itself — and it displays, live in the terminal, the processor time, the memory, and, where the machine has them, the graphics-card activity belonging to that job.

In an earlier article I described how a small language model can be taught to reason. That discussion rested on a claim worth restating: the capabilities of a fine-tuned model are largely determined by the quality of the underlying base model. The present article turns to that foundation directly — how the base models themselves were built, beginning from nothing more than a corpus of text and a randomly initialized set of parameters.

For the last few years, this blog has mostly been about tidy data, statistical models, and the occasional TidyTuesday chart. This post is different. Over the past few months I did something I had wanted to try for a long time: I built a small AI language model from scratch — the same basic kind of technology behind chatbots — and then tried to teach it to reason: to work through a problem step by step before answering, the way you might show your work on a math test.

In this blog post, I will use the tidymodels meta-package to predict the animal crossing rating. Previously, I had a blog post visualizing the same data set, and you can view it here. Since the data set contains user comments along with the game rating, and LASSO is a good model for using words as predictors, this blog post I will use bootstrap resampling method to illustrate how to tune the LASSO model and # of words as predictors in the feature engineering steps.

In this blog post, I will use LASSO model to predict the GDPR fines. Before this post, I have another post on visualizing the same data set. You can view it here. Also, the textrecipes package is used in the modeling process, as the data set privodes some text columns, which can be transformed as predictors.

In this blog post, I will use step_pca() provided by the recipe package to apply PCA analysis to a cocktail dataset. A similar blog post of mine, which you can view here, provides PCA analysis on the same dataset. The difference is that post uses svd() approach to carry out the analysis, but this post uses the tidymodels approach.

In this blog post, I will use a data set about beer brewing materials provided by TidyTuesday to make prediction for the monthly barrels of several beer materials. The tidymodels meta-package will be used, with bootstrap as the resampling technique. library(tidyverse) library(tidymodels) library(lubridate) theme_set(theme_bw()) brewing_materials_raw <- read_csv("https://raw.