No. 132 — Glitter and 16 000 dimensions ⫶ Rubber stamps ⫶ TKTKXXX
you don't get a kid
My name is Linda. I write a bi-weekly newsletter about computer science, childhood, and culture.
Official summer in France lasts until September 23, so we have still a few more weeks of sunshine ahead of us. This micro-season, La Rentrée, is my favorite (like I write every, single, year), but it was only this year I realised the sense of return comes from the verbs. In French you partir en vacances, you leave for vacation, you don't take one. So the return gets its own word, too.
1.
A few Saturdays ago we built a language model you can touch. The part about generative AI that interests me the most is that language becomes math, similar words clump close to each other in a multi-dimensional word space (also called latent space). Which means we can observe language and move along its dimensions in completely new ways: measure the distance between two words, find what lives halfway between home and robot, push a shoe three steps toward cat and see what it becomes.
A real language model has over 16,000 dimensions and the tools for exploring it are made for researchers, not for seven-year-olds. Ours had only three, but it came with glitter and was ready in an afternoon.
In Annantalo we had three anchor words (home, robot, forest) and the children stamped words onto tags and argued about where they live. A lawnmower hung between home and robot. A helmet appeared hanging underneath a bicycle. An Estonian family added words in Estonian, and one child added TKTKXXX, language only computers understand.
This is the first part of exploring what language models might look like as materials, and what play in latent spaces could look like. (My next workshop will be on October 17th in Annantalo, but more on that later.)
2.
The very first version of the accompanying zine for the workshop can be downloaded here (PDF, 0.2Mb). It will go through a lot of changes, so treat it as an early prototype. (I just came across this early version of the My First Computer activity and was reminded how important it is to document works-in-progress and how much I learnt from the experiences of others). While I know the zine becomes tiny, I love how playing with paper already introduces children to dimensionality. Here’s a tutorial if you’ve never tried it.
I’m looking for more ways to make language models tangible for children, so if you have ideas, reply to this letter.
3.
Robin Sloan writes about digital pantographs. A pantograph was originally an analog tool for changing the scale of a drawing. In his parlance, an LLM is a digital pantograph for text, as it can scale it up (expand a one-sentence prompt into thousands of lines of code) or scale it down (categorize and summarize millions of messages), just a very unpredictable and uninspectable one.
When I was looking for materials for the workshop, I ran into these tiny rubber alphabet stamps and was delighted how much they’re already like tokens. Discrete little units of language you can pick up, arrange, and stamp into sequences.
4.
LittleLearner is a language model trained only on what elementary schoolers are taught. While I was playing with it, I kept thinking, is this.. the childhood of a language model? Meanwhile, Llama 3.1 language model was trained on 150,000 times more words than a child ever hears. I liked this quote:
“Claude has seen the amount of language that an entire city will experience in one generation,” Georgetown cognitive scientist Ethan Gotlieb Wilcox told the magazine. Toddlers, by contrast, produce grammatically correct sentences after hearing something like 10 to 30 million words.
“If you train GPT-2 on 30 million words, you get a nonsense generator; you don’t get a kid,” says Stanford’s Michael C. Frank.
5.
As someone coming from a computational background to biology, a big culture shock is realizing just how much improvisation happens in the lab.
Small observations like the viscosity of the liquid, firmness of the pellet, resistance while pipetting can all drive on-the-fly adjustments: spin a little longer, mix a few more times, or aspirate with a bit more force…
These details are critical to troubleshooting and data interpretation - and can make the difference between signal and noise. But they’re rarely recorded.
Without knowing the messy physical reality of experiment execution, the AI scientist, no matter how brilliant, flies blind. This observability gap decouples AI progress from scientific progress.
Some Scientists Have ‘Magic Hands’ in the Lab. This A.I. Is Learning Why by Carl Zimmer sits squarely on the theme I’m interested in, how sometimes things can be written down and sometimes they have to be felt with hands.
6.
Sherry Turkle has a new book coming out called Artificial Intimacy. Turkle has long been my favorite sociologist among technology people. In past years I’ve kept referencing her work on childhood animism and computers as transitional objects and of course it helps that she also lived in Paris once upon a time! Out September 29th.
7.
My radio musings from last edition clearly struck a nerve and I got so many cool recommendations for new things to listen to. Here is Radioooo, a music time machine where you pick a country and a decade and hear what was playing then. And Ldial, a curated dial of independent and community radio stations across the US, I’ll turn to this for my next week’s travels.