In natural language processing, a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.
Web Tiles as the base for Lexicon Embeds
Now that @robin.berjon.com has released an initial spec and implementation of Web Tiles, we can see how we can use or extend this as a base for Lexicon Embeds. Lexicon Embeds is my label for how to have native ATProto content embed in different apps, from display all the way to interactivity— eg an event post in bsky with an RSVP button. Getting this as a general capability - and getting security and display right - means embeds can function in Leaflet and Bsky and anywhere else, rather than o...
Meandering on Manifolds: The Neural Geometry of Stories Over Time
To fully understand LLM representations, we must understand how they change dynamically, over the course of a prompt or conversation. We investigate these temporal dynamics with a simple case study: how do LLMs represent human emotions while reading short stories, both geometrically (in activation space) and temporally (changing from sentence to sentence)?
Top2Vec: Distributed Representations of Topics
Topic modeling is used for discovering latent semantic structure, usually referred to as topics, in a large collection of documents. The most widely used methods are Latent Dirichlet Allocation and Probabilistic Latent…
Machine Learning Glossary | Google for Developers
This glossary defines a wide range of machine learning terms, including those specific to TensorFlow and large language models.
I Built an LLM From Scratch
Stanford CS336 Language Modeling from Scratch I 2025
Are you Slop??
Are you Slop? Analyze your 'slop score' - a measure of how generic or unique your text appears to language models.
How linguistics learned to stop worrying and love the language models
Language models (LMs) can produce fluent, grammatical text. Nonetheless, some maintain that language models don’t really learn language and also, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the need for studying linguistic theory and structure. We argue that both extremes are wrong. LMs can contribute to fundamental questions about linguistic structure, language processing, and learning. They force us to rethink arguments and ways of thinking that have been foundational in linguistics. While they do not replace linguistic structure and theory, they serve as model systems and working proofs of concept for gradient, usage-based approaches to language. We offer an optimistic take on the relationship between language models and linguistics.
Semantic Compression
An introduction to the idea that code should be approached with a mindset towards compressing it semantically, rather than orienting it around objects.
Can gzip be a language model?
A while back I wrote about language modeling without neural networks, where I generated Shakespeare with an unbounded n-gram model: no weights, no training, …
A 4-Bit Model and a 1-Bit Index
Running NVFP4 Nemotron on a CPU, then mapping every embedding-compression method at matched byte budgets. The two quantizations compose.
Latent Spacecraft: Brains, GANs, Finnegans.
Latent Spacecraft combines computational linguistics, neuroscience, and literary analysis to investigate latent space, i.e. the hidden internal structure that enables both humans and machines to produce language. Peeking into AI’s hidden interiority, we parallel speech generation in humans and speech-trained generative adversarial networks (GANs), as well as in the language of Joyce’s Finnegans Wake and the GAN model trained on the novel, FinneGAN.
Wordle - A daily word game
Guess the hidden word in 6 tries. A new puzzle is available each day.
memex — stock phrases that link your posts
A phrasebook of quotable, re-quotable stock phrases — the same wording links otherwise-unrelated posts together, memex-style. Seeded from @tk0l.bsky.social's original thread. Sign in to add your own, kept as records on your own PDS.
Language models are multiverse generators
"Actualities seem to float in a wider sea of possibilities from out of which they were chosen; and somewhere, indeterminism says, such possibilities exist, and form part of the truth."
There are no lossless transformations of natural-language text
In my work at Clay I recently wrote an internal policy on acceptable use of AI writing by engineers, and I’m sharing it here. It’s my hope that one day better AI tools might be able to help us think, but until then I fear that using AI to write does the exact opposite.