The arithmetic of attention: why FlashAttention still matters
Memory bandwidth, not FLOPs, is what bounds modern inference. A walk through the numbers behind a kernel that quietly reshaped the field.
Machine learning,
from to clusters.
An open community for learning, writing, and tinkering on the infrastructure behind modern AI — inference engines, training systems, ml stacks, and everything in between.
16
articles
12
contributors
FIG. 1.1Per-head attention pattern at layer 14. Causal mask + induction circuit + sink token, visualized over one sentence.
Featured
Editors' picks.
The arithmetic of attention: why FlashAttention still matters
Memory bandwidth, not FLOPs, is what bounds modern inference. A walk through the numbers behind a kernel that quietly reshaped the field.
Liam Chen Nov 12, 2026 · 14 min
Continuous batching, revisited
Three years after the original paper, what does state-of-the-art serving actually look like? A field report from a team running 12B tokens a day.
Priya Raghavan Nov 06, 2026 · 19 min
What we've been getting wrong about MoE routing
Top-k routing has become a default. It shouldn't be. A look at the tradeoffs nobody's measuring and the experiments that change my mind.
Hugo Belmar Oct 28, 2026 · 22 min
Latest
Recently published.
Quantization-aware training, end-to-end
FP4 is here, and the gap between PTQ and QAT has widened. What's actually working in production today, and why the recipe is messier than it looks.
Ana Voinescu Oct 19, 2026 · 17 min
A research-grade trainer in 400 lines
Most training frameworks are 50,000 lines of code in a trench coat. This is what falls out when you start from FSDP and a will to delete.
Toma Iliescu Oct 09, 2026 · 25 min
Embeddings as compression: the bitter lesson, retold
We've stopped treating embeddings like first-class data. A case for revisiting them, with measurements from a 200M-document corpus.
Naoko Ide Sep 22, 2026 · 12 min
World models and the cost of imagination
Latent rollouts are cheap. World rollouts are not. What we learned trying to scale a JEPA-style world model on robotics data.
Felix Marin Sep 14, 2026 · 21 min
The hidden latency in agent loops
Every tool call is a round trip. Every round trip is a context append. Why naive agent loops compound latency faster than you expect, and what to do about it.
Felix Marin Sep 12, 2026 · 14 min
RAG is not retrieval-augmented generation
It is a feedback loop, an index policy, and a re-ranker pretending to be a system. Why most RAG postmortems mistake the symptom for the disease.
Sasha Petrov Aug 28, 2026 · 18 min
Index
Browse by topic.
-
-
Continuous batching, revisited
Three years after the original paper, what does state-of-the-art serving actually look like? A field report from a team running 12B tokens a day.
-
Speculative decoding without the speculation
Draft models work. They also fail in ways the original papers didn't surface. A small bag of tricks for keeping acceptance rates high in real workloads.
-
Notes on KV cache paging at scale
PagedAttention is a good idea poorly understood. A primer, plus the second-order effects you only see at 10,000 concurrent requests.
-
What we've been getting wrong about MoE routing
Top-k routing has become a default. It shouldn't be. A look at the tradeoffs nobody's measuring and the experiments that change my mind.
-
Testing Final Blog
Blog testing Write portal!
-
Test First Blog Post
Testing blog post from website!
-
Neural Networks From Zero: From a Single Number to a Billion Parameters
A neural network never sees a word, an image, or a sound — only a list of numbers. Starting from that one fact and a single neuron, this day-zero guide builds the whole machine: how any input becomes numbers, why weights, biases, and activations each exist, how neurons stack into layers and layers into a model, and how to compute a model's size and running cost by hand.
-
Embeddings as compression: the bitter lesson, retold
We've stopped treating embeddings like first-class data. A case for revisiting them, with measurements from a 200M-document corpus.
-
RAG is not retrieval-augmented generation
It is a feedback loop, an index policy, and a re-ranker pretending to be a system. Why most RAG postmortems mistake the symptom for the disease.
Tools
Run the math yourself.
Attention Visualizer
Inspect attention patterns layer-by-layer for any Hugging Face model. Click any head to see its causal mask, induction behavior, and sink tokens.
LiveThroughput Calculator
Estimate tokens/sec for any model, precision, batch size, and GPU combination — memory-bound roofline only, no kernel-quality wishful thinking.
LiveTraining Memory Calculator
Per-GPU VRAM breakdown for training and inference — params, gradients, optimizer state, activations, KV cache, with ZeRO/FSDP sharding.
LiveEval Harness Playground
Run a small set of evaluations against any inference endpoint and get back a structured scorecard — quality, latency, cost, and refusal rate side by side.
BetaModel Card Generator
Generate a structured model card from a checkpoint and an evaluation log — covers intended use, training data summary, evals, limitations, and ethics.
BetaKernel Benchmark
Side-by-side timings for Triton, CUDA, and PyTorch implementations of the same kernel — attention, layernorm, GEMM, custom — across shapes and dtypes.
SoonDiscourse
The conversation.
GitHub Discussions
Long-form threads tied to GitHub identities. Best for technical questions, paper discussions, and feature proposals. Searchable, attributed, permanent.
Open Discussions →Discord
Real-time chat for the working day. Quick questions, debugging help, paper club, and the occasional argument about whether MoE is overrated.
Join the server →