MoE vs Dense vs Hybrid LLM Architectures

XiongjieDai/GPU-Benchmarks-on-LLM-Inference

Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?

Build agents, not pipelines

There are only two ways to use LLMs in a computer program: as part of a pipeline, or as an agent. In other words, either you express the control flow of the program in code, or you give a LLM tools and allow it to manage the control flow itself1.

Fast and Hard Code

People pick languages differently with LLMs.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

Artificial

An LLM is a computer program. We should talk about it like a computer program.

What the AI Wizard taught us about LLM code generation at scale - PostHog

You know what’s boring? Writing integrations. It’s a task you’ll do once, but you always need to do it right, which means carefully reading docs…

Scaling Laws Across Model Architectures: A Comparative Analysis of...

The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and...

Xuhui Zhou on Twitter / X

Creating user simulators is a key to evaluating and training models for user-facing agentic applications. But are stronger LLMs better user simulators?TL;DR: not really.We ran the largest sim2real study for AI agents to date: 31 LLM simulators vs. 451 real humans across 165… pic.twitter.com/SkswSzeBrz— Xuhui Zhou (@nlpxuhui) March 19, 2026

MemGPT

Memory-GPT (MemGPT) - Towards LLMs as Operating Systems - Teach LLMs to manage their own memory for unbounded context!

Modern GPU Programming For MLSys — Modern GPU Programming For MLSys

Machine learning systems sit at the heart of modern AI workloads. In these systems, performance often comes down to the quality of a small number of GPU kernels. Attention kernels, LLM prefill and decode kernels, low-precision block-scaled GEMMs, fused MoE layers, and other large fused kernels all directly shape end-to-end speed in both training and serving.

Extensible Software in the age of LLMs | Jeremy Morrell

Solid core + capability-based sandboxes + LLMs = Users with superpowers

Fight fire with pedantry

Over the last two years, I’ve heard a bunch of speculation about how to design a programming language specifically for an LLM. It usually centers around typechecking, formal verifiability, or context engineering. It’s a fun thought experiment, but always felt aimless. First, my experience has been that designing for humans is the best way to design for an LLM, as long as you restrict yourself to text. Second, being in-distribution is so critical that there’s no point unless you’re also training a frontier coding model.

Compression is prediction | ngrok blog

Compression and LLMs are trying to solve the exact same problem: predicting what comes next. Learn the fundamentals of compression and how better prediction leads to better shrinkage.

The Future of Software Engineering Isn’t What You Think

Over the past two decades, I’ve watched software eat the world, then hardware catch up, then Generative AI reshape both at the same time…

Import AI 446: Nuclear LLMs; China's big AI benchmark; measurement and AI policy

Will AIs be jealous of one another?