Phillip Isola, professor at MIT, joins us to talk about representation learning: what makes a representation good, why different models seem to converge on similar representations, and whether pre-training is really over.

We discuss the platonic representation hypothesis and its limits, why clustering structure matters more than global geometry, and Phillip's new neural thickets paper arguing that post-training is easier than people think because pre-trained weights already sit near solutions to downstream tasks. Phillip also explains why he thinks LLMs are already world models, why he's betting on RNNs making a comeback, and why his most exciting current direction is artificial life: putting LLM agents in open environments with no fixed task and studying them like new organisms.

Timeline:

00:00 Intro song
00:13 Intro
01:05 What is representation learning and why it matters
04:09 What makes a representation good: minimality and sufficiency
10:03 How cross entropy and contrastive learning shape representations
14:35 Dimensionality reduction and why dimension isn't the right complexity measure
16:35 Compression and geometric clustering during training
19:27 The platonic representation hypothesis and what actually converges
22:53 Local neighborhoods vs global structure: the Aristotelian follow-up
24:33 When convergence is strong: truth vs the space of possibility
28:09 Is there true similarity in the world? The Bouba-Kiki effect
30:56 World models vs autoregressive LLMs
32:14 Diffusion LLMs as a special case of autoregressive models
33:42 What architectures win in five years: the case for RNNs
36:11 Grad student descent, or do we actually have principles?
40:51 Feathers and wings: what to take from biology
43:17 How close are we to brain-like models? Marr's three levels
47:01 Are better models becoming less human-like?
49:38 Is pre-training all you need? The neural thickets paper
54:18 LoRA, low rank fine-tuning, and why post-training is easier than we thought
56:01 RL environments and what our benchmarks actually test
1:01:11 Artificial life: LLM agents as new organisms
1:07:20 What's overlooked in AI research right now
1:08:36 Why stay in academia, and doing science in the age of Opus

Music:

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Most AI-for-science companies are selling shovels. Qichao Hu wants the gold.

In this episode, we talk with Qichao, the founder and CEO of Molecular Universe, the AI-for-science platform that grew out of SES AI, a high-energy-density battery developer he's run for fourteen years. His core distinction is that companies from the AI world build tools, such as foundation models that predict properties, while companies from the science world care about the final product, such as the new battery or material that actually ships. Molecular Universe sits firmly on the science side, and the difference shows up everywhere from what they publish to what they refuse to.

We get into the actual workflow of materials discovery and where AI compresses it. A single trial in a traditional lab can take a year with maybe a 40% success rate; the goal is to run a thousand candidates in parallel and turn that year into a week. Qichao walks through improving low-temperature fast-charging for EV batteries: from hypothesis generation through molecule-, material-, and device-level property prediction, down to autonomous labs that synthesize and test the top candidates without a human touching a pipette.

The hardest problem, it turns out, isn't predicting molecular properties or measuring device performance, but it's the black box connecting the two. In batteries, that's the solid-electrolyte interface, which the field has been hand-waving about since the seventies. And the thing standing in the way of cracking it isn't a clever training trick but data: companies sitting on twenty years of records are finding it too messy, incomplete, and poorly labeled to train on, and are having to start collecting from scratch with new protocols and robots.

Timeline

• 00:13 — Intro and welcome;
• 01:19 — Shovel vs. gold
• 05:18 — Why the world's smartest scientist doesn't automatically give you a better battery
• 07:25 — The discovery workflow
• 09:37 — Exploration vs. exploitation
• 11:54 — Safety and filtering: screening novel molecules against banned and toxic-substance lists
• 17:55 — How hypotheses get generated, and where frontier LLMs help
• 20:29 — From hypothesis to ~400 formulations: property prediction, ranking, and handing off to autonomous labs
• 26:37 — "A foundation model for everything" — and the black box between molecular properties and device performance
• 30:01 — World models and physics
• 33:09 — The great unknown in batteries
• 37:08 — Simulation vs. reality: calibrating massive simulated datasets with a sliver of experimental data
• 41:47 — Lab robotics: how fast the hardware has caught up, and what a floor of autonomous labs looks like
• 43:50 — The real bottlenecks
• 50:21 — Pre-training from scratch vs. post-training LLMs, and why training tricks haven't reduced the need for good data
• 52:42 — Evaluation
• 55:42 — Publish the B+ model, keep the A model
• 58:05 — Five years out
• 1:00:37 — Closing thoughts and wrap

Music:

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

We talk a lot on this show about RL, agents, and the move between pre-training and post-training, but not enough about the layer everything actually runs on. Benny Chen, co-founder of Fireworks AI, one of the largest inference platforms around, walks us through what it takes to serve models at scale: sourcing GPUs, writing the kernels, the runtime, and the routing layer that lets a customer hit one endpoint and forget the rest.

We talk why the real bottleneck is power, not chips, and why that favors Nvidia and Google. Why MoE keeps winning even when dense models look better on paper and why he'd rather run fungible capacity at 95% than specialized chips at 60%. We also talk about quantization limits, where RL efficiency has to go next, and his case that AI is still under-hyped. We also get into cross-region training, sparse autoencoders and why interpretability hasn't taken off in open source, whether open models can close the gap, and a frank read on Anthropic's go-to-market.

Timeline

• 00:00 — Intro: the part of AI nobody talks about
• 01:20 — What "infrastructure for AI" actually means: the layers, from GPUs up to routing
• 02:59 — Why not just buy your own GPUs and do it yourself?
• 05:17 — The scale Fireworks runs at
• 06:35 — Hardware inflation, GPU costs, and the real risk hiding in commit duration
• 10:14 — Nvidia vs AMD vs TPUs, and why power is the bottleneck
• 11:57 — Mixing GPU types and generations; fungibility vs. specialization
• 14:22 — Once you have the GPUs, what's the next layer to build?
• 17:04 — Dense vs. MoE, and why the hardware picks the winner
• 21:07 — Quantization: is FP4 the floor? TurboQuant and INT vs. FP
• 24:28 — How tied are the algorithms to the hardware?
• 25:12 — DeepSeek, DeepGEMM, and next-token prediction as reconstruction loss
• 28:50 — Why RL is still wildly inefficient compared to pre-training
• 30:08 — Speculative decoding, AI-generated kernels, and auto-research
• 34:00 — The AGI question: why text gets automated but vision may stay expensive
• 37:07 — Hype check: why Benny thinks AI is still under-hyped
• 41:28 — Training vs. inference at the infrastructure level
• 44:12 — Scaling across data centers: cross-region training with Cursor
• 45:40 — Sparse autoencoders, interpretability, and why open source is human-constrained
• 49:04 — Will open models catch up — on quality and on compute?
• 51:41 — Are we plateauing? Opus 4.7 vs. 4.6 and the coming data wars
• 54:41 — Physical limits, HBM, and whether chips keep getting faster
• 58:17 — The belief about inference everyone gets wrong
• 59:31 — Anthropic, mythos, and a frank take on go-to-market
• 1:04:41 — Wrap-up

Music:

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.