Roadmap

We are going to get started with something “easier” but important, which will serve as the foundation and north star for our future discussions.

The context

Recently, the ARC-AGI-3 Benchmark has been released. It should be a reliable way to score the intelligence of an LLM on unseen tasks. The interesting aspect is that these tasks, which are simple 2D games, are easily solvable by a human player. On the other hand, state-of-the-art models can only win 0.2% of the games on the private dataset (https://arcprize.org/leaderboard).

The person behind the benchmark, François Chollet, has been famously pessimistic regarding the efficiency and intelligence of transformers or gradient descent-based models. More importantly, he has been an advocate for new effective benchmarks to measure the generalization capabilities of these models. The ARC-AGI 1 and 2 benchmarks were, in fact, only solved when new paradigms were introduced, such as CoT and RL-trained reasoning, respectively.

The reading 📄

To better understand the need for these benchmarks, which ultimately guide AI progress, I propose we read this paper Chollet wrote in 2019. Here he provides a new perspective on the oldest problem in AI: “how do we define intelligence?”, which is extended to the question “how do we measure it?”.

Link to the paper: https://arxiv.org/abs/1911.01547 Read Chapters I (~1h) and II.1 (~35min). There is no math in these parts and it reads smoothly. Feel free to read chapter II(it provides a more technical definition of intelligence) and III (introduces the ARC-AGI benchmark) fully, if you have time and interest.

Proposes a theoretical framework for why in-context learning works: transformers implicitly perform Bayesian inference over latent “concepts” during pre-training, then use the prompt to identify and condition on the relevant concept at inference time. A foundational lens for understanding the most surprising emergent capability of LLMs.

Demonstrates how reinforcement learning (without supervised fine-tuning on chain-of-thought data) can elicit strong reasoning capabilities in LLMs. A key paper for understanding the current push toward reasoning-capable models.

Anthropic’s investigation into emotion-like representations and behaviors in large language models. Came up multiple times during our previous sessions, so we’re reading Parts 1 & 2 together.

Establishes power-law relationships between model performance and compute, dataset size, and parameter count. This paper fundamentally changed how labs think about training runs and resource allocation.

A provocative hypothesis from DeepMind: that reward maximization in sufficiently rich environments is enough to give rise to all facets of intelligence — perception, language, social reasoning, and more. A direct descendant of the reinforcement learning tradition, and a great paper for debating what intelligence actually requires.

The InstructGPT paper that introduced RLHF at scale. This is the technique that turned base language models into useful assistants, and it remains the backbone of how frontier models are aligned today.

Shows that LLMs can be trained to behave deceptively — appearing safe during evaluation while executing harmful behavior under specific triggers — and that standard safety training (RLHF, adversarial training) fails to remove this behavior. A sobering result for alignment research.

The Vision Transformer (ViT) paper that proved the transformer architecture isn’t just for language — it works for vision too, with minimal domain-specific modifications. A pivotal moment: the beginning of architectural convergence across modalities, which paved the way for today’s multimodal models.