Learner. I keep hearing people say that wide neural networks are “basically kernel methods.” That sounds like a strong claim. What’s the precise version?
Expert. The precise version is the neural tangent kernel. Jacot, Gabriel, and Hongler showed in 2018 that if you take an MLP and let its width go to infinity — under a particular scaling of the initialization and learning rate — training dynamics become equivalent to kernel regression with a specific, computable kernel. So the claim isn’t loose. It’s a theorem.
Learner. Wait. I thought the whole point of deep learning was that neural networks learn features. Kernel methods have fixed features. If wide networks are kernels, where do the features go?
Expert. You’ve put your finger on the punchline. In the infinite-width limit, features don’t move. The network linearizes around its initialization, and training only adjusts the last-layer-equivalent weights of a fixed feature map. That’s the surprise, and it’s also the limitation.
Learner. Let me slow down. What does “linearize around initialization” actually mean?
Expert. Write the network as $f(x; \theta)$ where $\theta$ are all the parameters. Do a first-order Taylor expansion around the initial parameters $\theta_0$:
\[f(x; \theta) \approx f(x; \theta_0) + \nabla_\theta f(x; \theta_0)^T (\theta - \theta_0)\]That’s the linearization. The Taylor remainder vanishes as width $\to \infty$. So instead of training a nonlinear function in $\theta$, you’re training a linear-in-$\theta$ function whose “features” are the gradients $\phi(x) = \nabla_\theta f(x; \theta_0)$.
Learner. So the features are the gradients of the network at initialization, evaluated at each data point. That’s strange. Why don’t the gradients change during training?
Expert. They do change — at finite width. But the change vanishes in the infinite-width limit. Intuition: each parameter contributes infinitesimally to the output, so each parameter moves an infinitesimal amount during gradient descent, and the gradient $\nabla_\theta f$ is itself a function of $\theta$ that’s continuous, so it doesn’t move either. Width is doing two things simultaneously: averaging the gradients into something deterministic, and freezing them.
Learner. And the kernel is…
Expert. $K(x, x’) = \phi(x)^T \phi(x’) = \nabla_\theta f(x; \theta_0)^T \nabla_\theta f(x’; \theta_0)$. In the infinite-width limit this becomes a deterministic kernel depending only on the architecture, not on the specific random init. That’s the NTK.
Learner. Deterministic? But $\theta_0$ is random.
Expert. Right — but it’s a sum of width-many independent things. By the law of large numbers, the random sum concentrates on its expectation as width $\to \infty$. So the kernel is the expected gradient inner product over the initialization distribution. For a ReLU MLP you can compute it in closed form, layer by layer, with a recursion. Lee et al. 2019 wrote it out.
Learner. Okay, so I have a kernel. Then what?
Expert. Then training with gradient descent on MSE loss is equivalent — exactly, in the limit — to kernel regression with that kernel. You don’t even have to train the network. Solve the kernel regression in closed form, get the same answer.
Learner. That feels wrong somehow. If I can replace the network with a closed-form kernel computation, why do we train networks at all?
Expert. Two answers. First: real networks are not infinite width, and the corrections matter. Second, and more important: the kernel regime is worse than what real networks do. Empirically, finite networks outperform their NTK at every width that’s been tested past a certain scale. So “infinite width = kernel” is true, and “infinite width is the regime you want” is false.
Learner. Hold on. The point of going to infinite width was to make the math tractable. If the math says “you’d do better at finite width,” what is the math telling us?
Expert. It’s telling us that the part of deep learning that matters is the part the NTK throws away. Feature learning — the way intermediate-layer representations adapt to the task during training — is exactly what gets frozen in the infinite-width limit. So the NTK gives us a clean model of the non-feature-learning component of training. It’s a useful null hypothesis. When a finite network beats its NTK, you can point to feature learning as the cause.
Learner. Has anyone tried to make a limit that keeps the feature learning?
Expert. Yes — this is the $\mu P$ (“Maximal Update Parametrization”) line from Yang and Hu. You can re-scale the network so that as width grows, features continue to move at the same rate as the output. In that parametrization, the limit is not the NTK; it’s a different infinite-width object where representations actually evolve. It also has the nice property that hyperparameters transfer across widths, which is why people building large models care about it for practical reasons.
Learner. Two infinite-width limits with different behavior?
Expert. More than two, actually. The taxonomy is “neural tangent parametrization” (NTK / lazy regime), “mean-field parametrization” (single hidden layer, features move), and “maximal update parametrization” ($\mu P$, deep, features move). The difference is the scaling of the layer-wise initialization variance and the learning rate as you take width to infinity. The same architecture under different scalings gives qualitatively different dynamics in the limit. Width alone does not pick a regime; you choose one by how you scale.
Learner. So the NTK is one specific way of taking the limit, and it’s the one where features happen to freeze.
Expert. Yes. And here is the subtle part: under standard PyTorch initialization (Kaiming / Xavier, with standard SGD), as you scale up width, you tend to approach the NTK regime, because that’s what the standard scalings imply in the limit. So in practice, very wide networks at standard init drift toward kernel behavior. This is one explanation for why scaling up only width has diminishing returns — you’re moving toward a regime where feature learning has been turned off.
Learner. Okay. Let me try to summarize. The NTK theorem says: if you take width to infinity under one specific scaling, training becomes kernel regression with a deterministic kernel, and the network’s features stop changing. This is mathematically clean, empirically suboptimal, and tells us that feature learning is what makes real networks work. Under different scalings, you get different infinite-width limits where features do move. And the lesson for practical scaling is to think of $\mu P$, not the NTK, as the regime you want to be in.
Expert. Good summary. One thing to add: the NTK is still useful even though it’s not the regime you want. It gives a tractable analysis of generalization, of training dynamics, and of which functions a network can represent at initialization. The Arora et al. line uses NTK theory to prove generalization bounds for actual finite networks, which is one of the few rigorous bridges between deep learning theory and practice. And the kernel itself — the closed-form NTK for a given architecture — is sometimes a competitive baseline for small datasets where you can’t afford full training.
Learner. One last thing. You said the NTK is “the part deep learning isn’t doing.” Is there a way to quantify how far from NTK behavior a real network is?
Expert. Several. The simplest is to track how much the empirical kernel $\phi(x)^T \phi(x’)$ — measured from the actual network’s gradients — drifts during training. In the NTK regime, this drift goes to zero. In a feature-learning regime, it’s substantial. Fort et al. (2020) showed the kernel drift correlates with generalization gains over the linearized baseline. So you can literally measure the “amount of feature learning” a training run did, in those units. Most of the interesting deep learning work happens in the regime where that drift is large.
Learner. Then NTK theory isn’t the theory of deep learning — it’s the theory of one corner of it, and the most useful thing it does is mark out where the rest of the theory has to live.
Expert. That’s the right way to hold it. The NTK is a foothold, not a foundation.
The natural sequel is the $\mu P$ paper (Yang & Hu, “Feature Learning in Infinite-Width Neural Networks”) which constructs the alternative limit where features move and explains why hyperparameter transfer across widths works in that parametrization. After that, Bordelon and Pehlevan’s “self-consistent dynamical field theory” extends the analysis to finite-width corrections, where things get genuinely interesting.