← Back to archive

Why deep learning is less mysterious than it seems

24 March 2025 · 45 views · 0 likes · 0 reposts · 108 blog views

AI learning neuroscience research

I recently read "Deep Learning is Not So Mysterious or Different" by Andrew Gordon Wilson. It challenges a lot of common beliefs about deep learning.

1️⃣ Deep Learning follows existing rules Despite claims that deep learning defies conventional wisdom, he argues that its behaviors (overparameterization, double descent, and benign overfitting) are actually well-explained by existing frameworks.

2️⃣ Soft inductive biases Instead of restricting models to avoid overfitting, we should embrace flexible hypothesis spaces with soft biases that guide them toward simpler solutions naturally.

3️⃣ Overparameterization helps More parameters don’t necessarily mean overfitting. Larger models have a greater capacity to find good generalizable solutions, making them more robust.

4️⃣ Double Descent is not unique to Neural Networks Even simple linear models exhibit double descent, meaning this phenomenon isn’t exclusive to deep learning - it just wasn’t widely recognized before.

5️⃣ Why unique

  • Representation Learning. Neural networks learn their own features instead of relying on predefined ones.
  • Mode Connectivity. Different optima are often connected, helping models generalize.
  • Universality. Deep models work across different tasks, thanks to their ability to learn flexible representations.

Deep learning isn’t magic - it just builds on long-standing ideas in machine learning. Understanding why it works will help us make better models instead of getting caught up in supposed mysteries.

Why deep learning is less mysterious than it seems

Originally posted on X

PS: I'm pretty active on 𝕏 if you'd like to follow along

Related posts

Archive

2026

2025

2024

2018