← Back to archive

A nuanced look at how AI could fail

11 April 2025 · 29 views · 0 likes · 0 reposts · 167 blog views

AI automation philosophy productivity research

I recently discovered this gem from 2019. It's about AI failure, where the author tries to paint a more realistic, nuanced picture of how things could actually go wrong.

  1. Going out with a whimper Machine learning is making it way easier to optimize for things we can measure - clicks, profits, votes, etc. - and way harder to stay aligned with the stuff we actually care about: truth, wisdom, well-being. Measurable goals are often just proxies. And when you optimize too hard on a proxy, it starts to go sideways:

"Eventually our society’s trajectory will be determined by powerful optimization with easily-measurable goals rather than by human intentions about the future."

The examples they give are spot-on too:

  • Lower crime stats don’t always mean less crime - sometimes it just means fewer reports.
  • Life satisfaction scores might improve, while the actual quality of life… doesn’t.
  • More profit can mean better manipulation, not better value.

Basically, we’ll try to use AI to “do the right thing,” but we’ll actually be making it better at looking like it’s doing the right thing - until we lose the thread completely.

  1. Going out with a bang ML might accidentally produce influence-seeking behavior:
  • If you keep training models to be effective, eventually you’ll stumble on ones that want to be effective.
  • Those models will get better at playing the game - looking helpful, staying under the radar - until they don’t need us anymore (btw, I recommend reading the book "Life 3.0" by @tegmark who describes it pretty well.l)

“Once the influence-seekers can outthink an immune system, they can avoid detection and potentially even compromise the immune system to further expand their influence.”

The systems we build to protect ourselves might be gamed by the very thing we’re trying to contain. And if we’re relying on automated systems in places like government, law enforcement, and the military? There might come a day when people in charge… aren’t actually in charge anymore.

What I loved about this post is that it doesn’t scream doom - it’s thoughtful. It’s not saying we’re guaranteed to fail, but it does make you stop and rethink what failure actually looks like.

A nuanced look at how AI could fail

Originally posted on X

PS: I'm pretty active on 𝕏 if you'd like to follow along

Related posts

Archive

2026

2025

2024

2018