for AI systems to continue to improve in knowledge work, they need either a reliable mechanism for autonomous self-improvement or human evaluators who can capture errors and generate high-quality feedback. Industry has invested heavily in the former. He almost never thinks about what happened with the second year.
I would argue that we need to address the problem of human assessment with as much seriousness and investment as we put into building model capabilities. Fresh graduates have been hired at major tech companies It has decreased twice since 2019. Document review, first-pass research, data cleaning, code review: Models now handle these. Economists who follow this call it displacement. Companies that do this call it efficiency. Neither focuses on the future problem.
Why there are limits to self-improvement in knowledge work
An obvious impetus is reinforcement learning (RL). AlphaZero Go learned chess and Shogi at superhuman levels without human input, creating new strategies in the process. 37 moves in a match against Lee Sedol in 2016 that professionals said they would never play did not come from human annotation. It emerged from the AI playing itself.
It is the stability of the environment that makes this possible. Move 37 is a new move in Go’s steady state space. The Rules are complete, unambiguous and permanent. More importantly, the reward signal is perfect: Win or lose, and instantly, with no room for interpretation. The system always knows whether a move is good or not because the game eventually ends with a clear outcome.
Knowledge work has none of these characteristics. The rules in any professional field are dynamic and are constantly being rewritten by the people who operate in them. New laws are passed. New financial instruments are invented. A legal strategy that worked in 2022 may fail in a jurisdiction that has since changed its interpretation. Whether the medical diagnosis was correct or not has been unknown for years. Without a stable environment and an unambiguous reward signal, you cannot close the loop. You need people in the evaluation chain to keep training the model.
Formation problem
The artificial intelligence systems being built today are trained on the basis of the experience of people who have gone through this formation. The difference now is that entry-level jobs that develop this kind of expertise are automated first. This means that the next generation is not accumulating potential experts a kind of judgement this makes the human evaluator worth having in the loop.
History has examples of knowledge dying. Roman concrete. Gothic construction techniques. Mathematical traditions that took centuries to restore. But in each historical case the cause was external: plague, conquest, the collapse of the institutions that housed knowledge. What is different here is that no external force is required. Fields can atrophy not from disaster, but from thousands of individual rational economic decisions, each individually sensitive. This is a new mechanism and we don’t have much experience to recognize it when it happens.
When all the fields are quiet
Logically, this is not just a pipeline problem. This is a collapse of demand for the expertise itself.
Review advanced math. It’s not atrophy because we stop producing mathematicians. As organizations no longer need mathematicians for their day-to-day operations, the economic incentive to become mathematicians disappears, the number of people capable of performing frontier mathematical reasoning declines, and the field’s ability to generate new ideas quietly atrophies. The same logic applies to coding. Our question is not “Will AI write code” but “If AI writes all the production code, who develops the deep architectural intuition that really drives the new system design?”
There is a critical difference between an automated domain and an intelligible domain. We can automate a lot of structural engineering today, but the abstract knowledge of why certain approaches work lives for years in the heads of the people who got it wrong in the first place. If you eliminate practice, you won’t just lose practitioners. You lose the ability to know what you’re missing.
Advanced mathematics, theoretical computer science, deep legal reasoning, complex systems architecture: When the last person with a deep understanding of a subfield of algebra retires and no one replaces them as funding dries up and career paths disappear, this knowledge is unlikely to be rediscovered any time soon.
he went And no one notices because the models trained on their jobs continue to perform well in benchmarks for another decade. I think of it as a loophole: the surface ability remains (models can still produce expert-looking results), while the underlying human ability to validate, extend, or correct that experience is quietly disappearing.
Why rubrics do not completely replace?
The current approach is rubric-based assessment. Constitutive AI, reinforcement learning from AI feedback (RLAIF), and structured criteria that allow models to evaluate models are rigorous techniques that significantly reduce reliance on human evaluators. I’m not firing them.
Theirs restriction is: A rubric can only capture what the person writing it knows to measure. Optimize hard against this and you get a model that is very good at satisfying the rubric. This is not the same thing as a model that is actually true.
Rubrics measure the overt, expressible portion of judgments. The deeper part, the instinct, the feeling that something is broken does not fit the rubric. You can’t write it because you have to experience it before you know what to write.
What does this mean in practice?
This is not an argument for slowing down development. The skill gains are real. And maybe researchers will find ways to close the evaluation loop without human judgment. Maybe synthetic data pipelines will be good enough. Perhaps the models are developing reliable self-correction mechanisms that we cannot yet imagine.
But today we don’t have them. And meanwhile, we dismantle the human infrastructure that currently fills the gap, not as a deliberate decision, but as a byproduct of a thousand rational decisions. A responsible version of this transition is not to assume that the problem will resolve itself. It’s about treating the assessment gap as an open research problem with the same urgency we bring to capacity gains.
What AI needs most from humans is what we care least about protecting. Regardless of whether it is permanently or temporarily true, the cost of ignoring it is the same.
Ahmed Al-Dahle is the CTO of Airbnb.
