Rich Sutton begins a recent conversation about continual learning with a wonderfully blunt observation: “I’m not weird. The field is weird.”
His point is that we should not need the adjective. Biological intelligence learns continually. We act, observe what happens, adjust, and carry some part of that experience into the next decision. Learning is not a phase that ends before life begins.
Most large language models work differently. They undergo an enormous training exercise, followed by post-training and testing, and are then deployed with their weights effectively frozen. They can use the context in a conversation. They can retrieve information. They can appear to adapt beautifully. But when the conversation ends, the underlying model has not permanently learned from the experience.
This leads me to a fairly strong conclusion. Current LLMs are powerful tools, but they are not genuinely intelligent because they cannot permanently learn from experience.
They may know more facts than any of us. They may reason through problems that would take a human team days. Yet when they make a mistake on Monday, they do not necessarily arrive on Tuesday having absorbed the lesson. Someone still has to capture the failure, interpret it, and change the system around them.
Focus On: Borrowed Memory Is Not Learning
Retrieval-augmented generation, or RAG, has become one of the standard answers to the frozen-model problem. Connect an LLM to current documents, customer records, product information, or a corporate knowledge base, and it can ground its response in material that was never included in its original training.
This is immensely useful. It is also easy to anthropomorphise.
RAG retrieves information when the model needs it. A weight change becomes part of the reasoning process for every subsequent request and can influence how future learning happens. RAG changes what the model can see at a particular moment. Learning changes the machinery through which it interprets what it sees.
The distinction matters because a model can repeatedly retrieve a documented correction without internalising the broader lesson behind it. It has access to the experience, rather as I have access to a book in a library. Access does not mean the book has changed how I think. For that, I must read it, connect it to what I already know, and retain something.
In The Deployment Layer Is The Moat, I argued that companies need to own the feedback loops, evaluations, workflow memory, and human corrections that make generic models useful. Sutton and Khurram Javed’s argument raises the next question: what would it take for the model itself to participate in that learning loop?
Oak Lab’s wager
Sutton and Javed have founded Oak Lab to pursue agents that learn in real time from their own experience. Their starting point is the “big world hypothesis”: for many meaningful problems, the environment will always be vastly larger than the agent trying to understand it. No training corpus, however large, can contain every situation the agent will encounter.
If that is right, static knowledge eventually meets a moving world. Intelligence must be able to revise its approximations while it operates.
There is serious research behind part of this wager. A 2024 Nature paper involving Sutton showed that standard deep-learning systems progressively lose plasticity when learning continues across changing tasks. Units become dormant, representations lose diversity, and the network becomes worse at absorbing something new. Continual backpropagation countered this by selectively reinitialising underused units, continually injecting a small amount of fresh variation.
This is encouraging, but it is not the whole solution. The researchers explicitly distinguish loss of plasticity from catastrophic forgetting. Their algorithm helps a network remain capable of learning; in its current form, it does not ensure that new learning will preserve everything valuable from the past. Oak’s ambition is therefore exactly that: an ambition, not a trillion-parameter replacement for today’s LLMs waiting discreetly in a cupboard.
Frozen can be a feature
There is a good reason companies do not let production models rewrite themselves after every customer interaction.
A system that learns from experience can learn misinformation, manipulation, temporary anomalies, and bias. Research has already demonstrated data-poisoning attacks against online learning. A model willing to learn from everyone will eventually encounter a customer, competitor, or bored teenager eager to contribute to its education.
Frozen weights provide stability. External memory can be inspected, sourced, corrected, and removed. A changed model is harder to audit because the new knowledge is distributed through its parameters and may affect apparently unrelated behaviour.
The same nuance applies to synthetic data. Sutton calls it a mistake when it becomes a substitute for experience in the real world. He has a point: research shows that indiscriminate recursive training on model-generated material can cause model collapse. But synthetic data and simulation remain useful in bounded domains. The mistake is believing that a world we generated is large enough to replace the world we inhabit.
Build the learning estate now
Weight-level continual learning is not commercially ready. Waiting passively for it would still be a mistake.
Most organisations are already producing the experience future systems would need, then casually throwing much of it away. Prompts live in personal folders. Agent skills are rebuilt by different teams. Human corrections disappear into chat histories. Business outcomes are stored separately from the AI actions that influenced them. The organisation learns, but its systems do not retain a coherent record of the lesson.
CMOs should create a central, governed learning estate that preserves:
the prompts, skills, workflows, and evaluation rules teams have developed;
the model, data, permissions, and context in which they were used;
the actions produced, including exceptions and escalations;
the human corrections and business outcomes that followed.
This should not become a landfill for every AI interaction. Provenance, access controls, retention rules, and outcome quality matter. A future learning system trained on an immaculate archive of bad decisions will merely make the mistakes more efficiently.
Different parts of the estate should also learn at different speeds. Context can change immediately. External memory can change quickly when its sources are traceable. Workflows and evaluation rules should change through controlled review. Model weights should change slowly, with regression testing, versioning, and rollback.
Two measures become useful: how long it takes for an exception to become a governed improvement, and what regressed when the improvement was introduced.
Continual learning may eventually alter the architecture of AI. It should already alter the architecture of the organisation using it.
The companies best prepared for weight-level learning will not be those with the largest prompt libraries. They will be those that preserved the relationship between instruction, action, correction, and consequence. Everyone else may discover that they carefully archived what they told the machine and discarded what experience was trying to teach it.
Follow me
That’s all for this week. To keep up with the latest in generative AI and its relevance to your digital transformation programs, follow me on LinkedIn or subscribe to this newsletter.
Disclaimer: The views and opinions expressed in Chronicles of Change and on my social media accounts are my own and do not necessarily reflect the official policy or position of S&P Global.