Why AI may be building a knowledge debt alongside the technical one
By Louize Clark
Today the National Cyber Security Centre published its annual State of the Cyber Nation report. The headline is the kind that travels: the NCSC judges it highly likely that AI-powered malware will be used in a ransomware attack on the UK's critical national infrastructure by 2028.
The line that will probably stay with people, though, came from the NCSC's chief executive, Richard Horne, in his speech launching the report. The UK, he said, has "got away with doing tech on the cheap" for decades. His point wasn't that AI has found some brand-new way in. It's that AI will find the weaknesses we've lived with for years, and find them faster, and at a scale, we've never really had to face.
I spent today at a cybersecurity showcase at the University of Bedfordshire in Luton, running the same workshop twice, to between sixty and seventy people across two sessions. By the end of the day, I couldn't separate his phrase from what I'd watched happen in those rooms.
The exercise is about as simple as it gets. Everybody puts a hand up. I ask questions about their own organisation, nothing technical, nobody needs to know how AI is built, and they keep their hand raised for as long as they can honestly answer yes. If they're unsure, the hand comes down.
The first question was easy. Do you use AI in your business, or know that your employees do? Every hand stayed up.
Then I asked whether they knew if employees or contractors were using their own personal AI accounts for work, and something shifted. Hands didn't fall so much as drift. You could see people thinking it through, arms half-lowered, wondering whether they really knew. You could see it in their eyes. When I asked whether they knew which AI tools could reach company email, shared drives or customer records, the drift carried on, and the unease in the room grew with it.
By the fifth question, do you know who approves new AI connections, permissions and capabilities when they're introduced, most of the hands had gone. A handful of people held on. Then I asked whether they'd be confident of noticing when a software update materially changes what its AI can do, and those last hands came down too, firmly this time.
Nobody, in either session, kept their hand up to the end.
The second round turned from technology to people. Most hands went up again when I asked whether employees use AI to produce information, recommendations or content that shapes their work. Then I asked whether they're trained to question those recommendations rather than simply trust them. A lot of hands came down, and there was a nervousness about it, as though people were seeing something about their own organisation for the first time.
Fewer still stayed up for whether people have enough information and time to properly review what the AI gives them. By the time I asked whether employees know when and how to challenge, override or escalate an AI-assisted decision, there were perhaps one or two hands left. The final question was whether their targets and management practices genuinely support employees who question AI. Not a single hand.
A show of hands isn't a survey, and some of those hands came down simply because the person would never be told those things in their role. That tells its own story, though. Some of those answers may sit elsewhere in their organisations, spread across people who rarely compare notes. Others may never have been worked out at all. Either way, nobody in the room was sure where to find them.
What struck me most was that nobody in those rooms had done anything wrong. These were people who had given up part of a working day to come to a cybersecurity event because they care about getting this right. If anything, they were the ones paying attention.
Richard Horne's phrase was about a whole country's habits over decades, not any one organisation, and I think there's a parallel to it that belongs at that same level. Alongside the technology, we have been quietly building up something I'd call a knowledge debt: the growing gap between the capabilities an organisation brings in and the understanding it develops to use, oversee and challenge them.
When a new AI capability arrives, whether someone chose it, an update delivered it or a member of staff brought it in, the technology has a price attached. Someone sees the invoice, or at least the licence. The knowledge needed to use it well arrives with no price on it at all. It doesn't appear in a budget, it doesn't have an owner, and nobody is asked to sign it off. So it slips through, one sensible decision at a time, much as technical debt does. Nobody intends it. It's simply what happens to the things nobody was ever asked to pay for.
Watching the hands come down, I think you could see that debt in three different forms.
Some of it is knowledge that simply isn't there yet: what's running, what it can reach, how to tell when an output is wrong. The early questions in both rounds were picking that up.
Some of it is harder to see, because it's knowledge people don't know they need. Technical debt usually has someone who knows about it, the engineer who once said they'd fix it later. Knowledge debt often has nobody, because you can't learn about a capability you never noticed arriving. That's why the question about software updates took down the last of the hands. If nobody would notice the change, nobody would know there was anything new to understand.
And some of it is about not knowing where the knowledge lives. That was question five, the one that cleared most of the room. I hadn't asked whether people understood AI connections. I'd asked whether they knew who approves them. Legal can draft a policy and IT can secure the systems, but in most organisations nobody is quite responsible for making sure the people using the technology know what they need to know.
Part of the reason this has crept up on us is that AI itself isn't new. Fraud detection, spam filters, the recommendations on a shopping site: these have been running for years. What's changed is who holds it.
Using AI used to mean writing code, or at least sitting next to someone who could. That was a barrier, and although nobody designed it this way, it also carried a requirement to learn. You couldn't use the capability without picking up something about how it worked, what it touched and where it might go wrong.
Now you can talk to it in English, or whatever language you think in. That's an extraordinary opening up, and I wouldn't want to reverse it. But when the barrier left, the need to understand stayed behind, unseen. A busy person can connect an AI assistant to their inbox, and then to a shared drive, to save an hour a week without ever having to know what a permission or an access token is. Nothing in the experience tells them they've just made a cybersecurity decision. And if nobody else knows the connection exists, nobody can weigh it up either.
The same gap shows up higher in the organisation, just in a different form. One of the things we talked about in the room was how rarely the people who approve a technology are the people who live with it. A system is signed off at board level, and then it lands on a middle manager. Often they were hired to do something else entirely. Then operations change, AI arrives, and suddenly they're responsible for what it produces. They're the one redesigning the workflow, handling their team's frustrations when it doesn't behave, and making decisions with a technology they had no part in choosing, sometimes about things nobody ever gave them the knowledge to judge. The approval happened in one room. The understanding was needed in another.
And somebody has to be responsible. When I attended a call-for-evidence session on autonomous weapons at the House of Lords, a senior figure in defence AI made the point that you can't court-martial a bot. It has stayed with me, because it's just as true a long way from defence. You can't send an AI agent to prison. You can't fine a computer. When a decision shaped by AI goes wrong, the accountability lands on a person, and very often it's that middle manager, deciding about something they didn't choose, at the end of a long day.
That's why I talk about decision fatigue so often, and not only where AI is involved. It applies to anyone in a leadership or business role who carries decisions all day. When the question about time came up, I raised it in the room, and I watched people nod. Think about a day of back-to-back meetings where you've made difficult calls and made them well. You get home, someone asks what you'd like for dinner, and you have nothing left. "I don't know." You do care about dinner. You've simply run out of deciding.
Most of us recognise that evening. Now picture someone who works alongside AI all day, drafting emails with it, reading its analysis, approving what it suggests. A hundred small decisions, each one resting on its output. Reviewing what AI produces takes attention, and attention runs out. Somewhere in that day, checking quietly turns into assuming.
Researchers have started to put a name to the AI version of this. A study by Boston Consulting Group researchers, published in Harvard Business Review earlier this year, called it "AI brain fry": mental fatigue from excessive use or oversight of AI tools beyond one's cognitive capacity. Around one in seven of the nearly 1,500 US workers surveyed recognised it in themselves. Those who did reported a third more decision fatigue and made more major errors. And it was lower where managers gave people training and support.
Knowledge only helps when there's still the time and energy to use it, and even then, having it isn't the same as being able to act on it. Someone may know exactly how a process is supposed to work without having done it themselves for three years. A manager may be quite sure a recommendation needs questioning and have neither the time nor the authority to challenge it. We tend to talk about knowledge as something you either have or haven't got. But knowing something, being able to do it and being allowed to act on it are three different things, and an organisation can quietly run short of any one of them.
Which is why the last question, the one nobody could answer, matters most to me. Whether people feel able to question AI depends on whether their organisation makes room for it: time to look properly, a clear route to raise a concern, and the reassurance that saying "I'm not sure this is right" won't count against them. An individual can't supply that for themselves. An organisation has to decide to provide it.
The NCSC is right that years of underinvestment in technology are about to be tested, and right that we have a choice in how we respond. We're quite good at counting what we've spent on technology. We have asset registers, budgets and patch schedules. We have almost nothing that tells us how much we owe in knowledge, or where.
Technical debt can leave weaknesses that attackers exploit. Knowledge debt tends to stay hidden until something goes wrong, when someone asks who approved a connection, who understood the risk, or who was meant to step in, and the room goes quiet.
Somewhere between sixty and seventy hands went up in Luton today, and every one of them came down. I didn't find that alarming. It felt like honesty, from people who had turned up because they wanted to get this right.
But it has left me wondering how many organisations are investing in making their technology more capable without making the same investment in the people expected to understand it, question it and control it.
And if that gap keeps growing, what exactly will we still be capable of choosing?
AI Policies UK helps organisations see the AI infrastructure they're already standing on, chosen, inherited and embedded, and whether the people around it have what they need to judge it. Get in touch: louize@aipolicies.uk
Sources: NCSC: Cyber chief: UK facing AI-powered attacks on critical infrastructure in two years