The Human Oversight Paradox: Accountable for AI, Afraid to Admit What We Don't Know

The uncomfortable collision between regulation and reality

Two developments in October 2026 deserve to be considered together.

On 8 October, the UK's Information Commissioner's Office (ICO) launched a call for evidence on agentic AI, examining how existing data protection law should apply to systems capable of undertaking tasks, using tools and acting with increasing autonomy.

Its questions cover security, transparency, accountability, automated decision-making, fairness and lawful processing.

Meanwhile, research commissioned by Emergn, reported by ITPro, reveals something rather uncomfortable about the people expected to oversee this technology.

Of 350 senior UK business leaders surveyed, 23% admitted exaggerating their knowledge of AI. Some 38% feared their career prospects could suffer without substantial improvement in their AI capabilities, while only 43% reported receiving significant AI training during the previous year.

These are self-reported findings from one survey. They do not establish how competent those executives actually are, nor whether they make poor decisions.

But they expose a potential problem that deserves serious examination.

We are building governance systems that depend on meaningful human oversight while some of the humans expected to provide it feel pressure to appear more knowledgeable than they are.

That is the human oversight paradox.

A human in the loop is not necessarily a human in control

Much of the reassurance surrounding AI governance rests on a familiar proposition: a human remains responsible for consequential decisions.

It sounds reasonable. It is also incomplete.

The ICO's existing guidance on AI and data protection already distinguishes meaningful human review from superficial involvement. It considers the circumstances of a decision, the options available to reviewers and how AI outputs influence real-world outcomes.

This matters because a human can formally approve a recommendation without meaningfully examining it.

Consider a hypothetical executive reviewing an AI-generated investment recommendation.

The analysis is comprehensive. The financial modelling appears credible. The supporting narrative is persuasive. Several colleagues have already endorsed the proposal.

The executive understands the commercial objective but cannot independently assess some of the model's assumptions.

Do they challenge the recommendation?

Do they seek specialist advice?

Do they delay the decision?

Or do they approve it, reassured by the apparent sophistication of the analysis and the confidence of those around them?

The existence of an approval stage tells us very little about the quality of the judgement exercised within it.

Human involvement is a process characteristic. Meaningful human oversight is a capability exercised under particular conditions.

Those are not the same thing.

The psychological cost of saying "I don't know"

Why might an experienced executive hesitate to acknowledge uncertainty?

Part of the answer may lie in how organisations construct and reward leadership authority.

Senior leaders are expected to be decisive, knowledgeable and confident. Their judgement is scrutinised by colleagues, boards, shareholders and employees.

Yet the greater the expectation of certainty, the more professionally uncomfortable it can become to admit its absence.

Amy Edmondson's research on psychological safety offers an important perspective here. Psychological safety concerns whether people believe they can take interpersonal risks, including asking questions, acknowledging mistakes and expressing concerns, without being punished or humiliated.

Edmondson's foundational research established an association between team psychological safety and learning behaviour.

It does not prove that psychological safety explains the Emergn survey findings. Nevertheless, it provides a credible mechanism worth investigating.

An executive may possess sufficient intelligence, experience and professional integrity to recognise that something is uncertain.

But recognition is only the beginning.

The organisational environment influences whether that uncertainty is expressed, examined or concealed.

There is also a distinction between confidence and competence.

A confident executive may lack the knowledge required to evaluate a particular AI recommendation.

A less confident executive may exercise excellent judgement by recognising that limitation and seeking independent expertise.

Which behaviour does the organisation reward?

And which does it mistake for leadership?

The danger is not that leaders sometimes lack knowledge. It is that organisational expectations may discourage them from acknowledging when knowledge is insufficient.

Three conditions for meaningful human oversight

This collision between regulatory expectations and leadership behaviour suggests that meaningful oversight depends on at least three connected dimensions.

1. Capability: Can the individual exercise independent judgement?

Effective oversight requires more than familiarity with AI terminology.

The responsible person must understand enough about the decision, its evidence, limitations and potential consequences to recognise when further scrutiny is warranted.

That does not mean every CEO or board member must become an AI engineer.

It means they must possess sufficient decision competence to distinguish what they can reasonably assess from what requires specialist validation.

This includes the ability to question assumptions, recognise uncertainty, examine conflicting evidence and identify when a recommendation exceeds the limits of available knowledge.

2. Conditions: Does the environment permit genuine challenge?

Capability becomes less useful when exercising it carries unacceptable personal or professional consequences.

An organisation may formally encourage dissent while informally rewarding agreement.

It may invite challenge while treating delays as obstruction.

It may tell executives to seek advice while interpreting requests for clarification as weakness.

Meaningful oversight therefore depends partly on the conditions surrounding the decision.

Can people acknowledge uncertainty?

Can they question an apparently authoritative AI output?

Can they challenge a recommendation supported by senior colleagues?

Can they interrupt a process without being punished for doing so?

A policy granting authority to challenge is not evidence that challenge is psychologically or operationally safe.

3. Conduct: Does meaningful challenge actually happen?

Even when individuals possess the necessary capability and work within supportive conditions, oversight must still be exercised.

This is where governance moves from intention to observable behaviour.

Did the executive examine the assumptions?

Were alternatives considered?

Was contradictory evidence sought?

Was the AI recommendation independently validated where necessary?

Was dissent recorded and considered?

Did new evidence lead to an appropriate revision?

These questions are important because a well-documented process can still contain poor judgement.

And a good outcome does not necessarily demonstrate that the underlying decision was sound.

Capability, conditions and conduct are different dimensions of oversight. A credible governance approach needs to examine all three.

The deeper problem: AI may be changing the judgement it depends upon

There is another dimension to this problem.

AI doesn't merely introduce new information into existing decision processes. It can change the cognitive work people undertake.

When AI summarises evidence, evaluates options, drafts recommendations and proposes conclusions, it may release time for higher-value thinking.

But it may also reduce opportunities to practise the reasoning through which professional judgement develops.

Research on automation bias and cognitive offloading has long raised concerns about how reliance on technological assistance can influence attention, verification and independent reasoning.

The implications are not uniformly negative. AI can expose people to alternative perspectives, improve access to evidence and help identify weaknesses in an argument.

The distinction lies partly in how the technology is used.

Does it replace the exercise of judgement, or provide additional material against which judgement is exercised?

This creates a further challenge for organisations.

It may not be sufficient to assess whether people possess the judgement required for oversight today.

They may also need to consider whether their increasingly automated working environment continues to develop and sustain that capability.

Otherwise, an organisation could strengthen its formal AI controls while gradually weakening the human capability on which those controls depend.

Why this belongs in the boardroom

The implications extend beyond technical governance.

Boards and executive teams are responsible for consequential decisions involving investment, risk, strategy, people, reputation and organisational direction.

AI is increasingly participating in the information and analysis informing those decisions.

The governance question is therefore not simply whether an organisation has an AI policy, risk register or accountable executive.

It is whether those arrangements support meaningful scrutiny when the evidence is incomplete, the recommendation is persuasive and the consequences matter.

A board might reasonably ask:

• Where does AI influence consequential decisions, and who is accountable?

• What must the responsible individual understand to exercise meaningful oversight?

• How do we know they can challenge, validate or override a recommendation?

• What happens when somebody admits they do not understand?

• What evidence demonstrates that challenge occurs in practice?

• How are failures, near misses and revised judgements used to improve the system?

These questions move governance beyond allocating responsibility towards examining whether responsibility can actually be discharged.

They also reveal why judgement cannot be reduced to an individual's intelligence, experience or confidence.

Judgement is exercised within an environment. That environment can support independent thinking or quietly suppress it.

The case for independent executive challenge

There is a reason experienced leaders sometimes benefit from a conversation outside their immediate organisational hierarchy.

Within the organisation, uncertainty can carry consequences.

An executive's admission of doubt may affect how colleagues interpret their authority. Challenging an established position may create friction. Reconsidering a previously endorsed decision may be mistaken for indecision.

Independent executive coaching can create a confidential environment in which those concerns can be examined without the same immediate organisational pressures.

But confidentiality alone is not enough.

A useful independent coach should not simply provide reassurance or reinforce the executive's preferred interpretation.

They should help the leader examine assumptions, consider disconfirming evidence, distinguish confidence from certainty and recognise when additional expertise is necessary.

This connects with the broader psychological insight commonly described as Solomon's paradox: people may sometimes reason more wisely about another person's problems than their own.

Research by Grossmann and Kross (2014) explored this difference in reasoning about personal and others' conflicts, including how psychological distance can influence perspective.

It would be an overstatement to claim that the research proves executive coaching improves AI governance.

The more defensible proposition is that independent perspective and structured challenge may help leaders examine decisions that are difficult to assess from within their own position.

The quality of that challenge matters.

So does the willingness of the executive to act upon it.

From AI governance to organisational judgement

This is where the discussion becomes particularly relevant to the work we are developing through Agile Mindset Mastery and The Boardroom Coach.

Our interest is not simply in whether organisations have appropriate decision processes.

It is in whether the people, behaviours and conditions within those processes support sound judgement.

Through Diagnostic Discovery, we examine how decisions actually happen, rather than relying exclusively on how an organisation says they happen.

Through Decision Integrity, we introduce structured examination of assumptions, evidence, bias, alternatives and dissent.

Through Leadership Actualisation and independent executive coaching, we work with leaders on how they exercise judgement under pressure, uncertainty and competing expectations.

And through our developing Organisational Judgement and Judgement Simulator propositions, we are exploring how these capabilities might be examined and developed more systematically.

These are not substitutes for technical AI assurance, regulatory compliance or specialist risk management.

They address a complementary question.

When governance depends on human judgement, how do we know that judgement is sufficiently capable, independent and supported to perform its intended role?

That question deserves more attention than another declaration that humans must remain in control.

The question regulators and boards cannot afford to overlook

The ICO's current call for evidence is an important opportunity to examine the accountability and data protection implications of increasingly autonomous AI.

But the wider governance challenge extends beyond the design of the technology.

We must also consider the humans expected to supervise it.

The Emergn findings do not establish that nearly a quarter of UK executives are incapable of governing AI.

They establish something narrower but still important: a material minority of surveyed senior leaders acknowledge overstating their knowledge.

When considered alongside regulatory expectations for meaningful oversight, that finding creates a legitimate question about the relationship between competence, confidence, organisational culture and accountability.

It is a question worth investigating empirically rather than answering through assumption.

Because meaningful human oversight cannot be established simply by placing a person at the end of an automated process.

That person must possess the relevant capability, operate within conditions that permit genuine challenge and actually exercise independent judgement when required.

Before asking who is accountable for AI, perhaps we should establish whether they have the capability, conditions and confidence to exercise that accountability meaningfully.

And perhaps the most revealing test of leadership in an AI-enabled organisation will not be how confidently someone approves a recommendation.

It will be whether they can recognise when they should challenge it, seek further evidence or say:

"I don't know enough to approve this yet."

 

 

 

 

The Boardroom Coach, through Agile Mindset Mastery, provides independent executive challenge, Leadership Actualisation, Decision Integrity and organisational Discovery for leaders navigating consequential decisions, uncertainty and change.

 

 

 

 

References and further reading

• Information Commissioner's Office (2026). Agentic AI: Call for Evidence. Open 8 October to 20 November 2026.

• Information Commissioner's Office (2026). ICO secures changes from leading AI developers as scrutiny extends to AI agents.

• ITPro (2026). Business leaders are bluffing over their AI skills. Reporting Emergn survey findings.

• Edmondson, A. (1999). Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly, 44(2), 350–383.

• Grossmann, I. and Kross, E. (2014). Exploring Solomon's Paradox: Self-Distancing Eliminates the Self-Other Asymmetry in Wise Reasoning About Close Relationships in Younger and Older Adults. Psychological Science, 25(8), 1571–1580.