Skip to content
← All notes

Why being good at using AI is not the same as being good at your job

Fluency with the tools and competence at the work now look identical on the page, and organisations are quietly promoting the first while believing they are rewarding the second.

Skills7 min read

There is a person on most teams now who produces more than anyone else. Their documents are structured, their analysis is thorough, their turnaround is fast, and their work reads extremely well.

Sometimes that person is excellent. Sometimes that person is excellent at prompting. From the outside, over a quarter, these are very hard to tell apart, and the tell only arrives when something goes wrong.

Output stopped being evidence of competence

For most of working history, a good artefact was weak but real evidence that somebody understood the subject. Not proof, but correlated, because producing it was hard without understanding.

That correlation broke. Fluency is free now and understanding is not, and a reader cannot separate them from the artefact alone. Every organisation that evaluates people on what their work looks like is now measuring something different from what it thinks.

The difference shows up under pressure, not under review

Tool fluency and job competence diverge in specific, predictable moments, and they are all moments where the prepared answer runs out.

  • When the question is under-specified. A competent person notices the ambiguity and raises it. A fluent one gets a confident answer to whichever interpretation the model chose, and does not know a choice was made.
  • When the output is subtly wrong. Catching it requires knowing the domain. No amount of prompting skill substitutes.
  • When somebody senior pushes back. Explaining why a conclusion holds is different from having produced it, and this is where the gap becomes visible in about ninety seconds.
  • When the situation is genuinely novel. Models are strong on what resembles their training and weakest exactly where a person is most needed.
The moments where tool fluency and real competence come apart: an under-specified question, an output that is subtly wrong, a senior person pushing back, and a situation nothing in the training resembles.
They look identical until the prepared answer runs out.

How to tell the difference, if you manage people

The trick is to stop assessing artefacts, which have stopped carrying the information, and start assessing reasoning, which has not.

  • Ask what they considered and rejected. Understanding shows in the discarded options, and generated work rarely has any.
  • Ask what would change their conclusion. A person who understands can name the specific fact that would flip it.
  • Ask about the weakest part of their own argument. Competence answers this readily. Fluency treats it as an attack.
  • Check predictions after the fact. Whether the thing they said would happen actually happened is the only measure that AI cannot produce on their behalf.
Assessing the reasoning rather than the artefact: what was considered and rejected, what would change the conclusion, where the argument is weakest. The discarded options are where understanding shows.
The artefact stopped carrying the information. The reasoning still does.

This is not an argument for using the tools less

Somebody who understands their field and uses AI well is straightforwardly better than the same person without it. The tools are a genuine multiplier on real capability.

The failure is treating the multiplier as the capability. A large number multiplied by zero is still zero, and it takes a surprisingly long time for anyone to notice, because the output looks the same until the moment it matters.

Common questions

Is prompt engineering a real skill?
It is a real and shrinking one. Getting good output from a model genuinely helps, and every model release makes it matter slightly less by design. What holds its value is knowing what to ask and being able to tell whether the answer is right, and both of those come from the domain rather than from the tool.
How can managers assess capability when AI writes the work?
Assess the reasoning rather than the artefact. Ask what was considered and rejected, what would change the conclusion, and where the argument is weakest. Then check predictions against outcomes over time. None of those can be generated on somebody's behalf in a live conversation.
Can you be good at your job and bad at using AI?
Yes, and it is a shrinking advantage. Deep expertise still produces better decisions than shallow expertise with good tooling. But refusing the tools means being slower at the production half for no gain in the judgement half, and that difference compounds.

PDP Quest exists because of the problem underneath all of these: when output stops indicating capability, you need another way to know who can actually do the work.

See how verification works →