AI in talent work
·
7 min read
Key findings
Recruiters given higher-quality AI support performed worse, checked less and did not improve over time.
On the one task outside AI capability, consultants using AI fell from 84 per cent accuracy to 60 to 70 per cent.
AI belongs on drafting and sorting work that a practitioner checks, and away from ranking, rating and inferring potential.
A function should name the practitioner on every output and keep a register of decisions that used machine input.
Recruiters given higher-quality AI support spent less time on each application and followed its recommendations without challenge.
AI in talent management belongs on drafting, summarising and sorting work that a practitioner checks, and away from decisions about individual people.
The machine drafts and a named practitioner decides.
Experiments with consultants and recruiters show that stronger AI support makes people check less, so the line has to be written down and audited.
Talent functions are being asked what they are doing about AI, and an answer that lists tools leaves the main question open.
A more useful answer states which tasks the machine may touch, which decisions it may not, and who is accountable for each.
Where AI Earns Its Place
Five uses are defensible in an internal talent function today.
First drafts of documents that a practitioner will rewrite
Summaries of long source material, read against the source
Interview questions generated from a role brief
Free-text comments coded into themes for a person to check
Alternative framings of a problem before the practitioner commits to one
The five share one condition.
Someone who could have produced the output without the machine checks it before it goes anywhere.
A Test For Any Task
Four questions sort a task before anyone opens a tool.
Does the output describe, rate or rank a named person? If so, the practitioner writes it
Could the practitioner produce it without the machine? If not, nobody can check it
Can the prompt be written with roles and descriptors in place of names? If not, the task needs personal data that stays in approved systems
Who will sign the output? If nobody would, nobody should send it
Most tasks that fail the first question appear below under Where AI Does Not Belong.
The other three catch the tasks that look routine and carry the same risk.
The Frontier Is Jagged
Mollick and colleagues from Harvard, Warwick and MIT ran an experiment with Boston Consulting Group and nearly eight hundred consultants.
Participants were randomised into a group that worked the standard way and a group that used GPT-4, and they were given eighteen tasks designed to resemble a consultant's job.
The tasks included proposing ideas for a new shoe, segmenting the footwear market, drafting marketing copy and writing a persuasive memo to employees.
The group using the tool did significantly better, and the result held across 118 different analyses.
Their work was faster, and it was judged more creative, better written and more analytical.
Most participants simply pasted in the question they were given and used the answer that came back.
Mollick calls the limit of what the machine can do the Jagged Frontier, because the boundary is invisible.
Tasks that look equally hard to a person sit on opposite sides of it.
The study included one task chosen to fall outside the frontier, combining a tricky statistical issue with misleading data.
Consultants working alone got it right 84 per cent of the time.
Consultants using the tool got it right 60 to 70 per cent of the time.
The Recruiter Study
Dell'Acqua ran the study closest to talent work.
He hired 181 professional recruiters and asked them to evaluate 44 job applications for mathematical ability.
The scores came from an international test of adult skills, so they were not obvious from the CVs.
Recruiters received good AI support, poor AI support or none.
Recruiters with the higher-quality support performed worse than those with lower-quality support.
They spent less time and effort on each application and followed the recommendations without challenge.
They also did not improve over time.
Recruiters with weaker support stayed alert, critical and independent, and their use of the tool improved.
Dell'Acqua modelled the trade-off and concluded that when the AI is very good, people have no reason to work hard and pay attention.
He called the result falling asleep at the wheel.
Centaur And Cyborg
Mollick describes two ways of combining person and machine.
Centaur work keeps a clear line between them, with tasks allocated according to the strengths of each.
Cyborg work blends them, moving back and forth inside a single task.
For talent work the centaur pattern is the safer default, because the line between person and machine is also the line of accountability.
The practitioner decides the approach and the machine produces the draft, and each can be identified in the record.
Machine output meets the same evidence standard as any other source, which means it is attributed and checked.
Where AI Does Not Belong
Four uses fail the test that the output must be checked by someone able to produce it, and each has its own reason.
Ranking people. The recruiter evidence shows that strong assistance degrades the judgement it is meant to support
Inferring potential from performance data. A model trained on past decisions reproduces them, and potential is defined differently by different executives
Writing the evidence in a talent review. The record is valuable because a named person committed to a claim
Drafting the words for a hard conversation. A manager reciting unfamiliar phrases loses the listener, whoever wrote them
The second reason has a documented case.
Amazon built a tool to sort résumés, trained on ten years of its own hiring decisions, and within a year it was favouring men.
The company cancelled the programme.
A tool meant to remove human bias reproduced the pattern in the data it learned from.
The same risk applies to any model asked to spot potential, as the trouble with high-potential lists explains.
AI In Reviews And Ratings
A manager may use AI to organise their own dated notes into a first draft of a performance review.
The rating and the example behind it remain the manager's, and the draft is read against the notes before anyone sees it.
That split keeps the evidence rule of a talent review that changes decisions, under which no rating goes on the grid without an example behind it.
It also keeps the manager able to defend the rating when a rated person or an executive challenges it, as performance ratings under pressure describes.
AI cannot supply an example the manager never observed.
Accountability For Machine Output
Kakade and Vispute list accountability among the legal and ethical issues of AI in HR, alongside bias, transparency and privacy.
They call for clear guidelines and mechanisms for assigning responsibility when an AI-related decision goes wrong.
The accountable people are the practitioner named on the output and the executive who signed the resulting decision.
The tool and its vendor are not accountable, because neither can be questioned by the person affected.
A Position A Function Can State
Three sentences cover most of what an executive team needs to hear.
The machine drafts and the practitioner decides, and the practitioner is named on the output.
Prompts name roles and never individuals, and no personal data leaves the systems already approved for it.
Machine output meets the same evidence standard as any other source, which means it is attributed and checked.
Prompts a practitioner can defend sets out the working rules for the second sentence.
The position is short enough for a director to remember and specific enough to be audited.
Protecting Judgement
The recruiter finding points to specific countermeasures.
Three of them add little time to the work.
Require the practitioner to form a view before seeing the machine's, on any decision about a person
Sample a share of outputs for independent review, and report the disagreement rate
Rotate people through work done without assistance, so that the underlying skill stays in use
The second produces the evidence that a board will accept.
A disagreement rate of zero across a sample means either that the output is excellent or that the reviewer has stopped checking.
The Entry Level Problem
Mollick identifies a consequence that sits closer to the talent function than to technology.
People have traditionally become experts by starting at the bottom, as an apprentice, an intern or a resident, and learning from experienced people while making mistakes.
A senior person under time pressure will do the junior task with AI and skip the training.
Mollick warns of a training gap, and he points out that experts become the only people able to check the work of capable AI.
The pipeline that produces those experts is the one at risk.
Two responses belong in a workforce plan and not in an AI strategy.
Name what a person in their first two years is now expected to learn, and where they will learn it
Give juniors the checking work, and supervise their reasoning and not only their output
Workforce planning after the headcount freeze covers the decisions about which work the remaining people do.
What To Measure
Most AI reporting counts licences and logins, which says nothing about value or risk.
Three measures say more.
Time saved on a named task, measured once
Disagreement rate between the machine and the reviewer
Decisions taken with machine input, by type
The third is the register that matters if a decision is later challenged.
A function that can say which decisions used machine input, and who signed each one, is in a different position from one that cannot.
Claims To Avoid
Two claims damage a function when examined.
The first is that AI removes bias.
Sumser, an analyst who writes on HR technology, observes that data records what happens in a biased culture.
In his view, trying to bury the pattern is the most likely way to amplify it.
The Amazon résumé tool, described above, is the documented case.
The second is that productivity gains are proven inside the organisation.
The consultants in the BCG experiment did tasks designed to resemble their own work, with footwear as the subject.
Those results do not carry over automatically to the tasks of a talent function.
Run one measured comparison on one real task in the function before claiming a gain.
Where The Tools Sit
Trusted Talent Advisor ships with The Prompt Library, which holds a prompt for each of the 51 tools and five more for the diagnostic lenses.
The library opens with a position on where AI helps and where it does not, and every prompt describes people by role and not by name.
Related reading: AI in instructional design, AI agents in the team, AI skills for HR professionals.
Sources
1. Mollick, E. Co-Intelligence: Living and Working with AI. Portfolio.
2. Kakade, K. S. and Vispute, S. A. AI Dynamics in HRM. Apple Academic Press and CRC Press.
3. Meacham, M. AI in Talent Development: Capitalize on the AI Revolution to Transform the Way You Work, Learn and Live. ATD Press.
From the system
Trusted Talent Advisor sets out the full method: seven components, 51 tools, The Talent Template Pack and The Prompt Library.
See the system
Questions this answers
How is AI used in talent management?
It drafts documents a practitioner will rewrite, summarises long source material, generates interview questions from a role brief, codes free-text comments into themes and offers alternative framings of a problem. A person who could have done the task checks each output.
What are the risks of AI in HR?
Bias reproduced from past decisions, people checking less as the tool improves, personal data leaving approved systems, and unclear accountability. Kakade and Vispute list bias, transparency, privacy and accountability among the legal and ethical issues.
Should AI be used in performance reviews?
It can organise a manager's own dated notes into a first draft. The rating and the example behind it stay with the manager, who reads the draft against the notes and can defend the result if it is challenged.
Can AI predict high potential?
It should not be used for that purpose. A model trained on past decisions reproduces them, as Amazon's résumé tool did with ten years of hiring data, and executives define potential differently from one another.
Who is accountable when AI makes an HR decision?
The practitioner named on the output and the executive who signed the decision. The tool and its vendor cannot be questioned by the person affected, so accountability cannot rest with them.
How do you keep a human in the loop on AI decisions?
Require a view before seeing the machine's, sample outputs for independent review, rotate people through unassisted work, and report the disagreement rate. The recruiter study shows why: strong AI support reduces the effort people put in.
Keep reading
Measuring HR impact
Internal Mobility Metrics Without A Platform
Seven measures an HR adviser can build from existing records, each with a stated definition, a base and a way to read it.
7 min read
Workforce and future skills
Job Rotation As A Talent Gap Strategy
A job rotation programme closes a talent gap when the role, the person, the cover and the evidence are agreed with both managers before it starts.
8 min read
Talent reviews and succession
Seeing Talent Below The Top Team
Visibility of leadership potential drops below the top team. A cross-unit forum, run on one page per unit and a written evidence standard, is the method for seeing it.
8 min read