Assessment and performance
·
9 min read
Key findings
Sackett, Zhang, Berry and Lievens revised the 1998 validity estimates in 2022, and structured interviews rose to first place at .42.
Cognitive ability tests fell from .51 to .31 and work sample tests from .54 to .33, the two largest falls among the top five.
Two unstructured interviewers rating one candidate overlap by about 11 per cent, against about 50 per cent for well-structured interviews.
Validity varies widely between settings, so an employer should compare any method with its own appointments.
Cook adds cost, practicality, generality, acceptability and legality to validity, and finds that selectors give cost too much weight.
Structured interviews now have the highest mean validity at .42, ahead of job knowledge tests at .40 and biodata at .38.
The best predictors of job performance are structured interviews, job knowledge tests, empirically keyed biodata and work sample tests, with cognitive ability tests and integrity tests tied for fifth. Sackett, Zhang, Berry and Lievens published this ranking in 2022, and it lowered the long-cited 1998 Schmidt and Hunter estimates for most of the methods that ranked high.
An adviser meets the ranking in two situations.
One is a hiring manager who wants a further stage after two failed appointments.
The other is a promotion that rests on tenure and the sponsor's impression.
What Validity Measures
Validity is the correlation between a method's result at selection and performance in the job later.
Sackett and colleagues estimate operational validity.
That is the correlation in a pool of applicants, measured against performance free of measurement error.
Two corrections move an observed correlation towards that estimate.
One adjusts for error in the performance measure, and the other for range restriction, which arises when the people studied were already selected.
The 1998 estimates used large range restriction corrections.
Sackett and colleagues found that many of those corrections applied one value to every study or relied on assumed values.
They corrected only where the data supported a correction, so many revised estimates adjust for measurement error alone.
The Revised Ranking
Each figure below is a mean operational validity against job performance, with the 1998 Schmidt and Hunter estimate beside it.
Structured interviews, .42 against .51
Job knowledge tests, .40 against .48
Empirically keyed biodata, .38 against .35
Work sample tests, .33 against .54
Cognitive ability tests, .31 against .51
Integrity tests, .31 against .41
Assessment centres, .29 against .37
Unstructured interviews, .19 against .38
Conscientiousness, overall, .19 against .31
Years of job experience, .07 against .18
Reference checks, years of education, peer ratings, job tryouts and graphology rest on 1998 figures that could not be re-evaluated.
Sackett and colleagues therefore leave them outside the ranking.
What Changed From 1998
Three methods lost the most validity.
Work sample tests fell by .21, cognitive ability tests by .20 and unstructured interviews by .19.
The 1998 work sample estimate of .54, the highest single-method figure in that summary, came from a study that predated meta-analysis.
The cognitive ability estimate fell mainly because Sackett and colleagues revised how far the underlying studies were affected by range restriction.
The revised .31 pools US Employment Service test battery data with a European and a UK meta-analysis.
It covers jobs of all complexity levels.
Structured interviews fell by .09, from a tie for second place to first.
All five of the 1998 top five remain in the new top five, and empirically keyed biodata has joined them.
The mean validity of the top five fell from .49 to .37.
The falls within it range from .08 for job knowledge tests to .21 for work samples.
The four strongest predictors, structured interviews, job knowledge tests, biodata and work samples, are all job-specific measures.
Two predictors moved for a different reason.
Interests rose from .10 to .24 because the new estimate uses the fit between a person's interests and the job.
Experience fell from .18 to .07 because the new estimate uses experience at the point of hire.
What The Revision Leaves Open
A mean validity conceals variation between settings.
Structured interviews have the highest mean at .42.
The value above which 90 per cent of operational validities are expected to fall is .18.
An employer therefore cannot count on the mean applying in its own organisation.
Ranked by that lower value, the order becomes empirically keyed biodata, contextualised conscientiousness, job knowledge tests, work samples and structured interviews.
Two practical limits apply as well.
In their traditional in-person form, structured interviews are generally not viable for high-volume jobs, where a quicker method screens candidates first.
Work samples and job knowledge tests presuppose that candidates arrive with training or experience, which ability tests do not.
Group Differences And Validity
Sackett and colleagues pair each validity with the difference in mean scores between Black and White candidates.
The standardised difference is .79 for cognitive ability tests, .67 for work samples and .54 for job knowledge tests, which draws on incumbent data.
It is .23 for structured interviews and .10 for integrity tests.
Cognitive ability tests, biodata, integrity tests and personality-based emotional intelligence now have similar validity.
Cognitive ability tests carry far more potential for adverse impact.
Sackett and colleagues do not call for setting aside a predictor because of group differences.
They note that combining predictors reduces those differences.
The choice of predictors follows the needs of the organisation.
Interviews And Agreement
Furnham describes the interview as one of the most commonly used and least reliable selection techniques.
Reliability here means whether two interviewers reach the same view of the same candidate.
For the most well-structured interviews, ratings from two interviewers overlap by about 50 per cent.
For unstructured interviews the overlap is about 11 per cent.
That is too low for the sole source of information on a pivotal hire.
Furnham lists the sources of error, which include the following.
Interviewers prefer candidates who share their interests
Interviewees manage the impression they give
Interviewers form a first impression and read later information through it
Scales get used differently when criteria are not written down
Furnham's tips for improving interviews include a job analysis, identical questions for every candidate and anchored rating scales.
He also advises against unrelated questions, such as asking which type of fish a candidate would be.
Scoring independently before any discussion lets the record show whether assessors agreed before they compared views.
Cook's Six Criteria
Cook sets validity beside five other criteria for judging a selection method.
They are cost, practicality, generality, acceptability to candidates and legality.
Cook finds that selectors give cost far too much weight.
A valid method is almost always worth using, however elaborate it is.
Furnham surveyed over 400 HR professionals, who rated references just slightly better than personal hunch on validity.
Cost and practicality explain why references persist despite that rating.
Cook's ratings predate the 2022 revision, and they place the 2022 top five high on validity and the ordinary interview low.
They add what the 2022 estimates do not cover.
Ability tests are cheap and suit every job, but fall foul of the law in the United States
Structured interviews are expensive to set up and transfer poorly between roles
Work samples are easy to use and legally safe, but expensive because each is specific to the job
Job knowledge tests are cheap because they are commercially available, but more likely to meet legal challenge
Assessment centres are legally fairly safe, but difficult to install and expensive
Personality, References And Experience
Overall conscientiousness has a validity of .19, and the other four personality traits sit at .10 or below.
Items worded for the workplace give higher values.
Adding "at work" to each statement gives .25 for conscientiousness and .23 for emotional stability.
Sackett and colleagues treat contextualised personality as a separate predictor.
Personality inventories are self-reports.
A candidate who intends to deceive can fake them, although Furnham notes that most people answer honestly.
Cook's summary is that inventories predict job proficiency poorly and conformity to a job's norms better.
A questionnaire added after a failed appointment therefore addresses fit with those norms and leaves the performance question open.
Cook rates references and education moderate on validity, and he flags major doubts about the legality of education as a criterion.
Use references to verify facts, and for evidence arrange a structured conversation with a named former manager about two specific behaviours.
Promotion And Succession
Internal decisions face the same choice of method.
Experience before hire has a validity of .07.
A sponsor's impression is formed without the predesigned questions that mark a structured interview.
Furnham, MacRae and Reed call the usual error the performance delusion.
It means promoting the best performer in one job into a different job.
Their example is screening the best nurses as candidates for nurse manager.
The two jobs differ fundamentally, although both carry the word nurse.
Past performance predicts future performance in proportion to the overlap between the two tasks.
The trouble with high potential lists covers how the same confusion enters a list.
A role that does not yet exist needs three questions before anyone is assessed against it.
What will this role have to decide that no current role decides
What will it have to hold together that is currently separate
What does it require that nobody in the organisation currently demonstrates
Two unstructured interviewers who meet the same candidate agree about 11 per cent of the time.
A Worked Method Choice
Take a regional finance controller role that the business fills twice a year from candidates who already hold an accounting qualification.
The hiring manager proposes a second panel interview and a personality questionnaire.
The adviser starts from the revised validity of each candidate method.
A structured interview scores .42, a job knowledge test .40, a work sample .33 and a cognitive ability test .31.
An unstructured interview scores .19, and a personality questionnaire between .05 and .25 depending on the trait and the wording.
Cost, practicality, legality and group differences then narrow the choice in three steps.
Check cost and practicality, where a work sample built from one real variance report is affordable at two hires a year
Check legality and group differences, where a cognitive ability test has the largest difference, .79 against .23 for the structured interview
Choose a structured interview scored independently, plus a review of one real variance report
A job knowledge test is the cheaper fallback if the work sample proves too slow to score.
Sackett and colleagues report single-method validities and no combined figure.
The pairing therefore rests on judgement and on a local check.
The adviser agrees in advance to compare both scores with performance ratings once appointees complete a full cycle.
How ratings behave under budget pressure limits what that comparison can show.
A few appointments settle nothing, and people analytics without a data team describes how to build the record over several years.
Conversation With The Hiring Manager
A hiring manager who wants a further stage after two failed appointments is drawing on a sample of two.
Furnham advises statistical rather than clinical prediction, which means evidence from large studies over personal experience.
Furnham adds that the greater the consequence of an error, the more rigorous the testing should be.
Four steps move the conversation from the manager's impression to the evidence.
Ask which past appointments the method predicted correctly, and what the manager knows about how those people performed
Show the revised validity of the methods in use beside the alternatives
Compare the cost of each method with the cost of a wrong appointment
Propose scoring the next three appointments both ways, and comparing the scores with performance after a full cycle
Two Changes To Make First
Two changes improve prediction in most processes at low cost.
Structure the interview, with the same questions, a written scoring standard and independent scoring before discussion
Add one job-specific measure, either a work sample or a job knowledge test, chosen by the role
The first costs preparation time once per role.
The second costs more, and it suits roles hired often enough to justify it.
Fairness And Records
In employment law the test is a bona fide occupational requirement.
That means a criterion reasonably and justifiably important to success in the job.
Three choices follow.
Every criterion is defensible as something the job requires
Every assessor applies the same criteria to every candidate, in the same order
Every decision is recorded with the evidence that produced it
Furnham also advises removing barriers unrelated to performance, such as geography, class or nepotism, which widens the pool of strong candidates.
Succession below the executive team applies the same evidence standard to internal appointments.
Where The Tools Sit
Trusted Talent Advisor places the evidence standard for an assessment decision in component 06, Evidence and Impact.
Its template pack includes the role specification and the assessment record.
Sources
1. Sackett, P. R., Zhang, C., Berry, C. M. and Lievens, F. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range. Journal of Applied Psychology.
2. Schmidt, F. L. and Hunter, J. E. The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings. Psychological Bulletin.
3. Furnham, A., MacRae, I. and Reed, M. High Potential: How to Spot, Manage and Develop Talented People at Work. Bloomsbury.
4. Cook, M. Personnel Selection: Adding Value Through People, as cited in Furnham, MacRae and Reed. Wiley.
From the system
Trusted Talent Advisor sets out the full method: seven components, 51 tools, The Talent Template Pack and The Prompt Library.
See the system
Questions this answers
What is the best predictor of job performance?
Structured interviews have the highest mean validity in the 2022 revision, at .42. Job knowledge tests follow at .40, then empirically keyed biodata at .38, work sample tests at .33 and cognitive ability tests at .31.
Are structured interviews better than cognitive ability tests?
On validity they are, at .42 against .31 in the 2022 revision, where the 1998 estimates tied them at .51. Cognitive ability tests are cheaper and suit every job, but show a larger group difference, .79 against .23.
How valid are unstructured interviews?
The 2022 revision puts their validity at .19, down from .38 in 1998. Furnham reports that ratings from two unstructured interviewers overlap by about 11 per cent, against about 50 per cent for well-structured interviews.
Does years of experience predict job performance?
Weakly. Experience before hire has a validity of .07 in the 2022 revision, against .18 in 1998, when the figure described the tenure of current employees. Experience helps most where the past work closely resembles the new work.
Do personality tests predict job performance?
Overall conscientiousness has a validity of .19, and the other four traits sit at .10 or below. Items worded for the workplace do better, at .25 for conscientiousness. Inventories are self-reports and can be faked.
What did the Sackett 2022 meta-analysis change?
Sackett and colleagues found that earlier corrections for range restriction were too large. The 1998 top five fell by between .08 and .21, and structured interviews moved to first place.
Keep reading
Measuring HR impact
Internal Mobility Metrics Without A Platform
Seven measures an HR adviser can build from existing records, each with a stated definition, a base and a way to read it.
7 min read
Workforce and future skills
Job Rotation As A Talent Gap Strategy
A job rotation programme closes a talent gap when the role, the person, the cover and the evidence are agreed with both managers before it starts.
8 min read
Talent reviews and succession
Seeing Talent Below The Top Team
Visibility of leadership potential drops below the top team. A cross-unit forum, run on one page per unit and a written evidence standard, is the method for seeing it.
8 min read