
personnel-selection
Validating and using personnel selection procedures, per the SIOP/APA Principles (5th ed., 2018): work analysis, criterion/content/internal-structure validity, generalization, fairness, scoring decisions, reporting, administration, accommodations.
- GitHub stars
- 10
Install
/plugin install personnel-selection@io-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add OpenMatter-Network/agent-io-skills.
Contents
Discovered from sourceSkills 11
Administration documentation
The operational counterpart to the technical-validation-report. It tells administrators and
users how to deliver the procedure so that scores mean what the validation says they mean. Keep it
separate from the technical report so confidential research information is protected and
administrators get only what they need. A publisher's manual often must be supplemented with
organization-specific addenda for local decisions.
A test is valid for an inference only if administered, scored, and used as it was validated. Deviations from prescribed procedures threaten the inference.
Who administers — qualifications & training
State the qualifications required for each administrative responsibility (administering, scoring, maintaining scoring algorithms, handling retest requests). Train administrators on the specific procedure and on why standardized administration matters — failures to follow protocols can render research results and operational scores partly meaningless. Build observational checks/ quality control; periodically review pass rates / mean scores for spikes or dips signaling a compromised key or administration problems. Administrators should be aware of personal limitations (physical, perceptual, cognitive) and decline to administer/score when they can't meet role demands.
What to document
- Introduction & overview — purpose; why the org uses validated procedures; importance of security and consistency. Don't oversell the program's applicability.
- Contact information — whom to contact for questions/problems.
- Selection procedures — names, editions, forms; ordering and security; sample items (not full secure content); for proprietary tools, a description of items and constructs.
- Applicability — to whom the procedure applies, exceptions/exemptions, and rules about when it's administered (e.g., only when a vacancy exists vs. periodic pooling). Include applicability to candidates with disabilities and different cultural/linguistic groups.
- Administration responsibilities & guidelines — detailed administration instructions; rules/tips for an appropriate testing environment and verifying candidate identity; for remote/unproctored internet testing (UIT/RPIT), the administrator still must inform test takers of environmental factors and security expectations.
- Administration environment — workspace, lighting, quiet, freedom from distractions, and effects of technology (browser, monitor size, touch screen). At minimum, an environment responsive to candidate concerns and dignity; inform test takers of conditions that may affect performance.
- Scoring instructions & interpretation guidelines — how to score and interpret; keep scoring keys out of manuals/technical reports (release only to those who score). Document any administrator role in interpreting scores; ensure accuracy in scoring/recording (responsibility can't be delegated away). For computer-based scoring/automated feedback, give detailed decision-use instructions.
- Reporting and using scores — how scores are reported and used, who has access; caution against using scores for unintended purposes (e.g., later retention decisions) without evidence; if scores drive differential job assignments, require evidence linking scores to those jobs.
Security
Public disclosure of content/scoring is a serious threat to reliability, validity, and subsequent use. Retain all materials at a security level allowing access only to those with a need to know. In UIT/RPIT, adopt mechanisms to reduce content compromise and cheating and to confirm identity. Communicate and enforce protections (verification codes, content rotation, supervised destruction of confidential documents); document a retention period for confidential information; consult evolving international data-privacy law.
Candidate-facing communications
- Information provided to candidates — clear, pertinent, timely: intended use, procedures, format/ interface, completion strategies (e.g., can they skip/return), time parameters, feedback/score access, confidentiality, accommodation request process, retesting policy, conduct expectations. Standardize content and delivery for consistent treatment.
- Candidate feedback — if feasible/appropriate, give clear, understandable feedback without compromising test security or scoring.
Special situations
- Nonstandard administrations — document how to handle accommodations and disruptions (power failures, emergencies, illness), errors (mismatched booklet/sheet, timing mistakes). Establish a clear process to document and explain any deviation in administration, scoring, or handling. Where reporting a nonstandard event (e.g., dropped internet connection) is left to the test taker, explain how and when to report.
- Reassessing candidates — generally provide reassessment opportunities when feasible; state clearly whether and how candidates may be reassessed (required time intervals, prerequisite developmental activities). Some one-time examinations may preclude it.
- Corrective reassessment — define when it's appropriate (administrator errors like timing mistakes/nonmatching materials; extraordinary disturbances). Document procedures, required timing, and documentation; inform remote test takers of the conditions under which they may request it.
Data retention & test-score databases
Decide what administration records/scores to keep; document record-keeping and retention policies (duration, security, accessibility) consistent with federal/state/local law, industry best practice, and recent court rulings. Retain raw item data and scores (derived scales limit future research). Secure personally identifying data to need-to-know access. Maintain databases long enough to support periodic audits and ongoing evaluation of the selection system.
Review & updating
Periodically review operational use — including timeliness of normative data — to decide whether more research is needed. The greater the changes to test format, mode of administration, instructions, or language, the more likely score interpretations no longer hold. Revise documentation (or add an addendum) whenever changes make any statement or instruction incorrect or misleading. Changes in organizational demands (new assessments, cutoff adjustments) require further study of the procedure.
Pitfalls
- Putting scoring keys or live items in the manual.
- One-size communications that aren't standardized → inconsistent candidate treatment.
- No documented process for deviations/disruptions.
- Vague or absent retest/reassessment rules.
- Letting the manual drift out of date after format/mode/language changes.
- Using operational scores for purposes never validated.
Checklist
- Administrator qualifications & training (incl. standardization rationale) documented
- Testing environment and identity-verification guidance (incl. UIT/RPIT) provided
- Scoring/interpretation documented; keys kept secure and out of the manual
- Security mechanisms and retention period specified; privacy law considered
- Candidate communications standardized; feedback approach set
- Nonstandard-administration and corrective-reassessment processes defined
- Reassessment/retest policy stated
- Data-retention and database-audit policy set
- Review/update triggers and addendum process in place
See also
technical-validation-report (keep separate) · candidate-accommodations (nonstandard
administrations) · fairness-and-bias-analysis (equitable treatment) ·
selection-decisions-and-scoring (how scores are used)
Source: Principles (5th ed., 2018), "Operational Considerations → Administration Information and Other Circumstances Regarding the Validation Effort and Use of Selection Procedures."
Candidate accommodations
Assessing candidates with disabilities may require deviations from standardized procedures to
remove construct-irrelevant barriers so a candidate can show their true standing on the
job-relevant construct. The goal is to minimize the impact of a disability that is not part of
the construct being assessed (e.g., a motor impairment depressing a cognitive ability score). This
applies throughout selection and intersects fairness-and-bias-analysis (equitable treatment and
access to the construct).
The Principles address psychological method, not the law. Accommodation obligations are governed by statute and case law (e.g., the ADA in the U.S.). Coordinate with qualified counsel.
Accommodation vs. modification — the pivotal distinction
- Accommodation — a change to environment, medium, or timing that does not alter the construct measured (e.g., high desks, Braille, text reader, adjusted time limit). Scores remain comparable to standard administration.
- Modification — a change that alters the construct (adaptations to test content are rare in employment and often change what's measured). When a test is modified so it no longer measures the same construct, its scores can no longer be directly compared to standard scores — though they may still yield useful information about the candidate's skills (dyscalculia + calculator example).
Professional judgment is required to choose accommodations with the least negative impact on the validity of the inferences. Empirical research on the effect of specific accommodations is usually lacking, so few cases support local study.
Process
- Dialog with the candidate about possible accommodations. The appropriate accommodation depends on the facts of the individual's situation.
- Decide accommodate / modify / waive / use other information. Sometimes the construct cannot be assessed without modifying it; sometimes the disability doesn't affect performance on the procedure and no accommodation is needed. Consider components of a battery separately. Users may choose to alter the procedure, develop an altered form, waive it and use other job-related KSAO information — but weigh the standardization, fairness, and job-relevance tradeoffs of those latter choices.
- Preserve comparability where possible. Retain standardized administration features to the extent feasible to maximize score comparability. Pre-specified, commonly used accommodations that are irrelevant to score interpretation (e.g., adjusting table height) may be delegated to administrators.
- Standardize the accommodation process. Document rules for eligibility, how candidates request accommodation, and how requests are evaluated; monitor implementation. Inform test takers of the process and the confidentiality provisions regarding their disability status.
- Develop & validate altered forms following the Principles to the extent possible. Pilot test when feasible (at minimum to confirm the altered procedure runs smoothly). Practical limits (small samples) restrict statistical equating, reliability, and validity estimation — pursue them whenever possible.
- Score handling & documentation. Document the changes made and the psychometric characteristics of accommodated/modified procedures; when volume permits, summarize performance of candidates with disabilities on accommodated/modified and original forms. Legal considerations may prohibit telling decision-makers whether a score was earned under accommodation and its nature — but users may flag accommodated scores for special handling in data analysis. If a test is modified so the construct changes, indicate the deviation on score reports and discuss how it may affect results/interpretation, to the extent permitted by law.
Keep selection procedures in their lane
Procedures used with candidates with disabilities should resemble as closely as possible those used for other candidates. Selection procedures are for making selection decisions, not for assessing the existence or extent of a candidate's disability — adding a procedure designed to detect disability is inappropriate as a selection tool and unlawful in many situations.
Linguistic and cultural background
- Determine the language requirements of the job to decide which language(s) the procedure is offered in; creating/maintaining a diverse workforce is usually an organizational goal.
- Ensure content and reading level are appropriate and equivalent across language forms (via translation/adaptation and relevant psychometric/statistical comparisons between groups). Inform test takers of linguistic options when appropriate.
- Cultural background can also introduce construct-irrelevant barriers; review test content and
format to minimize them while keeping the procedure consistent with the requirements of the job.
(DIF analysis is particularly useful in cross-cultural/linguistically different settings — see
fairness-and-bias-analysis.)
Pitfalls
- Conflating accommodation (same construct) with modification (different construct) and then comparing scores as if equivalent.
- Using a selection procedure to gauge a disability.
- Skipping the individualized candidate dialog / applying a rigid one-size rule.
- Disclosing accommodation status to decision-makers where law forbids it.
- Treating a translated form as equivalent without adaptation and psychometric comparison.
Checklist
- Construct-irrelevant barrier identified; accommodation chosen to least harm validity
- Accommodation vs. modification classified; comparability implications stated
- Individualized candidate dialog held; battery components considered separately
- Eligibility/request/evaluation process standardized; confidentiality assured
- Altered forms piloted; reliability/validity pursued where feasible
- Score handling documented; legal limits on disclosure respected; deviations flagged on reports
- Procedure not used to assess disability
- Language forms equivalent (content, reading level) via translation/adaptation + comparison
See also
fairness-and-bias-analysis (equitable treatment, access, DIF) · administration-documentation
(nonstandard administrations, request process) · validation-planning
Source: Principles (5th ed., 2018), "Operational Considerations → Assessing Candidates With Disabilities and Candidate Linguistic and Cultural Background."
Content-based validation
Evidence that the content of a selection procedure is a representative sample of important work behaviors, activities, and/or worker KSAOs defined by an analysis of work. The inference rests on the fidelity of the procedure's content to the work — not on an empirical predictor–criterion correlation. Often the strategy of choice when a job is unique, samples are too small for a criterion study, or the procedure is built directly from the work (e.g., work samples, job-knowledge tests).
When content evidence is and isn't the right tool
- Strong fit when procedure content is closely linked to job content.
- A construct labeled an ability or personality characteristic does not by itself rule out a content strategy — but when content is less clearly linked to job content, other sources of evidence take precedence (e.g., criterion-related, internal structure).
- Content evidence supports construct interpretation and generalization to job performance/ training when the procedure samples the work well.
SMEs define the domain — qualify them
The study's success hinges on SME qualifications. SMEs:
- define the work domain (important behaviors, activities, worker KSAOs),
- participate in the analysis of work, and
- judge the link between selection-procedure content and work/worker requirements.
SMEs must be competent to perform the work and knowledgeable about relevant organizational characteristics (shift, location, equipment, software). For behavior definition, include people fully knowledgeable about the work. Standardize and document the method for translating SME judgments into the procedure; if SME ratings evaluate the content–work match, standardize the rating procedures and criteria for each aspect.
Process
- Confirm feasibility. Are the work and worker requirements reasonably stable? Can irrelevant content be minimized? Are qualified, unbiased SMEs available? Do cost/time allow the needed fidelity?
- Define the content domain from accurate, thorough work information (behaviors, activities, responsibilities, prerequisite KSAOs). The domain need not include everything done on the job. State which important elements are included, how each links to the procedure, and why any parts were excluded. Ground the definition in organizational needs, labor markets, and specifications.
- Decide what the candidate must already have vs. will be trained. Base the procedure on the analysis that defines the balance between KSAOs expected at entry and training the organization provides. (Read-the-manual example: if employees are trained on the technical manuals, test reading ability to the required level — not manual-specific knowledge.)
- Develop or choose the procedure; sample the domain. Random sampling of the domain is usually neither feasible nor appropriate. Instead, sample the most important behaviors/KSAOs or those prerequisite to others, or a smaller set predictive of critical outcomes. Document the rationale in a test plan specifying which KSAOs are measured by which methods.
- Set the level of specificity deliberately. More fidelity to exact job components → stronger content evidence, but less resilience to change and less transferability. If the work is likely to change, build less job-specific content (e.g., software principles/techniques rather than one program). Measure skills and knowledge that are job-relevant, never irrelevant content, even when a full simulation isn't feasible (a single-step problem may substitute for a multistep work sample).
- Establish administration and scoring guidelines in advance — especially when the procedure includes work behaviors or activities that raise administration/scoring challenges.
- Evaluate the content evidence and document everything.
Evaluating content evidence
Content evidence rests on showing the procedure adequately samples and is linked to the important work behaviors/activities/KSAOs from the work analysis. The documented development methods are the primary evidence. The sufficiency of the match is a matter of professional judgment based on evidence collected in the effort. Determine reliability of performance on the procedure when feasible, choosing a reliability type that fits the measurement design and intended use (e.g., pass/fail vs. rank ordering).
Experience and training requirements as content
A procedure may include evidence of specific prior training, experience, or achievement — judged by the relationship between the content of that experience and the content of the work. Superficial resemblance is not enough: course titles and job titles may poorly indicate actual content or proficiency. Evaluate the similarity of behaviors/activities/processes/KSAOs, not labels.
Competency modeling
A competency model can found a content study only if it is detailed and rigorous — as rigorous
as a traditional work-analysis project. Judge that rigor before relying on it. See work-analysis.
Pitfalls
- Including content not part of the work domain (standardization artifacts aside) or, worse, irrelevant content.
- Over-specifying content for work that is about to change.
- Resting on a thin competency model.
- Treating face validity as content evidence (acceptability ≠ validity).
- Failing to document why domain elements were excluded.
Checklist
- Work/worker requirements reasonably stable; feasibility confirmed
- Content domain defined from thorough work information
- Entry-vs.-trained KSAOs distinguished
- Sampling rationale (importance/prerequisite) documented in a test plan
- Specificity level chosen against likelihood of work change
- SMEs qualified; linkage method standardized and documented
- Administration/scoring guidelines set in advance
- Reliability assessed where feasible
- Excluded domain elements and their rationale recorded
See also
work-analysis (defines the domain) · validation-planning ·
internal-structure-validation · generalizing-validity-evidence ·
technical-validation-report
Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence for Validity Based on Content."
Criterion-related validation
Evidence that scores on a selection procedure are statistically related to one or more measures of work-relevant behavior or outcomes. The strongest form of "does it predict?" evidence — but also the most demanding on sample size, criterion quality, and analysis.
Feasibility first
Three things determine whether a criterion-related study is even sensible:
- A relevant, reliable, uncontaminated criterion is available or buildable. Relevance is the most important property.
- A representative research sample of the workforce/candidate pool exists.
- Adequate statistical power is attainable. Estimate the needed sample size before the study from the expected effect size, the statistic, and the chosen alpha. Range restriction and criterion unreliability inflate the corrected coefficient but also inflate its standard error — so distinguish observed from corrected coefficients when judging power. Underpowered studies are a chronic failure mode; give Type II error equal attention to Type I.
If any of these is badly deficient, a criterion-related strategy may not be feasible — consider
content-based-validation or generalizing-validity-evidence.
Design: predictive vs. concurrent
- Predictive — predictor collected at/around selection; criterion collected later (after a retention interval or once performance stabilizes).
- Concurrent — predictor and criterion collected on incumbents at about the same time.
For stable cognitive abilities, predictive and concurrent estimates tend to be comparable. For noncognitive self-reports (personality, interests, SJTs) and experience-based measures (biodata), the designs can diverge — e.g., applicant faking motivation differs from incumbents'; biodata responses may reflect on-the-job experience. So results don't automatically generalize across designs or predictor types. Match the design's inference to your use.
Other design choices that matter: the basis on which sample members were selected, the population sampled (applicants vs. recent hires vs. fully experienced), and whether you're predicting higher-level work (acceptable if a substantial share advance and you use criteria at both the hire level and the higher level).
Criterion development
Choose criteria for relevance, freedom from contamination, and reliability — not availability or convenience. Criteria should represent important organizational, team, or individual outcomes.
- Relevance — reflects standing on an outcome critical to success (job performance, training success, turnover, OCBs, advancement). Need not be all-inclusive, but the link to the proposed use must be clear and rationale documented.
- Contamination — systematic variance unrelated to the construct (machinery quality, sales territory, rater knowledge of predictor scores, shift, location, rater attitudes). Minimize via standardized administration; measure and statistically control contaminants where possible.
- Deficiency — excludes relevant variance (a "performance" criterion missing behaviors critical to performance). If the criterion can't cover the full domain, state what is omitted and the implication for the inference.
- Criterion bias — systematic error from contamination/deficiency that differentially affects subgroups. Cannot be detected from criterion scores alone; anticipate and guard against it with professional judgment.
- Reliability — identify the conditions of measurement (raters, items, occasions) you want to generalize across and design to estimate the matching reliability. Internal-consistency estimates may be inadequate for ratings (they ignore rater and time variance).
Common criterion types: supervisory performance ratings (most common; ratings collected for research are preferable to administrative ratings — Jawahar & Williams, 1997), other performance indices, archival/HRIS data (verify accuracy, alignment, consistency, and data-privacy compliance before use).
Choice of predictor
- Predictors need a theoretical, logical, or empirical foundation; specify the rationale before the study. Verify serendipitous findings (especially from small samples) by independent replication.
- Base preliminary choices on the work analysis and scientific knowledge, not personal familiarity or bias.
- Keep the construct vs. method distinction explicit (e.g., "the interview" is a method, not a construct) to avoid uninterpretable comparisons.
- Address predictor contamination (unstandardized administration, irrelevant content, cheating, faking — unstructured interviews and unproctored internet tests are higher risk) and predictor reliability (estimate it for the conditions of intended use).
- When algorithms score structured/unstructured inputs (text, resumes, stimuli responses), document the conceptual/methodological basis, provide cross-validation evidence, and ensure the algorithm does not introduce systematic bias against subgroups.
Choice of participants
- The validation sample should represent the situation you'll generalize to (demographics, motivation, ability, experience). Convenience samples are discouraged to the extent they're unrepresentative.
- You generally cannot validate separately for every subgroup. Only test for bias when there is
credible evidence of potential bias and sufficient data (adequate power and precision) for the
proper analysis — see
fairness-and-bias-analysis. A subgroup too small to analyze cannot be compared until more data exist.
Data analysis
- Ensure analyses fit the data (level of measurement, sample size, non-independence/clustering). Don't pick a method because the software is handy; if you delegate analysis, you retain responsibility for its suitability and accuracy.
- Report effect sizes, statistical significance, and standard errors / confidence intervals for predictor–criterion relationships; describe distributions (central tendency, variance) and interrelationships.
Adjustments (corrections) to validity estimates
- Aim for an unbiased estimate of operational validity. Observed coefficients misestimate the population value because of range restriction and criterion unreliability; apply suitable bivariate/multivariate corrections when an appropriate estimate is available.
- Do not correct for predictor unreliability for operational use (you use the actual, unreliable predictor).
- For ratings criteria, internal-consistency reliability may be inadequate — account for rater and time variance.
- Corrected coefficients are point estimates: usual significance tests/SEs for unadjusted coefficients don't apply. Report both corrected and uncorrected values, and use procedures built for corrected coefficients when testing significance / forming CIs. State explicitly when a coefficient is theoretical and not the operational validity.
Combining predictors and criteria
- Document the method of combination: multiple regression, Pareto-optimal weights, unit weights, rational weights, etc. Effective weights ≠ nominal weights (they depend on variances/covariances), especially when predictors are differentially range-restricted.
- Compensatory vs. noncompensatory combination (and cutoffs/sequencing) affects rank order,
expected mean criterion standing, and subgroup differences — see
selection-decisions-and-scoring. - Cross-validate composites built from regression (or any data-driven weighting/keying) to guard against capitalization on chance, especially in small samples. Unit/rational weights don't shrink.
Interpreting results
Interpret against the cumulative research literature. Unusual findings (suppressors, moderators, nonlinearity, configural scoring, differential weighting of highly correlated predictors) are suspect — require a very large sample or replication before acting on them.
Pitfalls
- Choosing an available criterion over a relevant one.
- Ignoring power until after data collection.
- Reporting only corrected validities, or applying naive significance tests to them.
- Correcting for predictor unreliability and treating it as operational validity.
- Capitalizing on chance via regression/empirical keying without cross-validation.
- Generalizing concurrent results to a predictive (applicant) use for fakable measures.
Checklist
- Criterion is relevant, reliable, and as free of contamination/deficiency as feasible
- Design (predictive/concurrent) justified for the predictor type and proposed use
- A priori power analysis done; needed N attainable
- Sample represents the target population
- Predictor rationale documented; construct/method distinction kept
- Analyses appropriate; non-independence considered
- Corrected and uncorrected coefficients reported with SEs/CIs
- No predictor-unreliability correction for operational claims
- Composites cross-validated; combination method documented
- Bias examined only where evidence + power warrant (see fairness skill)
See also
work-analysis · validation-planning · generalizing-validity-evidence ·
fairness-and-bias-analysis · selection-decisions-and-scoring · technical-validation-report
Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Criterion-Related Evidence" and "Operational Considerations → Selecting Criterion Measures / Data Analyses."
Fairness and bias analysis
Two distinct ideas that are routinely conflated. Keep them separate.
Fairness is a social concept, not a single statistic
"Fairness" has no single agreed definition (statistical, psychometric, or social). Recognized meanings include:
- Equal group outcomes (e.g., equal passing rates). The Standards reject this as the definition of fairness: group outcome differences alone do not indicate bias — though they should trigger heightened scrutiny for possible bias.
- Equitable treatment of all examinees (testing conditions, access to practice materials, feedback, retest opportunities, reasonable accommodation, mode of administration).
- Comparable access to the construct — accessible testing so all candidates can show their standing without being advantaged/disadvantaged by construct-irrelevant characteristics (age, race, ethnicity, gender, SES, cultural/linguistic background, disability).
- Lack of bias.
There is broad agreement that equitable treatment, access, bias, and scrutiny when subgroup differences appear are important — but no agreement that "fairness" can be uniquely defined in terms of any one of them.
Bias is a technical concept — two forms
Bias = systematic error that differentially affects the performance of different subgroups.
- Predictive bias — slope and/or intercept of the predictor→criterion regression differs across groups (a predictor–criterion relationship issue).
- Measurement bias — construct-irrelevant variance producing systematically higher/lower scores for a subgroup (a score issue, for predictors or criteria).
Crucial point on consequences
A subgroup-mean difference (adverse impact) is a negative consequence, but it is evidence against validity only if it traces to a measurement property of the procedure (i.e., bias). If the group difference on the procedure mirrors a real difference in the work-relevant outcome (i.e., no predictive bias), the consequence is a policy issue for the user, not a validity defect.
Predictive bias / differential prediction
Test via moderated multiple regression (MMR): regress the criterion on the predictor, subgroup membership, and their interaction. Slope and/or intercept differences signal predictive bias. MMR is preferred over comparing separate subgroup correlation coefficients.
- Frame the question as "is the subgroup's performance underpredicted?" — only underprediction signals bias against that group. Simply knowing slopes/intercepts differ doesn't answer it. (In U.S. cognitive-ability research, slope differences are rare; when intercept differences occur they typically take the form of overprediction of minority performance — Schmidt, Pearlman, & Hunter, 1980; and corrected analyses, e.g., Berry & Zhao, 2015, still find little underprediction.)
- Consider effect sizes as well as statistical significance (Nye & Sackett, 2017; Dahlke & Sackett, 2017).
Technical cautions (predictive bias)
- Analyze predictors as operationally used (e.g., test the composite when selection uses a composite, not each test separately).
- A confident, unbiased criterion is a prerequisite.
- Statistical power is a chronic problem — small total/subgroup samples, unequal subgroup sizes, range restriction, and predictor unreliability all reduce power to detect slope/intercept differences.
- Check the homogeneity-of-error-variance assumption; use alternative tests when it's violated.
- Use an unbiased estimate of the intercept difference and operational validity parameters (not observed parameters).
Predictive bias and mean differences can exist independently; analyze predictive bias when there's compelling reason to question whether predictor and criterion relate comparably across subgroups and appropriate data exist. Where relevant research exists, generalized evidence can inform the question.
Measurement bias
Construct-irrelevant variance raising/lowering scores for a subgroup — hard to detect because it requires comparing an observed score to a true score. Approaches:
- Item sensitivity review — diverse reviewers examine items (and instructions to candidates and scorers) for language/content that could carry differing meaning across subgroups or be demeaning/offensive. Value depends on content; use is a matter of professional judgment.
- Differential item functioning (DIF) — identifies items on which members of different subgroups
with the same total score (or same IRT true score) perform differently. Notes:
- Needs large samples for stable results.
- Domains where DIF is common have rarely shown sizable, replicable DIF (Sackett et al., 2001); for cognitive tests it's common to find roughly equal numbers of items favoring each subgroup, netting to little test-level bias.
- DIF is not a routine/expected part of selection development; explore it when appropriate data exist. Especially useful in cross-cultural / linguistically different testing.
Pitfalls
- Equating adverse impact with bias, or "no bias" with "fair."
- Testing each component instead of the operational composite.
- Running underpowered bias analyses and reading a null as "no bias."
- Comparing subgroup correlations instead of using MMR.
- Treating any slope/intercept difference as bias without asking about underprediction direction.
Checklist
- "Fairness" meaning(s) at issue named explicitly
- Subgroup differences treated as a scrutiny trigger, not a verdict
- Predictive bias tested with MMR on the operational predictor/composite
- Underprediction direction (not mere difference) interpreted; effect sizes reported
- Power, range restriction, unreliability, error-variance homogeneity addressed
- Unbiased parameter estimates used
- Measurement bias considered (item sensitivity review and/or DIF) where data/justification exist
- Equitable treatment and access to the construct addressed (see accommodations skill)
See also
criterion-related-validation · selection-decisions-and-scoring (composites & subgroup tradeoffs)
· candidate-accommodations (equitable treatment/access) · internal-structure-validation ·
technical-validation-report
Source: Principles (5th ed., 2018), "Fairness and Bias."
Generalizing validity evidence
When accumulated evidence is strong enough, you may justify a procedure in a new setting without a local criterion study — by demonstrating generalized validity and making a compelling argument for direct applicability to your situation. Three strategies (not mutually exclusive, not exhaustive): transportability, synthetic/job-component validity, and meta-analytic validity generalization. All require an analysis of the work to establish the link.
1. Transportability
Apply a specific procedure in a new situation based on a validation study conducted elsewhere, when key similarities make that evidence applicable.
- Carefully review the original study for technical soundness and conceptual/empirical relevance to the new situation.
- Compare on job content, job requirements, job context, and (if feasible) the applicant group.
- Document the comparison between the original validation sample/setting and your target use.
2. Synthetic validity / job-component validity
Justify use based on the validity of inferences for one or more components (job components) of the work, established for those components, then "synthesized" (empirically combined) for a given job or job family.
- Requires a detailed analysis of work that decomposes jobs into components and identifies which components a job comprises.
- Powerful when many jobs share common components — provides a validity source where a separate criterion study per job isn't feasible, and reduces burdensome data collection.
- Caveat: job requirements unique to a job may not be well covered by components and may need other evidence.
3. Meta-analytic validity generalization (VG)
Cumulate validity findings across studies to estimate the predictor–criterion relationship for the domains/settings of interest, and to determine how specific/generalizable the relationship is.
- VG shows that much of the observed variation in validity across settings is due to statistical artifacts (sampling error, range restriction, criterion unreliability). Well established for cognitive ability; accruing for several noncognitive measures.
- Organize around constructs. Generalization is straightforward when results are organized by predictor and criterion constructs. Method labels alone (e.g., "interview," "SJT," "biodata") are not constructs — a method can be designed to assess very different constructs, so you can only generalize to applications of the method that share the relevant features (content, scoring, meaning of scores).
- Evaluate the meta-analysis itself: methods and assumptions, their tenability, statistical artifacts that could bias results, and moderators. When a substantive moderator is plausible, consider statistical power and the precision of reported effects to detect it. Insist on full reporting of how studies were categorized and analyzed; missing/unreported information undermines the inference.
Local study vs. generalized evidence
Generalized evidence is often more useful than a single small local study. But a competent local study with a large, organizationally relevant sample using the same test for the same work can be more accurate and informative than an accumulation of small, heterogeneous, or deficient studies that don't represent your setting. Watch for representation gaps: if your setting (e.g., a managerial job) isn't represented in the meta-analytic database (e.g., limited to entry-level jobs), a local study may be more relevant. A Bayesian approach can formally combine meta-analytic priors with locally estimated coefficients (Newman, Jacobs, & Bartram, 2007).
Sole reliance on cumulative evidence may also be insufficient for operational needs (e.g., optimal placement, combining procedures in a broader system) — supplementary local or cooperative studies may still be warranted.
When generalization is NOT warranted
- Generalizing from a method studied for one construct to the same method used for a different construct (e.g., technical-knowledge interviews → interpersonal-skill interviews).
- Applying meta-analytic results from one set of procedures/settings to a new setting using different, unspecified procedures.
- Assuming predictor/criterion measures sharing a construct label are interchangeable without rational and empirical support.
Pitfalls
- Skipping the analysis of work that links the borrowed evidence to your job(s) — required for all three strategies.
- Organizing meta-analytic evidence by method label ("interview," "SJT") instead of by construct.
- Treating generalized evidence as automatically superior, ignoring representation gaps (your setting absent from the meta-analytic database) where a local study would be more relevant.
- Transporting a study without reviewing its technical soundness and comparing job content, requirements, context, and applicant group.
- Relying solely on cumulative evidence when operational needs (placement, combining procedures) require local data.
Checklist
- Strategy chosen (transport / synthetic / VG) with rationale
- Analysis of work links the borrowed evidence to your job(s)
- Transport: original study reviewed; content/requirements/context/applicant comparison documented
- Synthetic: job decomposed into components; component validities established
- VG: evidence organized by constructs, not method labels
- Meta-analysis methods, assumptions, artifacts, and moderators evaluated
- Representation gaps and operational-need gaps assessed; local/Bayesian supplement considered
- Direct-applicability argument to the current setting articulated
See also
validation-planning · work-analysis (required link) · criterion-related-validation
(local alternative/supplement) · fairness-and-bias-analysis · technical-validation-report
Source: Principles (5th ed., 2018), "Generalizing Validity Evidence."
Internal-structure validation
Evidence about how the components of a procedure (items, tasks, subscales) relate to one another and conform to the conceptual framework that defines the procedure. Useful for planning and developing a procedure and for supporting interpretation of total and subscores.
The critical limitation — state it up front
Internal-structure evidence does not, by itself, establish job relatedness. It shows the
procedure measures the intended construct(s); it does not show those scores relate to work
behavior or outcomes. Job relatedness still requires linking scores to the work via
criterion-related-validation, content-based-validation, or generalizing-validity-evidence.
Treat internal structure as support, not a stand-alone case.
Match the analysis to the conceptual framework
The relevant analyses depend on the procedure's intended structure:
- Single dimension/construct intended → offer evidence that covariances among components are accounted for by a strong single factor. Coefficient alpha / internal consistency can be appropriate evidence here.
- Multidimensional structure intended (hypothesized multifactor measure) → carefully examine dimensionality (e.g., confirmatory factor analysis fitting the proposed structural model). Overall internal consistency may be inappropriate as the index of choice.
Item inclusion should be driven primarily by relevance to the construct/content domain and only secondarily by intercorrelations. Well-constructed components with near-zero correlations with other components, scales, or the total score should not automatically be eliminated — if the procedure deliberately spans different construct/content domains (e.g., a battery of a reading test, an in-basket, and an interview), low inter-component correlations are expected and fine.
Cautions on coefficient alpha
High internal consistency can be misleading:
- A long, multi-dimensional measure can show high alpha simply because of the number of items (Cortina, 1993), masking multidimensionality.
- A performance-rating form with theoretically unrelated scales can show high alpha because of halo effect, not true unidimensionality.
So a high alpha is not proof of a single construct. Generic internal-consistency indices do not evaluate internal structure for multidimensional procedures.
Subscores
When a multidimensional structure is proposed, evidence supporting inferences about subcomponent scores may be needed if those subscores are interpreted or used. Don't report/interpret subscores you haven't supported.
Process
- State the conceptual framework: how many dimensions/constructs, and how components map to them.
- Choose analyses that fit that framework (single-factor evidence vs. CFA/dimensionality study).
- Select components by construct/content relevance first; don't purge low-correlating items that belong conceptually.
- Use alpha only where a single dimension is intended; otherwise examine dimensionality directly and beware item-count and halo artifacts.
- Support any subscores you intend to interpret.
- Remember to establish job relatedness separately.
Pitfalls
- Treating internal-structure evidence as establishing job relatedness — it does not; that requires a separate source linking scores to work behavior/outcomes.
- Reporting a high coefficient alpha as proof of unidimensionality (item-count and halo artifacts).
- Purging well-constructed components that correlate near-zero with others when they belong conceptually to a deliberately multidimensional procedure.
- Using overall internal consistency as the index of choice for a hypothesized multifactor measure.
- Interpreting or reporting subscores without supporting evidence for those subcomponent inferences.
Checklist
- Intended dimensionality (uni- vs. multi-) stated before analysis
- Analysis matched to that framework (single-factor evidence or CFA)
- Items/components chosen on construct/content relevance, not just correlations
- Alpha used only for intended-unidimensional measures; artifacts considered
- Subscores supported if interpreted
- Job-relatedness evidence obtained via a separate source
See also
criterion-related-validation · content-based-validation · generalizing-validity-evidence ·
fairness-and-bias-analysis (DIF is an item-level structural concern) · technical-validation-report
Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence of Validity Based on Internal Structure."
Selection decisions and scoring
How scores become hiring/promotion decisions. Every choice here affects validity, expected performance of those selected, and subgroup passing rates — so each requires a documented rationale. There are few absolutes; professional judgment driven by organizational goals governs.
Combining procedures into a system
When multiple procedures form the basis of a decision, both the individual components and the combination must be supported by validity evidence. Document the method and rationale for combining and sequencing. Organizations weight differently depending on whether they emphasize maximizing validity, minimizing subgroup differences, or balancing the two — state which.
Recall from criterion-related-validation: effective weights ≠ nominal weights (they depend on
component variances/covariances, and differential range restriction can distort them).
Compensatory vs. multiple hurdle
- Compensatory — candidates achieve a specified total across assessments; a high score on one can offset a low score on another.
- Multiple hurdle — candidates must clear a score on each assessment (often sequential).
- Hybrids are possible (a hurdle on one critical predictor, compensatory among the rest).
No model is universally correct. The method of combining scores can affect the overall reliability of the process and subgroup passing rates (Sackett & Roth, 1996). Present the rationale and supporting evidence for the model recommended.
Cutoff scores vs. rank order
Two common strategies: a cutoff score (reject below a point) or rank-order / top-down selection.
- There is no single best method for setting cutoffs. Options include criterion-referenced cutoffs (when the predictor links to a meaningful performance threshold) and others (Mueller, Norris, & Oppler, 2007).
- With valid predictors showing linearity, cutoffs may be set as high or low as needed to meet organizational requirements. Nonmonotonicity in the predictor–criterion relation should inform how scores are used.
- When local data can't establish linearity/monotonicity, consult past research and its implications (e.g., cognitive ability tends to relate linearly to performance; linearity for other predictors such as personality is less settled).
- When setting any cutoff, consider the conditional standard error of measurement at the cutoff region; document the SEM model used. Consider reporting the percentage of applicants classified the same way (pass/fail) across replications at the cutoff (Haertel, 2006).
Bands
Bands are score ranges within which candidates are treated alike (a form of cutoff that defines ranges). Methods vary (Cascio, Outtz, Zedeck, & Goldstein, 1991; Campion et al., 2001).
- Rationale can be psychometric (imprecision/SEM of scores) and/or administrative/ organizational.
- Tradeoff: because banded candidates with different scores are treated alike, banding generally yields lower expected criterion performance and utility than top-down selection — but may be balanced by administrative ease and possibly increased workforce diversity, depending on how within-band selection is done.
- Document the basis for development and the decision rules for administering the band.
Top-down vs. cutoff vs. bands — the driving factors
Decisions are typically driven by organizational goals and factors such as: estimated cost–benefit ratio, number of vacancies, selection ratio, labor market, expectancy of success vs. failure, consequences of selection errors, relative emphasis on performance vs. diversity goals, judgments about the level of KSAO/performance required, and the procedure's utility. Some organizations choose a cutoff over rank order to increase diversity, accepting possible reductions in performance and utility. Whatever the decision, document the rationale.
Norms
Present normative information for the applicant pool and incumbent population when appropriate. Describe the normative group's relevant demographic/occupational characteristics and the time frame. Note that large discrepancies between incumbents and the applicant pool can make incumbent-based cutoffs too high (or otherwise inappropriate).
Communicating effectiveness
- Expectancy charts relate score ranges to work performance; Taylor–Russell tables show the proportion of hires who will be successful under combinations of validity, selection ratio, and base rate.
- Utility estimates project productivity gains (in dollars, output %, or reductions in accidents/person-hours). Utility values rest on assumptions and uncertain parameters — report them as estimates, and present minimal and maximal point estimates to reflect uncertainty.
Appropriate use
Use a procedure only for purposes with validity evidence. Changing the mix or combination of components (especially in a compensatory system) can fundamentally change the supported inference — the original validation evidence may no longer suffice. Don't repurpose a procedure (e.g., diagnostic use, or an education-designed test for employment) without supporting evidence.
Pitfalls
- Combining procedures without validating the combination, not just the parts.
- Setting cutoffs without considering linearity/monotonicity or the conditional SEM.
- Presenting utility as precise dollar truth rather than an assumption-laden estimate.
- Using incumbent norms to set cutoffs when the applicant pool differs markedly.
- Silently changing weights/combination and assuming prior validity carries over.
Checklist
- Combination & sequencing method documented with rationale
- Compensatory/hurdle/hybrid choice justified; reliability & passing-rate effects considered
- Cutoff/rank/band choice tied to org goals (selection ratio, vacancies, error costs, diversity)
- Linearity/monotonicity and conditional SEM addressed for cutoffs
- Band development basis and within-band decision rules documented
- Norms described (group, demographics, time frame); incumbent-vs-applicant gap considered
- Effectiveness communicated (expectancy/utility) with uncertainty ranges
- Validity evidence covers the actual operational use
See also
criterion-related-validation (weights, composites, corrections) · fairness-and-bias-analysis
(subgroup tradeoffs; analyze the operational composite) · technical-validation-report ·
administration-documentation
Source: Principles (5th ed., 2018), "Operational Considerations → Data Analyses (Combining procedures, Multiple hurdles vs. compensatory, Cutoff scores vs. rank orders, Bands, Norms) and Communicating the Effectiveness / Appropriate Use of Selection Procedures."
Technical validation report
The durable record of a validation effort. Standard: enough detail that a competent professional not involved in the study could understand what was done, evaluate it, replicate it, and make recommendations. Report findings, interpretations, and the decisions based on them accurately, including findings that qualify the conclusions or limit generalizability.
Keep it separate from the administration manual and from secure operational content (don't put
scoring keys or live items in it). See administration-documentation.
Required sections
Use report-outline.md as a fill-in template. Each section below maps to it.
- Identifying information — authors, credentials, affiliations, dates, and what's needed to know who conducted the original research.
- Statement of purpose — the purpose of the validation research.
- Analysis of work — description of the analysis, participant characteristics, SME judgments, instructions given to participants for their tasks, analyses and results including reliability/precision. If any materials were translated, describe the translation/adaptation.
- Search for alternative selection procedures — document any search for procedures (or alternate combinations) showing substantially equal or greater validity with less subgroup difference.
- Identification or development of selection procedures — names, editions, forms of published procedures; for proprietary tools, the construct(s) measured, the content, and how content was developed. Do not include scoring algorithms/keys or live items (protect security); do document scoring procedures enough to ensure accurate, consistent scoring. Give the rationale for each statistical procedure and analysis.
- Establishing validity — describe the validation studies so another professional could
reproduce analyses and results; describe methods used to determine the procedure is statistically
and practically related to a criterion and/or representative of a job-content domain.
- Criterion studies: report criterion measures; the rationale for them; data-collection procedures; and a discussion of relevance, reliability, possible deficiency, possible contamination, and freedom from/control of biasing variance. If you developed the criterion, give the rationale and steps.
- Research sample — sampling procedure and sample characteristics relative to interpretation; the population the sample represents; sampling biases; significance of deviations from representativeness; statistical power results. Note range restriction in scores. State whether psychometrics refer to candidates or incumbents. Don't present concurrent results as predictive.
- Results — all summary statistics tied to the conclusions and recommendations; complete statistical results (not just significant/supportive ones), clearly labeled. Present both uncorrected and corrected values when correcting for artifacts (range restriction, criterion unreliability).
- Scoring and transformation of raw scores — methods/algorithms used to score content; rationale for weighted/derived/composite/categorical scores; for judgment-based scoring (work samples, performance tasks), describe rater training and scoring criteria.
- Normative information — normative parameters and how to interpret them; demographic/ occupational characteristics, data-collection time frame, and status of test takers (candidates/ incumbents/students); central tendency, variability (and skewness when appropriate); clear description of the normative data (percentiles, standard scores); percent passing; expectancy tables.
- Recommendations — recommended use and the rationale (rank order, bands, cutoffs; how information is combined). Note that some implementation rules may change over time; place subsequent modifications in an addendum.
- Caution regarding interpretations — help readers interpret data correctly and warn against common misuses.
- Technology-enabled selection procedures — if technology-enabled, document the technology requirements and any technology-based accommodations administrators can provide for test takers with disabilities.
- References — complete references for all published literature and technical reports cited. (Note: proprietary/confidential technical reports may not be generally available.)
Cross-cutting reporting expectations
- Report the rationale for each statistical procedure and analysis performed.
- If raters are integral (e.g., some work samples), determine and document rater reliability/agreement.
- Document decision rules used in preparing data (handling of missing data, outliers, inconsistencies across sources) and justify them.
- For transportability support, include the relationship between the original validation sample and the population for which use is now proposed.
Pitfalls
- Reporting only significant/favorable results.
- Reporting only corrected coefficients (always pair with uncorrected) or applying naive significance tests to corrected values.
- Presenting concurrent results as if predictive.
- Putting secure content (keys, live items) in the report.
- Omitting power analysis, range-restriction notes, or the criterion's deficiency/contamination discussion.
- Failing to document the search for less-adverse alternatives.
Checklist
- All 14 sections present (use
report-outline.md) - Replicable: methods/analyses described in enough detail
- Corrected and uncorrected results; power; range restriction noted
- Criterion relevance/reliability/deficiency/contamination discussed
- Sample representativeness, population, incumbent-vs-candidate status stated
- Alternatives search documented
- Secure content excluded; scoring documented enough for consistency
- Data-handling decision rules justified
- Misuse cautions and interpretation guidance included
- References complete
See also
administration-documentation (the separate operational manual) · every evidence skill
(criterion-related-validation, content-based-validation, internal-structure-validation,
generalizing-validity-evidence) · fairness-and-bias-analysis · selection-decisions-and-scoring
Source: Principles (5th ed., 2018), "Operational Considerations → Technical Validation Report."
Validation planning
The front end of any selection project. Validation should begin with a clear statement of the proposed uses and the intended interpretations and outcomes — then a design is chosen to support those inferences. Getting this wrong wastes the whole study: a poorly executed effort can lead an employer to reject a beneficial procedure or adopt an invalid one. "Misleading, poorly designed validation efforts should not be undertaken."
What validity means here
Validity is the degree to which evidence and theory support the interpretation of scores for a proposed use. You validate an inference (e.g., "scores predict job performance"), not a test. Plan around the inference you must defend.
The Standards name five sources of validity evidence; for employment the first three usually carry the argument:
- Relationships with other variables (test–criterion) — see
criterion-related-validation - Content — see
content-based-validation - Internal structure — see
internal-structure-validation - Response processes (often irrelevant to employment use)
- Consequences (relevant to validity only when a negative consequence traces to a measurement property of the procedure — e.g., bias; mere subgroup-mean differences are not, by themselves, evidence against validity)
Sources are not competing "types" of validity; they are complementary evidence. Plan to combine them where the inference is complex, novel, or high-stakes.
Steps
- Define the organization's needs, objectives, and constraints. Work collaboratively with HR, recruiting, legal/compliance, IT, labor relations, and affected stakeholders. Different units have competing objectives (rigor vs. applicant flow). Run a cost–benefit lens over candidate procedures.
- Specify the proposed use(s) precisely. Target job(s) or job family, candidate pool (experienced vs. inexperienced; applicants vs. incumbents), the decision (hire, promote, place, certify), and how scores will be used (rank order, cutoff, band). The use drives the design.
- Characterize the setting. One homogeneous organization or many? Stable work or rapidly changing? Workforce size and applicant availability — these constrain whether a local study is even feasible and which strategies (local criterion study, VG, synthetic, content) are open.
- Inventory existing evidence. Is there sufficient accumulated validity evidence (meta-analytic
bases for cognitive ability and a growing set of noncognitive measures; prior local studies;
transportable studies) to support the proposed use without new data? Weigh the informational
value of new evidence against its cost. Existing evidence alone may or may not suffice — judge
the strength of the generalization to your setting. See
generalizing-validity-evidence. - Apply the requirements of sound inference. Reliable and relevant measures, representative samples, appropriate analyses, controls over plausible confounds, and people qualified for the tasks they perform in the study.
- Assess feasibility. Time, cost, sample size/statistical power, organizational disruption, union–management relations. Constraints may legitimately narrow the scope of defensible generalizations — but never justify a misleading design.
- Choose a validation strategy that fits objectives, constraints, and the procedures/criteria.
Often more than one source of evidence is valuable. Three illustrative fits:
- Small population, strong cumulative evidence in similar settings → validity generalization.
- Same job extended from one business unit to another → transportability.
- Position unique to the organization, no external evidence → content-based strategy.
- Communicate the plan. Management and workers need the purpose, the research plan, and their roles. Decide and communicate confidentiality (e.g., concurrent study data kept out of employment decisions). Document the plan.
Decision guidance: which strategy?
| Situation | Lean toward |
|---|---|
| Adequate sample, relevant criterion obtainable, need local proof of prediction | criterion-related |
| KSAOs/behaviors sampled directly from work; content closely mirrors the job | content-based |
| Strong external cumulative evidence; small local sample | meta-analytic validity generalization |
| Same job already validated in a comparable unit | transportability |
| Many jobs share common work components | synthetic / job-component validity |
| Multidimensional procedure whose internal structure must be defended | internal structure (as support) |
Pitfalls
- Designing the study before pinning down the inference and use.
- Treating subgroup-mean differences as automatic evidence against validity (they trigger scrutiny for bias, not a verdict).
- Underpowered local studies: adequate power while controlling Type I error often needs samples that are hard to obtain — consider this before committing to a criterion-related design.
- Letting convenience (available criterion, available sample, available software) drive design.
- Promising legal defensibility — the Principles address psychological method, not the law.
Checklist
- Proposed use(s), target job(s), and candidate pool stated explicitly
- Inference(s) to be supported written down
- Existing evidence inventoried; gap (if any) justified
- Feasibility (power, cost, time, access) assessed honestly
- Strategy chosen and rationale documented
- Stakeholders identified; communication & confidentiality plan set
- Plan documented (feeds
technical-validation-report)
See also
work-analysis · criterion-related-validation · content-based-validation ·
internal-structure-validation · generalizing-validity-evidence · technical-validation-report
Source: Principles (5th ed., 2018), "Overview of the Validation Process" and "Operational Considerations → Initiating a Validation Effort / Selecting the Validation Strategy."
Analysis of work
"Analysis of work" is the Principles' umbrella term for what has been called job analysis, work analysis, and competency modeling. It is the empirical foundation that links a selection procedure to the job. Most other validation strategies depend on it.
Two purposes — be clear which you're serving
- Define worker requirements (predictors). Identify the KSAOs/competencies workers need to be successful, and how similar requirements are across settings or job families.
- Develop criteria. Assemble the information needed to understand the work, its setting, and the organization's goals so you can build relevant, uncontaminated criterion measures.
A single analysis can serve both, but design it deliberately for the purpose(s) at hand.
Level of detail — match it to the use
The required detail depends on the intended use and the availability of existing information.
- Less detail may suffice when prior research lets you generalize requirements across a job family, or when an indicator's relevance is self-evident (e.g., absenteeism, turnover are relevant to virtually all jobs).
- More detail is needed when little work information exists, or when you intend to develop predictors of specific job knowledge (content-based work especially).
There is no single preferred method. Choose methods based on the nature of the work, the organizational setting, the workers, and the study's purpose, informed by the research literature.
Constructs vs. methods (frame the requirements correctly)
Distinguish what is measured (the predictor construct, e.g., general mental ability, conscientiousness, job knowledge) from how it is measured (the method, e.g., interview, work sample, test). Confounding the two (e.g., "the interview" as if it were a construct) produces uninterpretable comparisons later. Define requirements at the construct level where possible.
Process
- Scope the work domain. Work complexity, environment, context, tasks, behaviors, activities, and worker requirements. Decide which dimensions of work matter for your purpose.
- Choose methods appropriate to the work and purpose (e.g., observation, interviews, questionnaires, critical incidents, task inventories, existing documentation, O*NET, competency modeling). Any method used should be understood by participants and have reasonable psychometric properties; note and investigate lack of SME consensus.
- Plan the SME sample. Include incumbents/SMEs spanning relevant shifts, locations, equipment, software, experience levels, and demographics. A broad, representative SME sample increases the representativeness of results. Don't assume same job title = same work — study multiple incumbents when complexity, context, or behaviors may differ.
- Vet existing documentation. Existing job descriptions and prior analyses may or may not serve the current purpose; evaluate relevance and currency before relying on them.
- Account for changing work. When work is being transformed (or the job doesn't exist yet), obtain reliable, relevant information about anticipated behaviors, activities, and KSAOs/competencies. Reanalyze when the nature of work has changed meaningfully since any prior analysis.
- Derive worker requirements and/or criterion content. Translate work information into KSAOs/competencies (for predictors) and into work behaviors/outcomes (for criteria).
- Document methodology, data collection, analyses, results, and implications — including the
major work activities, important worker requirements, and their relationships to selection
procedure content and scoring. Feeds the
technical-validation-report.
Competency modeling — when it counts as work analysis
Competency models are widely used (for training, selection, compensation, signaling aspirational goals). A competency model can serve as the foundation for content-oriented or other validation only if it is detailed and rigorous — i.e., built like a rigorous work analysis, not a list of aspirational labels. You must judge whether the model is rigorous enough to support the intended inference. (See Campion et al., 2011, on best practices.)
Pitfalls
- Letting a glossy competency model substitute for a rigorous analysis when content evidence rests on it.
- Grouping dissimilar jobs under one title and analyzing them as if homogeneous.
- Defining requirements as methods ("we need interviewing") instead of constructs.
- Using stale job descriptions for rapidly changing work.
- Insufficient/biased SME sampling.
Checklist
- Purpose (predictor requirements, criterion development, or both) stated
- Level of detail justified by the intended use
- Methods chosen and rationale recorded; psychometric quality considered
- SME sample spans relevant subgroups, shifts, locations, experience
- Requirements expressed as constructs (KSAOs/competencies), not methods
- Changing/future work accounted for if relevant
- Results documented and linked to selection procedure content and/or criteria
See also
validation-planning (upstream) · content-based-validation and criterion-related-validation
(downstream consumers) · technical-validation-report
Source: Principles (5th ed., 2018), "Overview → Analysis of Work" and "Operational Considerations → Understanding Work and Worker Requirements."
Sponsors
Details
- Marketplace
- OpenMatter-Network/agent-io-skills
- Source
- personnel-selection
- Category
- Community