Evaluating Job Applicants:
e Role of Training and Experience in Hiring
A Report to the President and the Congress of the United States
by the U.S. Merit Systems Protection Board
JANUARY 2014
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S. MERIT SYSTEMS PROTECTION BOARD
1615 M Street, NW
Washington, DC 20419-0001
e President
President of the Senate
Speaker of the House of Representatives
Dear Sirs:
In accordance with the requirements of 5 U.S.C. § 1204(a)(3), it is my honor to
submit this U.S. Merit Systems Protection Board report, Evaluating Job Applicants: The Role
of Training and Experience in Hiring. The purpose of this report is to help Federal agencies
improve how they assess candidates for Federal jobs. Better qualified new hires mean higher
productivity and better services to the public. Accurate assessments and unbiased selection
procedures also support merit system principles such as selection and advancement based
solely on relative ability, knowledge, and skills, fair and equitable treatment of applicants and
employees, and efficient and effective use of the Federal workforce.
In times of fiscal austerity, it is essential that training and experience assessments
play a central role in Federal hiring. Typically, training and experience assessments use past
accomplishments as an indicator of an applicant’s potential for job proficiency. While this
approach is generally well regarded by applicants and managers, research shows that some
of these assessments do little to predict successful job performance, and practical experience
shows that it can be difficult to obtain accurate and detailed information from job applicants.
This report examines how training and experience is assessed, identifies challenges in obtaining
accurate and useful information from applicants, and provides strategies for improving such
assessments.
As agencies become better at identifying and selecting the most productive workers,
the Federal Government will operate more efficiently. I believe you will find this report useful
as you consider issues affecting the Federal Government’s ability to assess and select a high-
quality workforce.
Respectfully,
Susan Tsui Grundmann
Enclosure
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
A Report to the President and the Congress of the United States
by the U.S. Merit Systems Protection Board
JANUARY 2014
Evaluating Job Applicants:
e Role of Training and Experience in Hiring
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S. Merit Systems Protection Board
Susan Tsui Grundmann, Chairman
Anne M. Wagner, Vice Chairman
Mark A. Robbins, Member
Office of Policy and Evaluation
Director
James M. Read
Deputy Director
James J. Tsugawa
Project Manager
John M. Ford, Ph.D.
Project Analysts
Julie K. Osowski, Ph.D.
Doug Nierle
Sharon Roth
Allison M. Wiley
Table of Contents
Executive Summary...................................................................................... i
Chapter 1 - Background................................................................................ 1
What is Assessment of Training and Experience?....................................... 1
Why Study Training and Experience Assessments?.................................... 3
In This Report.......................................................................................... 8
Chapter 2 - How is Training and Experience Assessed?................................. 9
Useful T&E Assessments........................................................................... 12
Assessments that Include T&E Information............................................... 17
Problematic T&E Assessments...................................................................20
Summary...................................................................................................24
Chapter 3 - What Are the Common Challenges?............................................27
Focus on the Past......................................................................................27
Unreliable Self-Assessment.......................................................................30
Applicant Dishonesty................................................................................33
Summary...................................................................................................34
Chapter 4 - How Can We Raise the Bar?........................................................35
Improvement Strategy 1: Make T&E Assessments More Accurate............. 35
Improvement Strategy 2: Increase Verification of Applicant Responses......42
Improvement Strategy 3: Use Assessments that Focus on
Present Proficiency...............................................................................48
Improvement Strategy 4: Evaluate Developments in T&E Assessment.......50
Chapter 5 - Conclusions and Recommendations............................................53
Conclusions.............................................................................................. 53
Recommendations..................................................................................... 55
APPENDIX A - Understanding Validity.........................................................57
APPENDIX B - Validity Coefficients............................................................59
A Report by the U.S. Merit Systems Protection Board
i
Executive Summary
Federal agencies are operating in a climate of scal austerity that has
constrained budgets, increased the demands on current sta, and reduced
opportunities for hiring. Although agencies must continue working towards
mission accomplishment, lling vacant positions—and molding the workforce
necessary for success—has become increasingly challenging. As such, it is
essential that agencies and hiring managers make wise decisions about how
to assess individuals’ qualications for any Federal jobs that they can ll.
is report discusses the advantages and disadvantages of using training and
experience (T&E) assessments to gauge an individual’s qualications for a job
and provides recommendations for the most eective use of such assessments.
Insights for this report were obtained from research and practice in the eld
of assessment, and from U.S. Merit Systems Protection Board (MSPB) survey
data. is report adds value by synthesizing multiple perspectives on the
advantages and challenges of T&E assessments and their implications for Federal
hiring. We discuss these resources together in one document to demonstrate
the role that T&E assessments can play in Federal hiring, highlight important
themes and conclusions about the usefulness of T&E assessments, and assist
Federal agencies and hiring managers in using T&E assessments appropriately.
Study Findings
Training and experience assessments are generally regarded as fair measures of job-
related abilities and they are widely used by Federal agencies. Further, there are
several T&E assessments that have practical value in Federal hiring. Training and
experience assessments that are considered useful for predicting an applicant’s future
job performance include direct questions for factual information, occupational
questionnaires, accomplishment records, and reference checks. Structured
interviews and biodata, which can include this type of information, are also useful
in gauging an applicant’s likelihood of success in a job.
However, some of the T&E assessments that are commonly used in Federal hiring
have poor ability to predict an applicant’s future job performance. ey include
tallies of the years of job experience applicants have, the number of training classes
they have attended, their grade point averages, and the contents of their resumes.
iiii
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Executive Summary
ese measures are weak proxies for true assessment of an individual’s capabilities,
or present prociencies, and should be avoided for hiring decisions.
Although there are several T&E assessments that can have value in Federal
hiring, there are challenges with relying on an applicant’s past to gauge his or her
current prociency or predict his or her future job success. First, trying to predict
future job behavior from what we are told about the past is dicult because:
• What an applicant reports is not necessarily what happened;
• Applicants may not learn effectively from past events; and
• Training and experiences in the past may not be applicable to the
present or future.
Second, there are challenges in how applicants assess their own prociencies.
Specically:
• Applicants may have trouble recalling or reporting the most appropriate or
applicable experiences;
• Applicants may not have an accurate perspective on the range of their
proficiencies; and
• Applicants may not accurately evaluate their proficiencies.
Finally, applicants are not always honest in reporting or evaluating their past training
or experiences.
Fortunately, our research shows there are strategies that can improve the usefulness
of T&E assessments. ese strategies focus on:
• Improving the accuracy of T&E assessments by grounding them in
job analysis, and improving T&E questions, rating scales, scoring,
and applicant self-assessment;
• Increasing the verification of applicant information by agency-
driven approaches that rely on corroborating applicant claims, and
through applicant-driven approaches such as warnings and requiring
documentation or elaboration of claims;
A Report by the U.S. Merit Systems Protection Board iii
Executive Summary
• Using present-oriented assessments that examine the abilities and
proficiencies that individuals currently possess; and
• Maintaining awareness of the latest developments in T&E assessment.
Conclusion
Training and experience assessments can bring value to a hiring or promotion process
if implemented appropriately. ere are advantages and disadvantages to using
such assessments, and each hiring manager’s threshold for the right balance will be
dierent. Whenever possible, agencies should augment T&E measures by adopting
one or more of the improvement strategies that are currently available. Further,
agencies should remain abreast of developments in T&E assessment. Training and
experience assessments, their improvement strategies, and future developments can
play a critical role in shaping the eectiveness of an agency’s overall assessment
approach, workforce composition, and ultimately mission success.
iviv
E J A: The Role of TRaining and expeRience in hiRing
Chapter 1 - Background
is study, which continues a series of studies focusing on Federal hiring and the
assessment of job applicants
1
, was conducted as part of the MSPB’s responsibility to
studies of Federal merit systems.
2
In its research, MSPB takes a Governmentwide
perspective, looking across the practices of numerous employment contexts in
Federal agencies with diverse missions and different cultures. e findings will be
most useful to agency decision-makers when adapted to the context of their own
workforce.
What is Assessment of Training and Experience?
is report examines how the previous learning and work-related experiences of job
applicants are evaluated by Federal employers for hiring and promotion decisions.
For the purposes of this report, training includes any structured experience
undertaken for the primary purpose of gaining knowledge or skill in a specific area.
Examples include training classes, college courses, webinars, individual study, and
other similar activities. Experience includes not only on-the-job work tasks, but also
any other activity that may convey job-relevant knowledge or skills or otherwise
enhance one’s ability to perform a particular job. In addition to official duties,
experience may be gained from volunteer activities, recreational pursuits, or other
non-work activities that develop abilities relevant to work.
Both training and experience can be evaluated, scored, and considered as factors
in hiring and promotion decisions. e various tools for doing this are called
training and experience (T&E) assessments. e fundamental assumption
behind all T&E assessments is that what we know in the present is shaped by what
we have experienced in the past.
3
is assumption is one aspect of the principle of
1
MSPB’s series of assessment reports includes: MSPB (2003). The Federal Selection Interview: Unrealized Potential.
MSPB: Washington, DC; MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington,
DC; MSPB (2005). The Probationary Period: A Critical Assessment Opportunity. MSPB: Washington, DC; MSPB (2006).
Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC; and MSPB (2009). Job simulations: Trying
out for a Federal job. MSPB: Washington, DC.
2
The Merit Systems Protection Board (MSPB) is an independent, quasi-judicial Federal agency that serves as the guardian
of Federal merit systems. MSPB’s responsibilities include conducting studies of Federal merit systems and the Federal workforce
to ensure that Federal employees are managed in accordance with the merit system principles and in a manner free from
prohibited personnel practices.
3
Weisbuch, M., Slepian, M., Clarke, A., Ambady, N., and Veenstra-Vanderweele, J. (2011). Behavioral stability across time
and situations: Nonverbal versus verbal consistency. Journal of Nonverbal Behavior, 34(1), 1-15.
A Report by the U.S. Merit Systems Protection Board 1
22
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 1 - Background
behavioral consistency: the best predictor of future performance is past performance.
Although this principle has proven reliable and useful across a variety of work tasks
and settings, it has its limitations and exceptions.
We note that much of the information in this report is not new, per se. Certainly,
readers can learn useful information about T&E assessments from a variety of
sources. However, this report is a synthesis of multiple research and practitioner
perspectives on the advantages and challenges of T&E assessments, and their
implications for Federal hiring. Our goal is to demonstrate the role that T&E
assessments can play in Federal hiring, highlight important themes and conclusions
about T&E usefulness, and therein assist Federal agencies and hiring managers in
using T&E assessments appropriately.
Advantages of training and experience assessments. Training and experience
assessments prove advantageous when measuring the competencies of job applicants.
For one, past experiences could appear to be more concrete than competencies,
which must be inferred from a person’s behaviors. Relying on past experiences
avoids this inference because they appear to reect actual competencies. In other
words, people seem likely to be able to perform a job—or one similar to it—if they
have done so in the past (experience). ey also seem likely to be successful at a
job if they have been told how to do it (training). For example, all applicants and
hiring ocials are familiar with resumes, which are basically lists of applicants’ past
experiences. Resumes can convey a wealth of information about an individual’s
qualications for a job, and are particularly convenient for managers because much
of the burden of preparation falls on applicants. ere are advantages for applicants,
too. eir job and training histories usually change slowly over time; once prepared,
a resume need only be updated periodically. Applicants and hiring managers alike
expect resumes to play a role in hiring. Most T&E assessments have at least a
supercial resemblance to resumes, inheriting this high degree of acceptance as part
of the hiring process. Commonly-used T&E assessments will be discussed in more
detail in Chapter 2.
Disadvantages of training and experience assessments. Despite their allure, research
has shown that there are challenges to using T&E measures. For example, many of
these measures have relatively poor track records predicting future job performance.
Some impose signicant time burdens on applicants. Many are less eective at
capturing skill levels accurately than measures that directly assess an individual’s
present prociency or future potential. Training and experience measures that require
applicants to evaluate their own experience present considerable opportunities
for applicants to unintentionally (or intentionally) misrepresent their prociency
in an area. Simply, applicants may not be the best judges of their job-related
qualications. Even honest applicants with good intentions may not do this well.
Evidence from both research and practice suggests that greater caution is appropriate
A Report by the U.S. Merit Systems Protection Board 3
Chapter 1 - Background
when using T&E measures to make important personnel management decisions.
ese challenges, possible strategies to address them, and general conclusions and
recommendations on the use of T&E assessments are discussed in more detail later
MSPB is studying T&E assessments as part of its mission to uphold the Merit System
Principles and support effective Federal workforce management. Additionally, T&E
assessments are widely used and well regarded, and hiring decisions made using
T&E assessments can have a lasting impact on the Federal workforce. Further,
given the austere economic climate in which Federal agencies currently operate—
and the resultant budget constraints and limited hiring opportunities—it is all the
more critical that hiring managers make wise decisions about which assessments
they use to bring in talent. As alluded to above, although T&E assessments can
be useful in Federal hiring, they also have disadvantages that require attention and
mitigation. Together, these factors make T&E assessments a prime candidate for
study.
Merit system principles. Assessment and selection of Federal employees is central
to MSPB’s mission to promote the management of the civil service in accordance
with the merit system principles.
4
e first principle requires that Federal employee
“…selection and advancement should be determined solely on the basis of relative
ability, knowledge and skills.” When a selecting official considers prior training
and experience as part of a hiring or promotion decision, the decision must be
made that best identifies true differences in ability as accurately as possible. No
assessment methods should be used which fail to meet professional testing standards
for validity.
5
e merit system principles further require that Federal employees be managed
efficiently and effectively, as well as fairly and equitably. When assessments are not
accurate in measuring abilities related to job performance, scores resulting from
them are less reliable and the quality of decisions made from such scores is reduced.
Such flawed decisions increase the likelihood of wasted resources, inefficient and
ineffective practices, unfair personnel decisions, and an erosion of individual and
organizational performance.
in this report.
Why Study Training and Experience Assessments?
4
Title 5 United States Code §2301 and §2302. The merit system principles and prohibited personnel practices referenced
in this report can be accessed on the web at http://mspb.gov/meritsystemsprinciples.htm and http://mspb.gov/ppp/ppp.htm.
5
Defined in the U.S. Office of Personnel Management’s (OPM) Assessment Decision Tool as “The extent to which the
assessment method has been shown to accurately measure a job-related competency and/or predict successful performance on
the job.” (Available at apps.opm.gov/ADT/content.aspx). The concept of validity will be discussed in more detail in Chapter 2,
pg. 10.
Chapter 1 - Background
Finally, it is also a merit system principle to educate and train employees when this
will result in better organizational or individual performance. While this report
does not focus directly on employee training, this principle is relevant because such
training, sometimes provided by agencies, is often considered part of the evidence
that an employee possesses certain proficiencies. is means that the relevance of
training to the job, the quality of the training, and the degree to which training
produces measurable learning collectively have an effect on the quality of a T&E
assessment which seeks to measure it. When training is ineffective or evaluated
ineffectively, error is introduced into the assessment of training and experience.
Such a situation further reduces the value of T&E assessments as measures of job-
relevant abilities.
Personnel assessment is an ongoing part of MSPB research because of the negative
consequences a bad hire can have on an agency. e human resources (HR)
community has long been aware of the damage a single individual can do through
instability, incompetence, or dishonesty.
6
ere are further costs in decreased
morale and productivity while the rest of the organization struggles to cope with
a bad hiring decision.
7
A previous MSPB study reported that the overall cost of
hiring the wrong person for a job can be up to three times the employee’s salary.
8
ese costs can be magnified in the Federal workforce if an agency is slow to address
a poorly-performing employee,
9
or supervisors are reluctant or unwilling to take
corrective action,
10
permitting substandard performance to continue.
11
Widely used, well-regarded. Training and experience assessments are used
extensively in Federal hiring.
12
For example, “A 1999 MSPB study found that about
60 percent of delegated examining unit (DEU) hires are assessed through [ratings
of] education and experience.”
13
T&E is highly preferred as an assessment method
when hiring Federal supervisors.
14
Further, “MSPB’s 2000 Merit Principles Survey
6
The costs associated with a bad hire are well known in the HR community; one organization’s web site includes a “Bad
Hire Calculator” that tallies them: www.adpselect-info.com/badHireCalculator.html.
7
See, for example, Jay Goltz, “The Hidden Cost of Bad Hiring,” The New York Times, March 1, 2011, available on the
web at boss.blogs.nytimes.com/2011/03/01/the-hidden-costs-of-bad-hiring/.
8
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC.
9
MSPB (2005). The Probationary Period: A Critical Assessment Opportunity. MSPB: Washington, DC.
10
MSPB (2010). A Call to Action: Improving First-Level Supervision of Federal Employees. MSPB: Washington, DC.
11
MSPB (2008). Attracting the Next Generation: A Look at Federal Entry-Level New Hires. MSPB: Washington, DC.
12
See, for example, Sharpe, P. & Sorensen, K. (2011). Assessment of training and experience. IPAC 2011, Washington,
DC. Also, MSPB (2003). The Federal Workforce for the 21st Century: Results of the Merit Principles Survey 2000. MSPB:
Washington, DC; and Partnership for Public Service (2004). Asking the wrong questions: A look at how the Federal
government assesses and selects its workforce. Partnership for Public Service: Washington, DC; and McDaniel, M., Schmidt,
F. & Hunter, J. (1988). A meta-analysis of the validity of methods for rating training and experience in personnel selection.
Personnel Psycholog y, 41, 283-314.
13
MSPB (1999). The Role of Delegated Examining Units: Hiring New Employees in a Decentralized Civil Service.
Washington, DC, p. 5.
14
MSPB (2010). A Call to Action: Improving First-Level Supervision of Federal Employees. MSPB: Washington, DC.
44
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 1 - Background
(MPS) asked supervisors what information they use in selection decisions. Almost
all—96 percent—reported using prior work experience to a great or moderate extent,
and 82 percent reported using level of education to a great or moderate extent.”
15
us, it appears that T&E measures are well-regarded by hiring managers.
16
A
more recent survey of Federal HR practitioners found similar results. Measures of
training and experience, such as resumes, occupational questionnaires, reference
checks, Knowledge, Skills, and Abilities (KSA) narratives, and educational records,
were among the most frequent assessments used in Federal hiring. In fact, the only
non-T&E assessment among the six most frequent assessments was the structured
interview.
17
Human resource practitioners report that training and experience
measures continue to be widely used across public, private, and nonprofit sectors.
18
T&E assessments are also used across a wide range of situations, from entry-level
assessment of basic qualifications to Executive Core Qualifications statements
written by applicants to the Senior Executive Service.
19
e preference for T&E assessments extends beyond Federal agencies. Job
applicants may desire that all of their accomplishments be reviewed and evaluated.
Some applicants may regard alternatives to this type of assessment, such as biodata
and tests of cognitive ability, as biased or unfair. Training and experience, which
acknowledge their work histories and achievements, may be more palatable. MSPB’s
study of fairness in Federal personnel practices notes that, “Education and training
and length of experience received wide acceptance as they are likely viewed as fairly
objective measures of an employee’s ability to perform on the job.”
20
Yet, this regard for T&E assessments is at variance with research that identifies
present-oriented assessments as more valid predictors of future job performance.
21
is report will discuss how such alternatives or focused improvements to T&E
15
MPS 2000, Question 77A.
16
For example, the 2000 Merit Principles Survey, which gathered information about the degree to which Federal
supervisors value information from different types of assessment tools when making hiring decisions, found that T&E measures
were well-regarded.
17
MSPB Fair and Open Competition Survey (See http://mspb.gov/studies/surveys.htm).
18
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ.
19
See www.opm.gov/policy-data-oversight/senior-executive-service/selection-process/#url=Process.
20
MSPB (2013). Preserving the integrity of the Federal Merit Systems: Understanding and addressing perceptions of
favoritism. MSPB: Washington, DC.
21
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.
A Report by the U.S. Merit Systems Protection Board 5
Chapter 1 - Background
measures can improve hiring across the Federal Government and how Federal
agencies, Federal employees, and the public they serve can benefit.
22
Lasting effects. Training and experience assessments can strongly shape who
enters the Federal workforce. at is because such assessments, particularly those
that can be implemented inexpensively, are most often used in the early stages of
hiring as part of assessing minimal qualifications
23
or as an early screen in a series
of assessments.
24
When properly documented with content validity evidence,
education- and experience-based minimum qualifications have been upheld as valid
and consistent with professional standards of assessment development.
25
However,
it is particularly important that initial assessments be as accurate as possible. is
is because decisions early in the hiring process that an applicant is “minimally
qualified” may be difficult to reverse later in the process, even when a more rigorous
assessment yields evidence that suggests an applicant has been misclassified.
Additionally, T&E assessments can affect an agency’s workforce composition
through unintentionally giving an advantage to internal applicants. is is a
particular concern for jobs requiring the use of specific equipment or software that
is not commonly used or available outside of an agency. Any internal employees
who have had experience with such equipment or software would appear to have
an immediate advantage over external candidates. Yet, both internal and external
hires bring value to an agency. For example, internal hiring can build institutional
knowledge and memory, and can provide employees with opportunities for growth
and advancement, while external hiring can bring in new skills and fresh perspectives.
e balance between internal and external hiring should be a conscious, strategic
decision—rather than an unintended consequence of the criteria and methods used
to assess job applicants.
Further, careless use (or misuse) of T&E assessments can have a lasting impact
on an agency’s reputation. It is a prohibited personnel practice to structure a job
competition to advantage or disadvantage a particular individual. Training and
experience assessments that focus too narrowly on a very specific set of training,
22
T&E assessments have also been used extensively in state and local government hiring. A review of best practices will
be useful to these efforts as well. See Johnson, J., Guffey, W. & Perry, R. (1980). When is a T and E rating valid? IPMAAC:
Boston, MA;IPMA-HR (2006). Recruitment and Selection Benchmarking. Alexandria, VA.
23
See OPM’s Operating Manual for Qualification Standards for General Schedule (GS) Positions retrievable from www.
opm.gov/qualifications. See also OPM’s Job Qualification System for Trades and Labor Occupations retrievable from www.
opm.gov/qualifications/x-118c.
24
For example, see Agarwal, S., Busciglio, H., Leaman, J., Simmons-Collins, T., Curtin, P. & Thompson, S. (2010).
Understanding the practical outcomes of self-report, task-based assessment questionnaires for assessing applicants for U.S.
Customs and Border Protection vacancies. SIOP, 2010.
25
Buster, M., Roth, P. & Bobko, P. (2009). A process for content validation of education and experience-based minimum
qualifications: An approach resulting in Federal court approval. Personnel Psycholog y, 58(3), 583-630.
66
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 1 - Background
education, and experience factors can be seen as inappropriately narrowing the field
of candidates, even if this was not the intent. As a result, some qualified individuals
may self-select out of the application process, believing it to be a waste of their time.
Given the relevance of T&E assessments to the merit system principles, their
widespread use, and their lasting effects on the Federal workforce, it is important
that Federal agencies understand how such assessments can be used most effectively.
Further, Federal budget constraints and limited hiring opportunities reinforce the
importance of agencies and hiring managers making wise decisions about assessments
and the hiring decisions that follow.
Study Methodology
Several sources of data and information were compiled for this report. ese sources
are of three general types: current best practices in measurement; Governmentwide
surveys; and study-specific surveys conducted by MSPB.
Best practices. Information about the relative validity (or usefulness) of T&E
measures was obtained from a review of the applied psychology literature as reported
in professional journals and paper presentations at meetings of professionals who
work in applied psychology. We have examined and referenced reports on T&E
assessments in both public and private sectors to identify techniques which improve
their effectiveness. Information about how T&E assessments are used by human
resources (HR) practitioners was obtained from HR professional associations and
through interviews with Federal HR and personnel specialists.
Governmentwide surveys. Data about the use of assessments in Federal agencies was
gathered from surveys conducted by MSPB. ese surveys are of two types. First,
MSPB periodically conducts Governmentwide Merit Principles Surveys (MPSs) of
a portion of the Federal workforce. ese surveys are conducted every three to
five years and collect responses from 40,000 to 50,000 participants. ese surveys
provide trend information about agency management practices and employee
perceptions of their work environment. Several MPSs have focused on the use
and value of assessments for selection and promotion. Details about sampling,
administration, and results of these surveys are reported in the pr
oject reports for
each survey.
Study-specic surveys. Second, MSPB also conducts study-specific surveys that
focus on topics such as fairness, management practices, and human resources (HR).
Several of these surveys provided data referenced in this report. Details of the design
and administration of these surveys are reported in the project reports referenced in
this report.
A Report by the U.S. Merit Systems Protection Board 7
88
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 1 - Background
In This Report
As part of MSPB’s study series on assessment, this report is written to help improve
assessment practices in the Federal Government.
26
Assessment can be improved
through strategies shown to increase their quality of T&E assessments or by using
alternative assessments. MSPB oers research-based steps for improvement.
Four chapters follow this introduction (Chapter 1):
• Chapter 2 reviews the common methods that are often used to
assess training and experience. These methods range from simplistic
counting of years of job experience to in-depth analyses of employee
accomplishments.
• Chapter 3 describes several factors that can reduce the ability of any
T&E measure to predict job performance, including a focus on the
past, unreliable applicant self-assessment, and applicant dishonesty.
• Chapter 4 suggests strategies to either improve the accuracy of
existing T&E assessments, or replace T&E assessments with
alternative methods.
• Chapter 5 outlines this report’s conclusions and recommendations
for using T&E assessment in the Federal workplace.
26
MSPB’s series of assessment reports includes: MSPB (2003). The Federal Selection Interview: Unrealized Potential.
MSPB: Washington, DC; MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington,
DC; MSPB (2005). The Probationary Period: A Critical Assessment Opportunity. MSPB: Washington, DC; MSPB (2006).
Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC; and MSPB (2009). Job simulations: Trying
out for a Federal job. MSPB: Washington, DC.
A Report by the U.S. Merit Systems Protection Board
9
Chapter 2 - How is Training and
Experience Assessed?
is chapter introduces common methods for assessing training and experience
and summarizes how they are administered and how well they predict future job
performance. e chapter examines T&E assessments and assessments using
T&E information which predict job performance reasonably well and then a small
number of problematic T&E assessments.
Determining Assessment Quality
MSPB has advised agencies to “[e]mploy rigorous assessment strategies that emphasize
selection quality, not just cost and speed. In particular, use assessment instruments
that have a relatively good ability to predict future performance.”
27
e quality
standards for assessments used for selection and promotion must be high for at least
two reasons. First, decisions made using them have a great impact on the lives and
careers of individuals and on the effectiveness of employing organizations. Second,
these decisions must sometimes withstand legal challenge. Both considerations
require that assessments conform to the best practices and standards of professional
assessment development.
Sources of standards. Fortunately, assessment of human capabilities is a well-
developed discipline. rough research and experience, the field has established a set
of professional standards for producing high quality assessments. MSPB guidance
is available from several assessment-related reports. MSPB has also produced a
report, Reforming Federal Hiring: Beyond Faster and Cheaper, that summarizes its
research and recommendations on assessment.
e Office of Personnel Management’s Assessment Decision Tool
28
summarizes much
of this information with an applied focus. It refers to the other sources listed below.
If your reading time is limited, this is probably the best single source.
e Department of Labor’s Testing and Assessment: An Employer’s Guide to Good
Practices (www.onetcenter.org/guides.html) is also written for practitioners, but
requires some background in personnel selection.
27
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC.
28
Available at apps.opm.gov/ADT/content.aspx.
1010
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
10
e Uniform Guidelines on Employee Selection Procedures (www.uniformguidelines.
com), contained in 29 CFR Part 1607, are somewhat more comprehensive.
Professional guidance intended for all sectors can be found in two documents. e
Standards for Educational and Psychological Testing (www.apa.org/science/standards.
html) was produced collaboratively by the American Psychological Association
(APA), the National Council for Measurement in Education (NCME), and the
American Educational Research Association (AERA). Finally, the Principles for
the Validation and Use of Personnel Selection Procedures (www.siop.org/_Principles/
principlesdefault.aspx) is maintained by the Society for Industrial-Organizational
Psychology (SIOP), the primary professional association for those who focus on
personnel selection.
Validity. In accordance with the Merit System Principles hiring managers are
responsible for selecting individuals for jobs based on their relative merits, knowledge,
skills, and abilities. Yet, it can be challenging to gauge an individual’s merits for a
particular job, or evaluate how likely it is that an individual will be a good performer
in that job. Fortunately, as discussed above, there is a professional discipline (and
standards) devoted to developing quality assessments of individuals’ knowledge, skills,
and abilities, and using such assessments to predict job performance. Also fortunate
is that there is a standard for determining how good, or useful, an assessment is for
a particular job. is standard is called “validity.” OPM’s Assessment Decision Tool
defines validity as, “e extent to which the assessment method has been shown to
accurately measure a job-related competency and/or predict successful performance
on the job.”
29
Validity can be reflected in a number that is called the “validity coefficient.” In
essence, the validity coefficient is a rule of thumb for gauging assessment usefulness
for predicting how someone will perform on a particular job. Just as an individual’s
credit score is a number used to reflect how likely a person is to be able to pay back
a loan, an assessment’s validity coefficient is a number that can be used to reflect the
likelihood that an applicant’s scores on a particular assessment can be used to predict
their future job performance. e process of determining a validity coefficient is
beyond the scope of this study.
30
For current purposes and subsequent chapters, it
is sufficient to keep in mind the guidelines in Table 1.
29
Assessment Decision Tool, OPM.
30
See Appendix A for a further discussion of validity and validity coefficients.
A Report by the U.S. Merit Systems Protection Board 11
Chapter 2 - How is Training and Experience Assessed?
Table 1: Range of Values for Validity Coefficients
31
Validity Coeff icient Value Usefulness for Predicting Job Performance
.35 to 1.0 Very Useful
.21 to .35 Likely to be Useful
.11 to .21 Probably Useful
.11 or below Unlikely to be Useful
It is also important to keep in mind that the validity coefcient can never predict
perfectly who is (or is not) likely to succeed in a job.
32
For example, a person’s
ability as measured at the time they are hired is only one factor that inuences their
future job performance. A person’s future job performance will also be aected
by their motivation, engagement, quality of supervision, organizational support of
performance, and other contextual factors. Measuring an individual’s performance
is also contaminated by factors such as favoritism, performance rating ination, and
idiosyncratic performance standards.
Agencies and practitioners should consider the validity coecients of T&E
assessments when deciding which are most appropriate for each hiring decision.
Organizational eectiveness research demonstrates that good personnel selection
procedures, made possible through the use of high quality assessments, contribute
measurably to organizational performance.
33
We will place each type of
assessment into one of the categories of usefulness in Table 1.
34
Such evidence
can assist agencies and hiring managers in understanding the usefulness of a
particular assessment for predicting applicant’s likelihood of success on a job.
In light of the above discussion of assessment validity, the following sections
provide an overview of common T&E assessments. e goal is to provide a general
understanding of the overall form and usefulness of each assessment for identifying
applicants who are likely to be successful in particular jobs. ese assessments
31
Sproule, C. (2009). Rationale and research evidence supporting the use of content validation in personnel assessment.
IPAC. Available at www.ipacweb.org.
32
In other words, we cannot expect validity coefficients to approach 1.0 for many reasons which have nothing to do with
assessment quality. See Schmidt, F., Caplan J., Bemis, S., Decuir, R., Dunn, L. & Antone, L. (1979). The behavioral consistency
method of unassembled examining (TM-79-21). Washington, DC: U.S. Office of Personnel Management, Personnel Research
and Development Center.
33
Van Iddekinge, C., Ferris, G., Perryman, A., Blass, F. & Heetderks, T. (2009). Effects of selection and training on unit-
level performance over time: A latent growth modeling approach. Journal of Applied Psycholog y, 94(4), 829-843.
34
Appendix B contains the numeric validity coefficients used to place each specific type of T&E assessment into the
appropriate category in Table 1.
1212
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
may be used in a variety of contexts—to evaluate minimum qualications, measure
entry level skills, distinguish between candidates later in the assessment process, or
determine eligibility for promotion.
Useful T&E Assessments
T&E assessments that have proved useful in predicting future job performance
include direct questions, occupational questionnaires, accomplishment records,
KSA narratives, and reference checks.
Direct questions. Applications for Federal jobs can ask for factual information, such
as college major, time in grade, credit hours completed, degrees or certications
held, past or present employment with the Federal government or with a particular
agency, veteran’s status, or whether the applicant meets some standard. Response
options for such questions are typically “Yes/No” or multiple-choice. is approach
is often encountered in assessing minimum qualications.
35
For example: “is
GS-14 position requires that you have at least one year of service at the GS-13 level
or equivalent. Do you meet this requirement?” Or, “is position requires that you
have earned at least 12 credits in computer science or a related eld at an accredited
college or university. Do you meet this requirement?”
Given the wide range of direct questions that could be asked, we cannot report one
validity coecient for this method. Yet, the usefulness will depend on whether the
information requested has been shown by job analysis to be related to success on
the job, whether the applicant supplies the requested information, and whether the
information is conrmed to be accurate.
As an aside, it is a good idea to list such questions in a vacancy announcement so
applicants can read through them and screen themselves out if they do not meet
the criteria described. is kind of self-selection is good for everyone—it reduces
the assessment burden on the agency to process applications from unqualied
applicants and reduces wasted eort on the part of applicants as well. However, this
only works if the question’s meaning is clear and seeks information that is veriable.
For example, it would not be a good idea to present a densely-written paragraph
of job qualications and ask applicants: “Do you meet these qualications?” Not
only is this overwhelming, but it requires applicants to make judgments they may
not know how to make.
Occupational questionnaires. Occupational questionnaires bridge the “Possibly
Useful” and “Likely Useful” categories for predicting future job performance (see
Table 1). Validity coecients for occupational questionnaires have ranged from .15 to
35
OPM Delegated Examining Operations Handbook available at www.opm.gov/deu/.
A Report by the U.S. Merit Systems Protection Board 13
Chapter 2 - How is Training and Experience Assessed?
.28, making the quality of the particular questionnaire a crucial factor in determining
its usefulness.
36
A typical occupational questionnaire presents a list of specic job
tasks or behaviors and asks applicants to rate their level of experience performing
each. e tasks (items) are often developed to measure specic competencies or
KSAs. ey are job related and generally easy for experienced workers to rate.
Dierent questionnaires vary in what information they ask applicants to provide
about each task. Common strategies are to ask applicants whether or not they have
ever performed the task, how often they have performed it, how much time they
have spent performing it, how eectively they have performed it, whether or not
they received training on how to perform it, how closely they were supervised, and
whether they have ever taught it to others.
37
Occupational questionnaires have been used extensively to quantify an applicant's
degree of experience with specic tasks performed or abilities (competencies)
required on the job and likely performance on a similar job in the future.
38
In 2006,
one study found that 70 percent of public sector organizations (local, state and
Federal) reported using questionnaire-based T&E ratings as one of their three most
frequently-used assessments.
39
More recent discussions note that such questionnaires
are used extensively by Federal agencies, particularly those employing automated
stang systems.
40
eir use has increased since hiring reform guidelines issued
by the U.S. Oce of Personnel Management
41
have removed KSA narratives and
similar written assessments from the initial screening of applicants.
Occupational questionnaires are best used to screen out poor candidates rather than
to distinguish between the best candidates. In particular, they are most helpful in
screening a large applicant pool representing a broad range of dierences in ability
36
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274; and Schneider, R. (1984). The
rating of training and experience: A review of the literature and recommendations on use of alternative E&E procedures.
Personal Assessment Monographs, 3(1), IPMAAC: Alexandria, VA.; and McDaniel, M., Schmidt, F. & Hunter, J. (1988). A meta-
analysis of the validity of methods for rating training and experience in personnel selection. Personnel Psycholog y, 41, 283-314.
37
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ.
38
Quinones, M., Ford, K. & Teachout, M. (1995). The relationship between work experience and job performance: A
conceptual and meta-analytic review. Personnel Psycholog y, 48, 887-910.
39
IPMA-HR (2006). Recruitment and Selection Benchmarking. Alexandria, VA.
40
Barton, M. (2010). Automated T&E questionnaires: Practical outcomes and development considerations. SIOP.
41
See www.opm.gov/policy-data-oversight/human-capital-management/hiring-reform/.
1414
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
rather than in assessing a more experienced pool of well qualied candidates.
42
ey
are therefore useful in the rst stage of multiple hurdle assessment plan.
43
Applicant ratings on occupational questionnaires are scored using question and/or
competency weighting strategies developed during job analysis. Some questions
or competencies may be weighted more heavily depending on the criticality of the
tasks/competencies to the job, or abilities that are dicult to acquire or rarely found
in the applicant pool. But the scores also depend on the judgment of the applicants
themselves as they decide how to translate their experiences into the language and
scales of the questionnaire.
Over time, research and practice have made improvements to occupational
questionnaires. ere is some evidence that future performance can be better
predicted when questions address how frequently tasks are performed rather than
the total amount of time spent performing them. Predictive power also increases
when assessments ask about more specic tasks or competencies that are closely job-
related than about general types of tasks or competencies.
44
ey are more eective
for predicting performance in professional/administrative/scientic positions than
in clerical/technical positions.
45
e U.S. Oce of Personnel Management (OPM) has a long history of using
occupational questionnaires, ranging from the initial use of scanned paper
forms to the current web-based questionnaires.
46
OPM has worked with other
agencies to rene and disseminate best practices for using these assessments.
Accomplishment records. Accomplishment records are in the “Very Useful” category
for predicting future job performance (see Table 1). Accomplishment records achieve
42
Agarwal, S., Busciglio, H., Leaman, J., Simmons-Collins, T., Curtin, P. & Thompson, S. (2010). Understanding the
practical outcomes of self-report, task-based assessment questionnaires for assessing applicants for U.S. Customs and Border
Protection vacancies. SIOP, 2010; Barton, M. (2010). Automated T&E questionnaires: Practical outcomes and development
considerations. SIOP; and McDaniel, M., Schmidt, F. & Hunter, J. (1988). A meta-analysis of the validity of methods for
rating training and experience in personnel selection. Personnel Psycholog y, 41, 283-314.)
43
Sproule, C. (2009). Rationale and research evidence supporting the use of content validation in personnel assessment.
IPAC. (Available at www.ipacweb.org.)
44
Quinones, M., Ford, K. & Teachout, M. (1995). The relationship between work experience and job performance: A
conceptual and meta-analytic review. Personnel Psycholog y, 48, 887-910.
45
Agarwal, S., Busciglio, H., Leaman, J., Simmons-Collins, T., Curtin, P. & Thompson, S. (2010). Understanding the
practical outcomes of self-report, task-based assessment questionnaires for assessing applicants for U.S. Customs and Border
Protection vacancies. SIOP; and Barton, M. (2010). Automated T&E questionnaires: Practical outcomes and development
considerations. SIOP.
46
OPM (2009). Developing and Administering Automated Task- and Competency-Based Questionnaires. Also Barton,
M., Bisges, J. & Holloway-Lundy, A. (2010). Practical outcomes associated with use of automated questionnaires at the U.S.
Office of Personnel Management. SIOP 2010.
A Report by the U.S. Merit Systems Protection Board 15
Chapter 2 - How is Training and Experience Assessed?
validities of .45, higher than for any other T&E measure.
47
“e accomplishment
record is a means of gathering self-reported and veriable descriptions of experience
on relevant behavioral job dimensions.”
48
It was developed to collect information
from professionals who sometimes do not respond favorably to more test-like
assessments.
49
Applicants provide detailed, written descriptions of a small number
of accomplishments that they choose to best illustrate their prociency level on
dimensions identied as important by a job analysis. ese accomplishments need
not be from the work setting, but may be from other areas of the applicant’s life.
Applicants can be asked to provide supplemental information about the context of
the accomplishment, the steps taken to accomplish it, the resources used, their share
of responsibility for the work, and the product or outcome.
One key to this method’s high validity is that the information applicants provide is
veriable. ey are asked to provide names and contact information for individuals
who can verify that their description of the accomplishment is accurate. Another
important factor is that scoring is based on information about successful job
performance gathered during job analysis.
However, accomplishment records are expensive to develop. e accomplishment
record is sometimes used later in a multiple hurdle selection process as a verication
method for an earlier occupational questionnaire because the scoring process for
accomplishment records can be more labor intensive than these other methods.
50
Past work samples, such as reports applicants present as their work, are a variation of
the accomplishment record; there is no need to have the applicant write a description,
since the product of the original work is available for evaluation. But this product
is of uncertain parentage without further information about the applicant’s role in
producing it. And past work samples from dierent applicants may not be easy to
compare. As with accomplishment records, contact information can be collected
from the applicant and be used to independently verify the applicant’s level of
involvement in producing the work sample.
47
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), p. 268.
48
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ.
49
Hough, L. M. (1984). Development and evaluation of the “accomplishment record” method of selecting and promoting
professionals. Journal of Applied Psycholog y, 69, 135-146.
50
Current OPM hiring reform guidance also precludes the use of the accomplishment record and other assessments
requiring written narratives as part of the initial screen in Federal hiring.
1616
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
KSA narratives. e KSA narratives are another variation of the accomplishment
record.
51
KSA narratives and accomplishment records are similar assessments,
diering largely in the degree of structure in development, presentation, and
scoring.
52
A KSA question presents a competency, job, or job task and asks
applicants to write a narrative evidencing that they have this competency or can do
this job or task. When a job is broken down into these pieces, it becomes easier for
applicants to highlight their best example for each KSA instead of selecting a single
accomplishment which may not demonstrate their full strength in every area critical
to the job. KSA questions are often no more than a sentence or two, and applicants
are encouraged to show brevity in their replies, which does not permit the in-depth
analysis that a longer accomplishment record narrative typically permits. Applicants
are typically not asked for a reference for each KSA, reducing the possibility that an
ocial can later conrm the accuracy of the applicant’s assertions.
Although the use of KSA narratives permits an agency to identify multiple applicant
capabilities, KSA narratives require a signicant time commitment from applicants
to write and an equal or greater commitment from agencies to score. MSPB has
long been concerned about this applicant burden: KSAs “can be time-intensive
and burdensome to complete. In addition, they can be alien and discouraging to
applicants because private sector organizations do not generally require these types
of submissions.”
53
is burden extends to selecting ocials as well: “In a 2004
OPM survey, almost 50 percent of agency respondents said that a key barrier to
timely hiring is the amount of time spent by selecting ocials in reviewing applicant
credentials and conducting interviews.”
54
In 2010, a signicant portion of this burden was lifted. President Obama signed
an Executive Order to remove KSA narratives from the initial stages of screening
for Federal jobs. OPM Director John Berry praised this change for introducing
“commonsense hiring” which would allow selections to proceed “eciently and
quickly.”
55
However, in some agencies the burden has been eased only slightly,
as KSA narratives were simply moved later in the hiring process. Other agencies
dropped KSA narratives in favor of less burdensome occupational questionnaires for
initial employment screening.
51
No validity coefficient is available for KSA narratives.
52
Bradley, K. (2011). Testing and Assessment Methods for Public Sector Hiring: Supplementing (or Supplanting?) T&E
Evaluations. IPAC.
53
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC.
54
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC, which cites United
States House of Representatives, Statement of the Honorable Dan G. Blair, Deputy Director, Office of Personnel Management,
Before the Subcommittee on Civil Service and Agency Organization, Committee on Government Reform, Chicago, IL, June 7,
2004.
55
“Remarks of OPM Director John Berry, Hiring Reform First Anniversary Event”, May 18, 2011. Available at www.opm.
gov/news/speeches-remarks/hiring-reform-first-anniversary-event/.
A Report by the U.S. Merit Systems Protection Board 17
Chapter 2 - How is Training and Experience Assessed?
ere is some indication that the impact of reducing applicant burden may not be
entirely positive. Many HR professionals report an increase in application volume
that is not necessarily accompanied by an increase in qualied applicants.
56
More
individuals apply for Federal jobs who may be less serious about obtaining those
jobs. eir application materials must still be processed and evaluated, increasing
agency burden signicantly.
Reference checks. Reference checks have an average validity coecient of .26,
57
and
are therefore in the “Likely Useful” category for predicting future job performance
(see Table 1). A reference check is a structured discussion between a potential
employer and those who are acquainted with an applicant’s previous work and job-
relevant behavior. OPM denes it similarly: “Reference checking is an objective
evaluation of an applicant’s past job performance based on information collected
from key individuals (e.g., supervisors, peers, subordinates) who have known and
worked with the applicant.” An MSPB survey of Federal HR professionals found
that reference checks play a role in nearly 80 percent of Federal hiring processes.
58
MSPB has discussed reference checking extensively in the report Reference Checking
in Federal Hiring: Making the Call.
59
Some T&E assessments suer from a lack of verication. Reference checks are at
the opposite end of the spectrum. ey are primarily verication of previously-
obtained information (often from a resume) using the applicant’s professional
network. Such verication allows reference checks to add to the usefulness of other
assessment methods, such as resumes, which will be discussed below. One of the
best features of reference checks is that they are typically performed at the end of the
assessment process when management has narrowed the applicant pool to a very few
job candidates. us, the investment of time is focused on only those individuals to
whom the agency expects to make a job oer. e ability to conrm information
given by candidates earlier in the process is an ample return on the investment of
time in checking a few references.
Assessments that Include T&E Information
e T&E assessments discussed in the previous section all focus on gauging an
individual’s current prociency from their experience gained and abilities developed
at some time in the past. ere are assessments which gather the same type of
information about the past, but combine it with other information to produce a
56
MSPB Fair and Open Competition Survey (See http://mspb.gov/studies/surveys.htm).
57
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.
58
MSPB Fair and Open Competition Survey (See http://mspb.gov/studies/surveys.htm).
59
MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington, DC.
1818
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
score. is section briey describes two such measures: structured interviews and
biodata questionnaires.
Structured interviews. is assessment has a predictive validity of .51,
60
and is
therefore in the “Very Useful” category for predicting future job performance (see
Table 1). Key characteristics of structured interviews include: each applicant is
asked the same set of questions in the same order; each question is linked to a job-
relevant skill or competency identied through job analysis; and applicant responses
are evaluated using the same scoring scale and procedure.
Structured interviews use two basic types of questions. Past-focused questions require
applicants to describe their performance in a job-relevant situation from their past
experience. For example, a potential past-focused structured interview question
for the competency interpersonal skills could be, “Describe a time when you had
to work with an uncooperative coworker on a project. What was the situation?
What did you do, and why? What was the outcome?” ey share the challenges
of T&E assessments in general (which are reviewed in the next chapter). eir
advantage, in addition to having a high predictive validity, is the opportunity for
a live demonstration of performance by the applicant, and a chance to clarify any
ambiguities in the questions, albeit in a structured manner.
Present-oriented structured interview questions, on the other hand, ask applicants
to demonstrate their current prociency by describing what they would do in a
hypothetical situation. For example, a potential present-oriented structured
interview question for the competency interpersonal skills could be, “Describe
what you would do if a colleague was dominating the discussion in sta meeting,
not leaving others time to talk? What actions would you take and why? What
outcome would you expect?” ese questions are not T&E measures. Although
applicants may draw from past experiences when responding to such questions,
present-oriented structured interview questions ask applicants to perform in the
present rather than just report on their past. ere are a dierent set of issues to be
considered and best practices for present-oriented assessments.
Structured interviews are exible, easy to develop, and widely used in both private
and public sector hiring. MSPB’s report, e Federal Hiring Interview: Unrealized
Potential, reviews best practices for structured interviews.
61
e Oce of Personnel
60
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY, p. 493.
61
MSPB (2003). The Federal Selection Interview: Unrealized Potential. MSPB: Washington, DC.
A Report by the U.S. Merit Systems Protection Board 19
Chapter 2 - How is Training and Experience Assessed?
Management also oers training
62
and guidance
63
to assist hiring managers in using
structured interviews eectively.
Validity studies have not distinguished between past- and present-oriented structured
interview questions because they are often used together in interviews. For this
reason it is not possible to report a separate validity coecient for past-focused
structured interviews.
Biodata. Biodata measures bridge the “Likely Useful” and “Very Useful” categories
for predicting future job performance (see Table 1). ey achieve validities in the
vicinity of .35. Some questions on a biodata questionnaire ask applicants to “…
recall and report their typical behaviors or experiences in a situation likely to have
occurred earlier in their lives.”
64
ese situations include work experiences, but may
also ask about volunteer and relevant non-work experiences that are indicators of
abilities relevant to work performance. Applicant behavior in these past situations
should be observable, veriable, and under the person’s control rather than just
something that happened to them.
65
ese biodata questions are similar to past-
focused structured interview questions, except that they are administered in
questionnaire form with response options without the opportunity to interact with
the questioner.
Biodata instruments are not pure T&E assessments because these past-focused
questions are not the only kind of question they contain. Some biodata questions
address attitudes, motivation, interests, and personality traits.
66
Applicants have
mixed reactions to biodata measures, some nding the questions intrusive and
inappropriate. Some Federal hiring programs have used biodata measures in the
past,
67
but such a decision must be made cautiously. Complexities in development,
scoring, and applicant reactions require professional expertise and thorough eld
testing before deploying a biodata instrument.
62
See www.opm.gov/services-for-agencies/assessment-evaluation/structured-interview/ and www.hru.gov for structured
interview training developed by OPM.
63
See www.opm.gov/policy-data-oversight/assessment-and-selection/structuredinterviews/ for OPM guidance on the use
of structured interviews.
64
Mumford, M., Whetzel, D., Murphy, S. & Eubanks, D. (2007). Background data. In D. Whetzel and G. Wheaton
(Eds.), Applied Measurement: Industrial Psychology in Human Resources Management (pp. 161-180). Lawrence Erlbaum
Associates: Mahwah, NJ.
65
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY, p. 512; Mael, F.
(1991). A conceptual rationale for the domain and attributes of biodata items. Personnel Psycholog y, 44, 763-792.
66
Mumford, M. (1999). Construct validity and background data: Issues, abuses, and future directions. Human Resource
Management Review, 9(2), 117-145.
67
Gandy, J., Dye, D., & MacLane, C. (1994). Federal government selection: The individual achievement record. In
H. Stokes, M. Mumford & W. Owens (Eds.), Biodata handbook: Theory, research, and use of biographical information in selection and
performance prediction. (pp. 275-309). CPP Books: Palo Alto, CA.
2020
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
Problematic T&E Assessments
Several measures of training and experience are widely accepted because they are
easy to use and appear to reect applicant possession of work-related competencies.
Although not an exhaustive list, examples include tallies of years of experience,
number of training classes attended, class rank or grade point average, and stand-
alone resumes. Despite their intuitive appeal, research has demonstrated that such
measures have limited usefulness in predicting job performance. ey are reviewed
here to ensure awareness of their shortcomings and to highlight potential “red ags”
that decision makers should look for when evaluating the quality of any assessment.
Where such methods and red ags appear in hiring processes, consideration should
be given to replacing or supplementing them with higher-quality measures.
Years of job experience. With an average validity coecient of .18, years of experience
is in the “Possibly Useful” category for predicting future job performance (see Table
1). ere is an intuitive appeal to the idea that the length of time an employee
has spent doing a certain kind of work can be used as a measure of the employee’s
skill in doing such work. It can be tempting to translate this directly into a T&E
assessment by creating a tally and treating it as a score. It is quick and easy. And,
there is some indication that length of experience may predict job success somewhat
more accurately when all applicants have less than three years of experience—when
all experience falls within the initial steep portion of the on-the-job learning curve.
However, after the rst two or three of years of work, the value of years of experience
has little practical value.
68
Overall though, the amount of time someone has spent working does not predict
future job performance very well. Using year count as an assessment requires
making fallacious assumptions such as that more experience automatically produces
an increase in competence and performance and that all people develop at more or
less the same steady rate. As we will discuss in the next chapter, there are reasons to
doubt that such assumptions are true.
Number of training classes attended. ere is no validity evidence indicating
that the number of training courses attended is useful in predicting future job
performance. Mere presence in a course is no guarantee of learning, and therefore
awarding points for attending such classes is a poor substitute for evaluating actual
learning. Training classes rarely have good end-of-course assessment that might
provide evidence of how much learning has occurred.
69
Even when they exist,
68
McDaniel, M., Schmidt, F. & Hunter, J. (1988). A meta-analysis of the validity of methods for rating training and
experience in personnel selection. Personnel Psycholog y, 41, 283-314.
69
Arthur, W., Bennett, W., Edens, P. & Bell, S. (2003). Effectiveness of training in organizations: A meta-analysis of
design and evaluation features. Journal of Applied Psycholog y, 88(3), 234-245.
A Report by the U.S. Merit Systems Protection Board 21
Chapter 2 - How is Training and Experience Assessed?
such assessment scores are almost never available to reviewers of job applications.
70
Further, some competencies cannot be eectively developed through training. For
such competencies, training classes should especially not be used as a proxy for
applicant prociency.
71
Overall, MSPB maintains that, “Selection ocials should avoid using training and
experience measures that credit applicants for competencies on the basis of lists
of classes they have attended. Such measures are particularly poor evidence that
applicants have acquired less trainable competencies.”
72
In doing so, MSPB concurs
with OPM’s conclusion: “Past behaviors, not past exposure, are the best predictors
of future behaviors.”
73
Transcripts and GPA. Grade point average is typically computed using a 0 to 4.0
scale. On the surface, GPA seems like it should predict job performance reasonably
well. GPA is easy to obtain, understand, and verify, and it seems to reect long-
term, sustained performance. ere is overlap between the skills needed to perform
well in school and on the job; the level of motivation and conscientiousness required
for school performance seems likely to transfer to job performance; and grades in a
major or area of concentration are reasonable indicators of the level of knowledge in
that area.
74
Applicants who have received instruction in a particular area require less
training on the job. But that relationship only holds for a year or two following the
applicant’s educational experience.
Yet, despite its intuitive appeal, research places overall GPA predictive validity around
.17.
75
erefore, GPA is in the “Possibly Useful” category for predicting future job
performance (see Table 1). A host of factors make GPA an unreliable measure.
ey include dissimilarity of school environments and work environments; dierent
70
Even when end-of-course assessments are conducted, they suffer from data quality problems. Analysis of survey
responses to the MSPB 2010 indicates that there are many non-training reasons that people attend training that have little to do
with learning. Some are unjustified, such as travel to a desired location or avoiding unpleasant work assignments. Others serve
a legitimate purpose, such as evaluating the training, preparing to teach it, or networking with other participants. These and
other factors affect the scores on end-of-training assessments, undermining their usefulness on the few occasions when they are
used.
71
MSPB (2010). Making the Right Connections: Targeting the Best Competencies for Training. MSPB: Washington, DC.
72
MSPB (2010). Making the Right Connections: Targeting the Best Competencies for Training. MSPB: Washington, DC,
p. 43.
73
Schmidt, F., Caplan J., Bemis, S., Decuir, R., Dunn, L. & Antone, L. (1979). The behavioral consistency method
of unassembled examining (TM-79-21). Washington, DC: U.S. Office of Personnel Management, Personnel Research and
Development Center.
74
Baird, L. (1983). The role of academic ability in high-level accomplishment and general success. College Board Report
82-6. College Entrance Examination Board: New York, NY.
75
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.
2222
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
grading policies; grade ination; student ability level; self-selection out of dicult
classes; and dierences in how GPAs are calculated.
76
Further, research has found
that the predictive validity is not only low, but seems to have declined over the last
two decades.
77
A large meta-analysis of studies relating GPA to a number of work
performance measures found little predictive validity for high school and college
GPA—although grades in classes taken post college degree (e.g., graduate school)
had higher predictive validity.
78
ere is some indication that academic achievement
is better at predicting training success on the job than work performance. ere
is also some indication that under some specialized circumstances, GPA can be
predictive of job performance. Overall though, the accumulated evidence suggests
that GPA is not as useful as many Federal hiring managers believe it to be.
79
Resumes. Resumes, “…typically describe the major duties associated with each
position in the [individual’s] work history, specic coursework, special skills and
credentials, and other qualications.”
80
ere are some common expectations about
what appears on a resume, but much is left up to the applicant.
Resumes are one of the most widely-used pre-employment assessments in private,
public, and nonprot sectors. Potential employers are greatly inuenced by
resumes, including aspects such as extracurricular activities and visual appeal which
have little relevance to future job performance.
81
MSPB found in 2000 that 96
percent of Federal supervisors reported using work history to a great or moderate
extent when making hiring decisions.
82
A recent survey of Federal HR practitioners
found that more than 90 percent of hiring decisions were based, to some degree, on
information from applicants’ resumes.
83
76
Schneider, R. (1984). The rating of experience and training: A review of the literature and recommendations on the use
of alternative E & E procedures. IPMAAC Personnel Assessment Monograph.
77
Roth, P. & Shippmann, J. (1996). Meta-analyzing the relationship between grades and job performance. Journal of
Applied Psycholog y, 81(5), 548-556.
78
Bretz, R. (1989). College grade point average as a predictor of adult success: A meta-analytical review and some
additional evidence. Public Personnel Management, 18(1), 11-22.
79
MPS 2000, Question 77A.
80
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ.
81
Chen, C., Huang, Y. & Lee, M. (2011). Test of a model linking applicant resume information and hiring
recommendations. International Journal of Selection and Assessment, 19(4), 374-387.
82
MPS 2000, Question 77A
83
MSPB Fair and Open Competition Survey (See http://mspb.gov/studies/surveys.htm).
A Report by the U.S. Merit Systems Protection Board 23
Chapter 2 - How is Training and Experience Assessed?
However, resumes have an average predictive validity of only .11,
84
which means they
bridge the “Unlikely to be Useful” and “Possibly Useful,” categories for predicting
future job performance (see Table 1). e validity of resumes depends not only on
quality of information provided, but also on how a set of resumes is evaluated and
scored. ey are often scored using the holistic method. e reviewer looks through
a stack of resumes and forms a general impression of each applicant. Unsurprisingly,
the holistic method is an ineective way of identifying applicants who perform well
on the job. In an unstructured evaluation, such as reviewing a resume without
guidelines, people adopt informal strategies that produce poor results. For example,
there is a “more is good” eect in which they unconsciously give more credit for
resumes that provide a lot of detail—even if much of it is irrelevant.
85
Resumes can also be scored using a point method which gives credit for various
indicators of experience. Point methods tend to focus attention on quantity of
experience at the expense of the quality of that experience.
86
OPM notes that
resumes have a “…particularly weak ability to predict job performance when
scoring method gives credit for factors such as: length and recency of education,
academic achievement, and extracurricular activities.”
87
Overall, job applicants are wise to craft their resumes carefully; hiring managers
would be wise to give them less weight. It is useful to consider resumes (and
application forms, which are basically standardized resumes), not as poor T&E
assessments, but as incomplete T&E assessments. While the unveried information
in a stack of resumes does not predict future performance, it does contribute to such
prediction when combined with a reference check that conrms or contradicts the
information.
Indicators of a low quality T&E assessment. e aforementioned examples of
problematic T&E assessments were included to highlight points of caution in T&E
assessments. is was not an exhaustive list of problematic T&E assessments; decision
makers will need to exercise wise judgment in determining the appropriateness of
a given T&E assessment for their particular hiring needs or situation. ey are
responsible for identifying the disadvantages of using any particular T&E assessment,
and for weighing such information against any perceived advantages. As discussed
above, red ags which warrant extra consideration include:
84
Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.
85
Highhouse, S. (1997). Understanding and improving job-finalist choice: The relevance of behavioral decision research.
Human Resource Management Review, 74, 449-470.
86
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ.
87
OPM, Assessment Decision Tool.
2424
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 2 - How is Training and Experience Assessed?
• Low or Unreported Validity. Hiring ocials and HR professionals should
be cautious about any assessment which has not been subjected to a
validation procedure to determine how well it predicts job performance.
Too often measures of training and experience may appear rational and
valid, but they have not been put to the test. e best approach is to
avoid the use of any such measures until their validity is established.
Whether or not they are eventually shown to produce valid results, their
use can be dicult to defend in the face of a legal challenge without
validity evidence in hand.
88
• Exclusive Focus on the Passage of Time. Practitioners should particularly
avoid any methods which create a score based solely on the passage of
time. ere is no assurance that time or attendance indicates learning.
Consequently, such measures are unlikely to be good predictors of job
performance. Measures based on length of experience are pervasive in
minimum qualications used as the rst hurdle in much of Federal hiring.
When such measures must be part of a hiring process, it is important to
check the results with a more accurate measure later in the assessment
process.
• Exclusive Focus on the Amount of T&E. Using measures which value the
amount of training and experience with little attention to whether it
resulted in any improved ability to perform work can result in hiring less
qualied applicants than those who may have spent less time in class but
have stronger abilities. Similarly, promising entry-level applicants can
be overlooked because their potential to learn new skills quickly is not
considered or because they learned from life experiences that a narrowly-
targeted training-focused T&E measure may not recognize.
Summary
is chapter reviewed both useful and problematic T&E assessments commonly
used in Federal hiring and promotion decisions. Some, such as occupational
questionnaires and accomplishment records, can contribute to hiring decisions.
Similarly, structured interviews and biodata, which can include T&E information,
88
See Albemarle Paper Co. v. Moody, 422 U.S. 405 (1975). In Albemarle, the employer used two different tests to assess
employees for promotion, but did not establish at the time of use what connection, if any, there was between the tests and the
jobs at issue. Id., at 410-11. “[O]n the eve of trial, Albemarle engaged an industrial psychologist to study the ‘job relatedness’ of
its testing program.” Id. at 411. The Supreme Court noted that “It cannot escape notice that Albemarle’s study was conducted
by plant officials, without neutral, on-the-scene oversight, at a time when this litigation was about to come to trial. Studies so
closely controlled by an interested party in litigation must be examined with great care.” Id. at n. 32. Ultimately, Albemarle lost
the case because it did not adequately prove the tests it used were related to the jobs for which they were used. Id. at 435-36.
A Report by the U.S. Merit Systems Protection Board 25
Chapter 2 - How is Training and Experience Assessed?
are also useful in gauging an applicant’s likelihood of success in a job. However other
T&E assessments, like years of work experience or training class attendance,provide
little reliable information about an applicant’s abilities and should be replaced or
supplemented by higher-validity measures.
While each method has its own prole of strengths and weaknesses, there are some
challenges common to all T&E assessments. e next chapter reviews several of
these challenges.
2626
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 3 - What Are the Common
Challenges?
e T&E assessment methods reviewed in the last chapter differ in a number of
ways, including their overall quality, what they require of applicants, how they are
scored, and the level of resources necessary to develop and administer them. But
as measures that target training and experience, they also face common challenges.
Such challenges include focusing on the past; applicant difficulty evaluating their
own abilities; and applicant dishonesty. It is useful to identify these challenges
and highlight how each one can impact the ability of a specific T&E assessment to
predict the future job performance of applicants. Decision makers should consider
this information when determining whether or not a particular T&E assessment is
appropriate for their hiring needs and situation.
Focus on the Past
ere are some commonsense reasons that a T&E assessment’s focus on the past seems
advantageous. T&E measures are grounded firmly in the principle of behavioral
consistency.
89
“A candidate who performed well on a job in the past is likely to
perform well on a similar job in the future; one who has behaved responsibly in the
past is likely to be responsible in the future.”
90
ere is a strong sense that the past already exists, cannot be changed, and is
therefore easy to verify. Further, many T&E assessment methods require minimal
development effort and can be put into place quickly.
Applicants also seem to agree that T&E measures are reasonable, or have good face
validity. As a result, hiring decisions based on them are challenged much less often
than other types of assessments.
Despite this allure, there are three kinds of challenges with trying to predict future
job behavior from what we are told about the past: (1) what an applicant reports
89
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In D. Whetzel and G. Wheaton (Eds.),
Applied Measurement: Industrial Psychology in Human Resources Management (pp. 161-180) Lawrence Erlbaum Associates:
Mahwah, NJ.
90
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY, p. 508. See
also Schmidt, F., Caplan, J., Bemis, S., Decuir, R., Dunn, L., & Antone, L. (1979). The behavioral consistency method of
unassembled examining. Washington, DC: U.S. Office of Personnel Management, Personnel Resources and Development
Center.
A Report by the U.S. Merit Systems Protection Board
27
2828
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 3 - What are the Common Challenges?
is not necessarily what happened; (2) applicants do not necessarily learn eectively
from past events; and (3) training and experiences in the past may not be applicable
to now, or to the future.
Challenge 1: What an applicant reports is not always what happened. e
past may be unchangeable, but our access to it is indirect and imperfect. Our
assumptions about what happened during training or a work experience may be
incorrect. Simple attendance at training is no guarantee that the attendee paid
attention or learned eectively. is may be true for reasons unrelated to the
attendee, including poorly designed or delivered training, training not targeted at
the right level (e.g., introductory courses when advanced courses are needed), and
dated or inaccurate training content. Even when a person is prepared, motivated,
and engaged, learning does not occur at the same rate over time. It typically follows
a curve of diminishing returns where a great deal is learned initially and the amount
learned decreases over time, approaching some “mastery maximum” where most
possible learning has occurred. is curve will have a dierent shape for dierent
skills, environments, and a number of other factors.
91
Performance-improving experience also may not be obtained on the job. Much
of an employee’s work may have been performed by others, performed poorly, or
performed under very dierent conditions or levels of assistance than implied.
Even if performance level is not an issue, some abilities may deteriorate if they are
not used. It is also not uncommon for a job title to imperfectly reect the duties
actually performed. e unused skills implied by the job title may have once been
possessed by an employee, but may have atrophied through disuse during the “years
of experience.”
92
Time passage is a particularly poor measure of whether an individual possesses
the kind of abilities that develop or emerge only in the face of unusual, highly-
demanding situations.
93
For example, reghters who have been through a southern
Californian brushre season have a very dierent year of experience than they would
in a rainy eastern city with only an occasional localized re.
Further, applicant descriptions of their past experience often conceal, or at least
deemphasize, gaps in their periods of employment. While omitted events could
mean something irrelevant to the job, like family medical leave, they could also
91
Goldstein, I. and Ford, K. (2004). Training in organizations: Needs assessment, development and evaluation.
Wadsworth: Belmont, CA.
92
Jacobs, R., Hoffman, D. & Kriska, S. (1990). Performance and seniority. Human Performance, 3, 107-201.
93
Elder, G. & Clipp, E. (1989). Combat experiences and emotional health: Impairment and resilience in later life. Journal
of Personality, 57, 311-342.
A Report by the U.S. Merit Systems Protection Board 29
Chapter 3 - What are the Common Challenges?
include something job-relevant, such as a period of incarceration. A history
gap often indicates a period of unemployment, a time during which no kind of
experience is being gained. It can be dicult to establish how many years really
were spent gaining experience.
Challenge 2: Applicants do not all learn effectively from past training and
experiences. ere are individual dierences in how people acquire, retain and
recall information. People come to both training and work experiences with
dierent levels of various abilities. is aects how quickly they learn and, in
some cases, what they are able to learn.
94
It has been suggested that there are three
primary factors which inuence how much two dierent people might learn from
the same experience in the same length of time. ese three factors are: “…
individual dierences, such as cognitive ability, conscientiousness, openness to
experience, and extraversion; situational characteristics such as opportunities
to perform tasks; and motivation to pursue opportunities.”
95
Individuals
learn and develop at dierent rates, making it dicult to estimate how much
a given applicant may have improved based on work experiences—and how
quickly such skills may atrophy when they are not exercised for a time.
96
is is
consistent with previous MSPB studies highlighting that people dier in their
preparedness and ability, resulting in dierential benets from the same training.
97
Also, length of time at a job might not always be a good thing. In some cases, the
length of time spent in a job may be negatively related to ability level, if it reects
an inability to obtain advancement or another position.
98
“e idea that anyone can
become expert given enough time is a myth; those who take a long time to learn a
task generally do not reach the level of prociency after training reached by those
who learn it more quickly.”
99
e risk that time on the job reects lower, rather than
higher, prociency is greater if the experience was gained in an organization that
does not eectively address performance problems, for reasons that may include
indecision or reluctance to invest the organizational eort needed to remediate or
remove a poor performer.
94
MSPB (2010). Making the Right Connections: Targeting the Best Competencies for Training. MSPB: Washington, DC.
95
McGonigle, T. & Curnow, C. (2003). Current theory and practice on the measurement of experience. Paper presented
at the Mid-Atlantic Personnel Assessment Consortium. Rehobeth Beach, DE.
96
P. L. Ackerman. (1988). Determinants of individual differences during skill acquisition. Journal of Experimental Psycholog y,
117, 288-318; and Thoresen, C., Bradley, J., Bliese, P. & Thoresen, J. (2004). The big five personality traits and individual job
performance growth trajectories in maintenance and transitional job stages. Journal of Applied Psycholog y, 89, 835-853.
97
MSPB (2010). Making the Right Connections: Targeting the Best Competencies for Training. MSPB: Washington, DC.
98
Jacobs, R., Hoffman, D. & Kriska, S. (1990). Performance and seniority. Human Performance, 36, 107-201.
99
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY.
3030
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 3 - What are the Common Challenges?
Further, as we will discuss in greater depth below, people have various kinds of
memory problems recalling past experiences. And people suer from these problems
to dierent degrees and have diering degrees of success using strategies to mitigate
them.
100
Challenge 3: Applicants’ past training and experiences may no longer be relevant.
Research suggests that the value of training and experience gradually decreases
with the passage of time as forgetting occurs, skills become rusty with disuse, and
technology and best practices evolve in the workplace.
101
Jobs also dier in the degree to which they require skills and knowledge from the
past. “While experience may be more important where success depends heavily
upon procedural or institutional knowledge, it may be less important where the
needed KSAs are rapidly shifting, particularly when the employee does not practice
continual learning in an eort to stay current in his or her eld.”
102
Jobs and
professional disciplines change at dierent rates over time. Ironically, the very
climate of speed and change that encourages managers to adopt T&E assessments
also contributes to the greater speed at which work skills become obsolete.
103
In summary, these three challenges are inherent to focusing on the past for
information about job-related performance and abilities. Below, we discuss an
additional set of challenges that emerge when we ask candidates to evaluate their
experience.
Unreliable Self-Assessment
It is generally accepted that self-ratings tend to overestimate experience, ability,
and performance, and do not distinguish well between qualied and unqualied
applicants.
104
Applicants can have problems recognizing, supplying, and in some
cases, evaluating information about their training and experience. ese issues
are based on shortcomings in human abilities to perform the activities required by
many T&E assessments. Below, we will explore three main challenges in applicant
self-assessment: (1) applicants may have trouble recalling or reporting the most
100
Schacter, D (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American
Psychologist 54 (3), 182–203.
101
McDaniel, M., Schmidt, F. & Hunter, J. (1988). A meta-analysis of the validity of methods for rating training and
experience in personnel selection. Personnel Psycholog y, 41, 283-314.
102
MSPB (2013). Preserving the integrity of the Federal Merit Systems: Understanding and addressing perceptions of
favoritism. MSPB: Washington, DC.
103
Jansen, P. (1997). Assessment in a technological world. In N. Anderson and P. Herriot (Eds.), International Handbook
of Selection and Assessment (pp. 125-145). Chichester, UK: John Wiley.
104
van Rijn, P. (1980). Self-assessment for personnel examining: An overview. (OPRD Report 80-14). OPM.
A Report by the U.S. Merit Systems Protection Board 31
Chapter 3 - What are the Common Challenges?
appropriate or applicable experiences; (2) applicants may not have an accurate
perspective on the range of their prociencies; and (3) applicants may not accurately
evaluate their prociencies. Fortunately, many of these issues can be corrected.
105
Challenge 1: Applicants have problems recalling past experiences. T&E
assessment questions often ask applicants to recall and evaluate the past, reecting
both the necessity and the assumption that applicants are the best sources for this
information. Yet, people experience diculty and make systematic errors when
trying to remember their past experiences.
106
Research indicates that recall of past
events is not only incomplete, but may be inuenced by people’s mistaken beliefs
that their past behavior is consistent with their current behavior, and with their own
conceptions of ideal behavior.
107
Further, applicants may not always report the most appropriate or applicable
experiences. For example, some T&E assessments impose brevity constraints that
require applicants to report only a subset of their achievements. e achievements
described are not necessarily the ones that show the upper range of applicants’
abilities.
108
Applicants may inappropriately focus on the accomplishments that are
the most recent; those that stand out because they were early in their career; or
those that were enjoyable, interesting, or otherwise memorable. Unfortunately,
such achievements might not best represent applicants’ full range of capabilities or
best match the kinds of experiences that a hiring manager is looking for.
Also, such decisions about which experiences to report can also be aected by the
number of options people must choose from.
109
While applicants with novice
experience in an area may be able to easily recall and report the few opportunities
they have had in a particular area, the applicant with expert experience has the far
more dicult task of selecting from a much larger number of accomplishments; it
may not be easy to come up with one that shows the expert applicant in the best
possible light for the position.
Challenge 2: Applicants lack perspective on their level of prociency. Readers
may be familiar with the phrases, “I know enough to know that I know nothing,”
105
Podsakoff, P., MacKenzie, S. & Podsakoff, N. (2003). Common method biases in behavioral research: A critical review
of the literature and recommended remedies. Journal of Applied Psycholog y, 88(5), 879-903.
106
Schacter, D. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American
Psychologist 54 (3), 182–203.
107
Pearson, R., Ross, M. & Dawes, R. (1992). Personal recall and the limits of retrospective questions in surveys. In J. M.
Tanaur (Ed.), Questions about questions: Inquiries into the cognitive bases of surveys (pp. 65-94). Russell Sage Foundation: New York,
NY.
108
Kahneman, D. (2012). Thinking, fast and slow. Farrar, Straus & Giroux: New York, NY.
109
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY.
3232
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 3 - What are the Common Challenges?
or “I know just enough to be dangerous.” ese derive from the reality that people
have diculty understanding how much they know about a given area. Many
errors that people make in self-ratings occur because they lack perspective about
their own abilities. People dier in the degree to which they are reective and able
to achieve accurate self-knowledge.
110
Further, low and high performers dier in
their awareness of the frontier of knowledge in a particular area. Low or novice
performers are unfamiliar with the nature of advanced performance—this is part of
what makes them low performers—and they may incorrectly believe they are close
to the top of the performance dimension. ey know very little about what they
do not know and they often cannot fully describe what is required on the job.
111
Yet, even top performers have trouble comparing themselves to others, and tend
to underestimate their own abilities.
112
High performers are aware of the frontier of
the unknown and the more complex problems they might face. is makes them
less likely to endorse that their level of knowledge or experience is at the top of the
continuum; they realize the truly advanced nature of expert level.
Challenge 3: Applicants have problems self-evaluating. Coupled with often having
poor perspective on the range of their abilities, applicants also have diculties
rating their own abilities. Sources of error in self-ratings include dierences
between applicants in memory as they attempt to recall their previous experiences,
and dierences in the degree to which individuals are reective and have accurate
self-knowledge about their work-related performance.
113
Further, people dier in
their abilities to self-rate and in other characteristics that aect the quality of these
ratings. Factors identied by research include people’s beliefs about their abilities
to accomplish tasks,
114
conscientiousness (as measured by a personality inventory),
115
general intelligence, and their cognitive complexity when rating on multiple
dimensions.
116
110
Graham, K., McDaniel, M. & Snell, A. (2002). Biodata validity decay and score inflation with faking: Do item
attributes explain variance across items? Journal of Business and Psycholog y, 16(4), 573-592.
111
IAG-CTEA, (2009).
112
Dunning, D., Heath, C. & Suls, J. (2004). Flawed self-assessment: implications for health, education, and the
workplace, Psychological Science in the Public Interest, 5(3), 69-106; and Thornton, G. (1980). Psychometric properties of self-
appraisals of job performance. Personnel Psycholog y, 33(2), 363-271.
113
Graham, K., McDaniel, M. & Snell, A. (2002). Biodata validity decay and score inflation with faking: Do item
attributes explain variance across items? Journal of Business and Psycholog y, 16(4), 573-592.
114
Bernardin, H. J. and Villanova, P. (2005). Research streams in rater self-efficacy. Group and Organization Management, 30,
61-88.
115
Tziner, A., Murphy, K., and Cleveland, J. (2005). Contextual and rater factors effecting rating behavior. Group and
Organization Management, 30, 89-97.
116
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. Routledge: New York, NY.
A Report by the U.S. Merit Systems Protection Board 33
Chapter 3 - What are the Common Challenges?
Additionally, applicants are unlikely to have received training in how to use rating
scales to eectively calibrate their prociencies. Unfortunately, untrained raters
tend to form inaccurate personal impressions about rating scales and how to apply
them. Such inaccuracies result in distortions in how applicants appraise and rate
their prociencies. For example:
•Central tendency bias occurs when ratings cluster closely around the
middle of the scale;
•Leniency or severity bias occurs when raters use either the top or the
bottom of the scale, making few distinctions; and
•Halo bias occurs when a general impression is used to self-rate across a set
of specic tasks.
While there are statistical methods to detect and correct for these biases, such
methods require data from other raters for comparison. In self-rating, this is not
possible as rating data only comes from one source: the applicant.
Challenges in appropriately recalling relevant performance information, maintaining
proper perspective, and correctly rating abilities are faced by applicants who
attempt to report and evaluate their experience fairly, accurately, and completely.
Unfortunately, not all applicants have such good intentions, leading to another area
of challenges with T&E assessments: deliberately false or misleading assertions by
candidates about their training and experience.
Applicant Dishonesty
Some applicants deliberately fabricate or exaggerate their past training, experience,
and accomplishments. One review of studies that examined applicant dishonesty
estimated that 30 to 70 percent of job applications contain untrue statements or
signicant exaggerations.
117
Another study found that between 1 and 78 percent
of occupational questionnaire scores for eligible applicants across six Federal job
openings were not supported by an applicant’s documentation. While a variety
of factors likely contributed to these results, exaggeration and dishonesty likely
contributed as well.
118
Some assessment developers discuss an ination bias that occurs when applicants
portray themselves in the best possible light, perhaps without realizing they are
117
Wood, J. L., Schmidtke, J. M. & Decker, D. L. (2007). Lying on job applications: The effects of job relevance,
commission, and human resource management experience. Journal of Business and Psycholog y, 22, 1-9.
118
Barton, M. (2010). Automated T&E questionnaires: Practical outcomes and development considerations. SIOP.
3434
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 3 - What are the Common Challenges?
doing so.
119
is is part of a general truthfulness problem researchers call social
desirability in survey research and faking in personality assessment. It is a signicant
threat to accurate T&E assessment.
Reference checking is a widely-used attempt to reduce the impact of applicant
dishonesty. MSPB has studied its use in Federal hiring and has found that it can
be accomplished using methods that are both legally defensible and eective.
120
Yet,
reference checking must be structured and thorough to be eective. Unstructured
reference checking, wherein a hiring manager asks unplanned and inconsistent
questions across candidates, is not wise and unlikely to provide useful information
in a balanced, transparent, or fair manner. Seeking only conrmatory information
is also unwise; irrespective of the impressions a hiring manager has about an
applicant’s qualications, hiring managers should be open to all information during
reference checking, both good and bad. Indeed, hiring managers need to avoid
falling into the trap of looking to conrm their preconceptions, instead of testing
them or disproving them.
121
Applicant dishonesty, abetted by ineective reference checking, remains a challenge
to accurate T&E assessment.
122
Summary
is chapter examined challenges encountered when evaluating past training and
experience. Some stem from the nature of the past. Others reect problems in
how the past is probed. e next chapter reviews strategies that can be used to
either improve the eectiveness of agency T&E assessments, or identify alternative
methods.
119
Barton, M. (2010). Automated T&E questionnaires: Practical outcomes and development considerations. SIOP.
120
MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington, DC.
121
Johnson-Laird, P. (2006). How we reason. Oxford University Press: Oxford, UK.
122
Some agencies are proactive about evaluating and improving their reference checking practices. See Reference Checking in
the Department of Justice (2013) available at www.justice.gov/oig/reports/2013/e1302.pdf .
A Report by the U.S. Merit Systems Protection Board 35
e last chapter highlighted challenges faced when using training and experience
assessments. Most of these are not fatal aws. Signicant improvements can be
made in how well T&E assessments predict future job performance if the right
techniques are used to counter their weaknesses.
123
In this chapter we review several strategies for improving T&E assessments. ese
strategies are:
• Improve accuracy of T&E assessments;
• Increase verification of applicant information;
• Use present-oriented assessments; and
• Monitor the latest assessment information and trends.
We discuss these strategies at a general level, but caution readers that such strategies
must be adapted to the specics of each type of assessment. Further, although these
techniques have the potential to mitigate several weaknesses in T&E assessments,
hiring managers are still responsible for exercising wise judgment to identify the
most appropriate assessment for their individual hiring needs and situations. is
includes weighing the disadvantages and any possible strategies to improve any
assessment, against its advantages. is analysis is not easy, and hiring managers
might consider guidance from consultants or HR specialists who specialize in
assessment development and use.
Improvement Strategy 1: Make T&E Assessments More Accurate
A number of techniques can be applied to increase the accuracy of T&E assessments.
ey include grounding T&Es in job analysis and improving T&E questions,
rating scales, scoring, and applicant self-assessment. Most involve improving how
applicants recall and evaluate their own training and experience.
123
See, for example, Quinones, M., Ford, K. & Teachout, M. (1995). The relationship between work experience and job
performance: A conceptual and meta-analytic review. Personnel Psycholog y, 48, 887-910.
Chapter 4 - How Can We Raise
the Bar?
3636
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
Ground in job analysis. Among other criteria, useful T&E measures focus on
job-relevant attributes. A current job analysis is necessary to establish job-relevant
attributes, such as knowledge, skills, abilities, and competencies.
124
MSPB has
recommended: “…the Government should be developing assessments that better
measure the quality of the skills actually developed through that training and
experience. is can be accomplished to a certain extent through T&E assessments,
but only if agencies improve the way they do job analyses and crediting plans. In
particular, the Government needs to use more rigorous procedures to determine
performance elements that separate high performers from low performers...”
125
Agencies that have used T&E measures extensively emphasize the importance of
using questions grounded in job analysis.
126
Ideally, a current, detailed, and well-documented job analysis will either already
be available, or the resources will be available to conduct one. If job analysis
documentation is not available, it is important to consult position descriptions,
vacancy announcements, performance standards, and training materials to determine
how and how well the job should be done. Subject-matter experts (SMEs), or those
who are familiar with the job as it is currently performed, should also be consulted.
e best SMEs are those who are exceptional performers. ey may be the go-
to experts for dicult problems; they may be supervisors or instructors of other
employees. In any case, SMEs can be an invaluable resource in generating—or
vetting—job analysis material
Improve T&E questions. An assessment can only be as good as the quality of the
questions it asks. Some quality standards for questions are obvious and apply to
all assessments in general. For example, assessment questions should be clear to
both experienced and inexperienced applicants, and they should be as concise as
possible. If an applicant cannot understand what a question is asking, it will be
very dicult for him or her to provide an accurate or appropriate response, even
with the best of intentions. Other question quality standards are more specic to
the kinds of questions that appear only in T&E assessments. We provide a sample
of such T&E specic standards below. Although these standards can be applied to
T&E methods in general, they are most pertinent to occupational questionnaires
and accomplishment records.
124
OPM’s Assessment Decision Guide defines job analysis as: “A systematic examination of the tasks performed in a job
and the competencies required to perform them.” See also Prien, E., Goodstein, L., Goodstein, J. & Gamble, L. (2009). A
practical guide to job analysis. Wiley: Hoboken, NJ.
125
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC.
126
Agarwal, S., Busciglio, H., Leaman, J., Simmons-Collins, T., Curtin, P. & Thompson, S. (2010). Understanding the
practical outcomes of self-report, task-based assessment questionnaires for assessing applicants for U.S. Customs and Border
Protection vacancies. SIOP, 2010.
A Report by the U.S. Merit Systems Protection Board 37
Chapter 4 - How Can We Raise the Bar
• Stimulate recall. When possible, questions and instructions should
be written in a way that helps stimulate accurate recall of past events.
One approach guides applicants through consideration of the length
of their experience with a task and the frequency of task performance.
This not only helps scoring, but stimulates the applicant’s recall of
the task and associated context.
• Distinguish between levels of proficiency. Items should be written
in a manner that allows for clear distinctions between acceptable
and unacceptable levels of proficiency. Tasks that either everyone
can do or that no one can do provide no useful information and
waste the applicant’s time. For example, a question that nearly every
applicant gets a high score on does not help the agency hone in on
the most qualified candidates. In contrast, a question that virtually
no candidates can answer is too demanding, and may cause high-
quality candidates to self-eliminate out of the application process on
the assumption that the employer is seeking—and expects to find—
an expert-level candidate.
• Encourage applicants to review their job qualifications. To facilitate
accuracy in rating their T&E, applicants should review their
performance appraisals and similar documents before a T&E
assessment to walk through their work history and prepare to answer
questions about it.
Developing questions that elicit useful, job-related information from applicants is a
specialized skill and is unlikely to be done well without a certain amount of training
and experience.
127
As mentioned above, hiring managers should seek guidance
from consultants or HR specialists who have specialized knowledge in assessment
development.
Improve T&E rating scales. In questionnaire-based T&E assessments, applicants
are typically asked to use a rating scale to self-evaluate their abilities or prociencies.
As discussed, the task of accurately recalling and appraising one’s past experiences or
overall expertise in an area is a challenging process. is challenge is compounded
when an individual is given a poorly designed rating scale that does not appropriately
capture or convey the full range of prociency in a given area. A poor rating
scale is confusing or dicult to use; a good one is easy to understand, grounded
in job analysis, and eectively represents the continuum of prociency in a given
127
OPM (2011). A training guide for developing assessment questionnaires in USA Staffing. OPM: Washington, DC.
3838
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
area, while distinguishing between milestone levels of prociency in that area.
Detailed guidance on the proper design of rating scales for T&E assessments is
beyond the scope of this report. Hiring managers are encouraged to seek guidance
from consultants, HR specialists, or others who have specialized training in scale
development. However, we provide a few points to consider when developing T&E
assessment scales.
• Specificity of Focus. When possible, T&E assessment scales should
ask applicants to self-rate their past on specific tasks rather than
their abilities in general. The “mental algebra” of estimating how
several abstract abilities contribute to performance is very difficult
for applicants. Rating specific task performance is an easier and
more natural undertaking.
• Number of Response options. T&E rating scales generally provide
applicants with 2, 3, or 5 response options for each question. Five-
point scales capture more information than 2 response options, such
as “Yes/No,” and are preferred. The only exception is when a T&E
question asks about a binary topic, such as possession of a license or
certificate necessary to perform the job (e.g., license to practice law
or board certified in medicine).
128
• Scale Labels. For scales with three or more points, descriptive
anchors on the ends, such as “novice level” and “expert level” make
the scale easier for applicants to use and less prone to errors.
• Scale Point Behavioral Examples. Greater accuracy and applicant
understanding of the self-assessment task can be achieved by using
behaviorally-anchored rating scales (BARS).
129
The scale includes
descriptions of the behavior that should be expected at different
levels of task performance, which are derived from a job analysis
or SME opinion. Applicants use the scale by comparing their
achievements to the example behaviors, and selecting the behavior
(and corresponding scale point) that best mirrors their achievement.
BARS helps give applicants a common frame of reference to use
128
Agarwal, S., Busciglio, H., Leaman, J., Simmons-Collins, T., Curtin, P. & Thompson, S. (2010). Understanding the
practical outcomes of self-report, task-based assessment questionnaires for assessing applicants for U.S. Customs and Border
Protection vacancies. SIOP, 2010.
129
Smith, P. & Kendall, P. (1963). Retranslation of expectations: An approach to the construction of unambiguous
anchors for rating scales. Journal of Applied Psycholog y, 47, 149-155.
A Report by the U.S. Merit Systems Protection Board 39
Chapter 4 - How Can We Raise the Bar
when evaluating their experiences, which can help avoid problems
with different applicant perspectives on how high, low, and middle-
range achievement should be defined.
• Response Scale Reusability. Scales can be generic, with the same
response options applied to all questions, tasks, or abilities on a T&E
assessment, or they may be customized for each question, task, or
ability. Both styles have been used successfully. See Figure 1 and
Figure 2 for examples of generic and customized response options,
respectively. Note how the response options in the Generic scale
(Figure 1) could be applied to many abilities or competencies, while
the response options for the customized scale (Figure 2) are specific
to the competency “Oral Communication.”
Figure 1. Sample Item with Generic Scale
Instructions: Select the statement that best describes your training and experience in [insert task].
A. I have not had education, training, or experience in performing this task.
B. I have had education or training on this task but have not yet performed it on the job.
C. I have performed this task on the job. My work on this task was monitored by a
supervisor or senior employee to ensure compliance with proper procedures.
D. I have performed this task as a regular part of a job. I have performed it independently
and normally without review by a supervisor or senior employee.
E. I have supervised performance of this task or I am normally the person who is consulted
by other workers to assist them in doing this task because of my expertise.
4040
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
Figure 2. Sample Item with Customized Scale
Instructions: Indicate the extent to which you have communicated orally with various levels of
employees to obtain and provide information.
A. I have had no experience in performing this task.
B. I have communicated orally with others to obtain or verify
information or to provide routine information.
C. I have communicated orally with supervisors, managers, or office personnel to notify them of
decisions, problems, or further actions needed, or to explain the organization’s programs or services.
D. I have given short oral presentations at departmental/organizational briefings
and meetings to convey information on program activities or to describe the
impact of new organizational policies on operational responsibilities.
E. I have led briefings or taught courses on highly technical or complex material
to audiences such as high-level managers, attorneys, or executives.
Improve applicant self-assessment prociency. When an applicant is asked to rate
his or her experiences or qualications for a particular job, he or she is acting as a
rater. ere is an abundance of research on rating processes which has insights for
understanding (and improving) how applicants rate their abilities. For example,
some self-rating diculties stem from how well applicants understand the rating
scales.
130
Investigations of untrained raters and of the errors they make during
training reveal that many use their own personal beliefs of how the rating scales
should work when making ratings. Although their individual rating behavior can
be quite consistent—they are consistently inaccurate with respect to how the scales
were designed to be used.
131
Training applicants on how to interpret and use T&E
rating scales could help them more accurately rate their job qualications, thereby
making T&E assessments which rely on such ratings more useful. Indeed, the
quality of applicant self-ratings will improve when scales and self-rating procedures
are more clearly explained to applicants before self-rating occurs.
132
e most eective training method is frame of reference training, which gives raters
guided practice using the scales they will be using on the assessment. Having learned
130
Primoff, E. (1980). The use of self-assessments in examining. Personnel Psycholog y, 33, 283-290.
131
Uggerslev, K. & Sulsky, L. (2008). Using frame-of-reference training to understand the implications of rater
idiosyncrasy for rating accuracy. Journal of Applied Psycholog y, 93(3), 711-719.
132
Bernardin, H. & Buckley, M. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212; van Rijn, P.
(1980). Self-assessment for personnel examining: An overview. (OPRD Report 80-14). OPM.
A Report by the U.S. Merit Systems Protection Board 41
Chapter 4 - How Can We Raise the Bar
the rationale for the rating task, raters use this perspective in their rating tasks.
133
is experience can be used to oer brief self-rating training to applicants prior to
beginning a T&E assessment.
134
e key is that applicants receive instruction in
how to use the rating scales, so that they will be in a better position to apply them
appropriately to their own experiences and abilities.
ere are various ways to deliver self-rating training to applicants. Instruction in
self-rating could be included in a mail or email message sent to applicants before
they engage with the T&E assessment. Some training can also be included in the
assessment session, although there are obvious time limits for this. Another option
is to integrate web-based training into USAJOBS
®
. Eective web-based training
has been developed for raters in other contexts,
135
and would likely be helpful for
applicants too. For example, OPM could develop several modules of such training
for dierent forms of assessments and make them available for agencies to use.
For its centrally administered assessments, OPM might even require applicants to
complete a training module before taking the assessment. Many agencies already
do something like this by including links to the assessment questions applicants
will encounter. Although job announcements may be longer than applicants can
bear,
136
this is another possible place for training. Regardless of method, however, it
is critical that all applicants are provided equal opportunity to utilize such training.
Improve T&E scoring. ere are a variety of strategies for scoring T&E assessments,
some notably better than others. A comprehensive discussion of the complexities
of developing a scoring strategy is beyond the scope of this report. However, we
emphasize that a key element in any good scoring strategy is that scores reliably
and accurately distinguish between levels of applicant quality. An MSPB review of
Federal hiring recommended that, “…the Government needs to use more rigorous
procedures to determine performance elements that separate high performers from
low performers and to develop valid scoring techniques for measuring applicants
against those dimensions.”
137
133
Bernardin, H. & Buckley, M. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212; Gorman, C.
& Rentsch, J. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of
Applied Psycholog y, 94(5), 1336-1344.
134
Lievens, F. & Sanchez, J. (2007). Can training improve the quality of inferences made by raters in competency
modeling? A quasi-experiment. Journal of Applied Psycholog y, 92(3), 812-819. For a discussion of training for raters for assessment
centers, see Schleicher, D., Day, D., Mayes, B. & Riggio, R. (2002). A new frame for frame-of-reference training: Enhancing the
construct validity of assessment centers. Journal of Applied Psycholog y, 87, 735–746.
135
Aguinis, H., Mazurkiewicz, M. & Heggestad, E. (2009). Using web-based frame-of-reference training to decrease
biases in personality-based job analysis: An experimental field study. Personnel Psycholog y, 62, 405-438.
136
MSPB. (2003). Help Wanted: A review of Federal vacancy announcements. MSPB: Washington, DC.
137
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC, p. 21.
4242
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
Overall, we recommend that hiring mangers seek and follow the advice of an
assessment development specialist when deciding how to score a T&E assessment
or an assessment of any type. Not doing so risks poor predictive validity for the
assessment.
Improvement Strategy 2: Increase Verification of Applicant Responses
Verifying an applicant’s T&E information is much more likely to produce an
accurate picture of the applicant’s abilities than relying on just the applicant’s
perspective alone. MSPB has previously discussed the importance of verifying
information obtained early in the assessment process.
138
Ideally, all information
from every candidate would be subjected to verication. Yet, for practical reasons,
it is not often feasible to cross check every bit of information that an applicant
provides. As such, agencies need to rely on one or more strategies to corroborate
a cross-section of applicants or applicant information. ere are a range of
strategies to choose from. For example, OPM has developed verication strategies
for several types of assessments, most notably occupational questionnaires and
accomplishment records.
139
Some strategies are agency-driven, in which the agency
takes responsibility for checking applicant supplied information. Such agency-
driven strategies include:
• Verifying information of a random sample of applicants, reducing
the resources required while giving each applicant the same chance
of investigation;
• Verifying information from misplaced candidates who score highly
on other assessments and are candidates for referral or selection;
• Verifying selective content, such as what is critical to the job, what is
most often falsified or exaggerated, or what does not seem consistent
with other information;
• Verifying information by inserting bogus (i.e., not real) job tasks
and looking for applicants who say they have performed such tasks;
• Verifying through additional assessments that seek to corroborate
the legitimacy of an applicant’s assertions; and
138
MSPB. (2004). Identifying Talent through Technology: Automated Hiring Systems in Federal Agencies. Washington,
DC.
139
Sharpe, P. & Sorensen, K. (2011). Assessment of training and experience. IPAC 2011, Washington, DC. See also
Barton, M., Bisges, J. & Holloway-Lundy, A. (2010). Practical outcomes associated with use of automated questionnaires at the
U.S. Office of Personnel Management. SIOP 2010.
A Report by the U.S. Merit Systems Protection Board 43
Chapter 4 - How Can We Raise the Bar
• Verifying information by using reference checks to consult with
people who are familiar with applicants’ behavior in past situations.
Agencies can also use verication strategies that are more applicant-driven. Such
strategies focus on encouraging honest and accurate applicant reporting, placing
the onus on applicants to verify (and certify) that what they have written is true.
Examples of applicant-driven strategies, include:
• Provide notice of verification or actual warnings about the
consequences of misrepresenting or falsifying application information;
• Requiring documentation from applicants that supports their claims
or proficiencies; and
• Requiring elaboration from applicants on their experiences to justify
their proficiency ratings.
e following subsections highlight several agency-driven and applicant-driven
strategies. We provide such strategies for illustrative purposes only; hiring managers
are responsible for exercising wise judgment in deciding which verication strategy
is most appropriate for their situation.
Notice of verication. One applicant-driven strategy is to encourage applicants
to provide more accurate information, and to exaggerate and falsify less, by telling
them that their information will be veried. Warnings about verication and
negative consequences of falsication have been shown to decrease falsication
even when no actual verication occurs.
140
Negative consequences in warnings may
include removal from consideration, removal from position after hire, and in some
cases prosecution.
141
e best results can be achieved by placing the warning at
the beginning of the T&E assessment and again at the end with a reminder that
applicants can go back and change their answers.
142
It can also be placed in the
vacancy announcement.
140
Dwight, S. & Donovan, J, (2003). Do warnings not to fake reduce faking? Human Performance, 16(1), 1-23. See also
Lautenschlager, G. J. (1994). Accuracy and faking of background data. In G. Stokes, M. Mumford & W. Owens (Eds.), Biodata
handbook: Theory, research and use of biographical information in selection and performance prediction. (pp. 391-419). CPP
Books: Palo Alto, CA.
141
See 5 C.F.R. § 731.202(b) which explains that one of the grounds upon which an individual may be found unsuitable for
employment includes making a “[m]aterial, intentional false statement, or deception or fraud in examination or appointment”);
§§ 731.203-731.205 authorizes cancellation of eligibility; removal; cancellation of reinstatement eligibility; and debarment if an
individual is found unsuitable.
142
IAG-CTEA, (2009).
4444
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
Warnings should be combined with an application attestation statement where
applicants indicate, by signing or entering initials, that they have read the warning,
understand it, and that everything they have said is true to the best of their
knowledge. Warnings carry additional weight if applicants are asked to provide
names and contact information for people who can verify the information they
provide. Figure 3 provides a sample warning statement.
Figure 3. Sample Verification Warning
Warning:
All of the information you provide may be verified by a review of the work experience and/or
education as shown in your application forms, by checking references, and through other means,
such as the interview process. This verification could occur at any stage of the application process.
Any exaggeration of your experience, false statements, or attempts to conceal information may
be grounds for rating you ineligible, not hiring you, or for firing you after you begin work.
Applicant Attestation:
By checking the box to the left of this statement and by typing my full name in the space below, I declare
and affirm that I have read and fully understand that:
1. Any misrepresentation or material omission of facts on this assessment questionnaire
or in any other materials I submit in support of my candidacy (including but not
limited to the application), or in any oral statements I may make during the selection
process shall be sufficient cause to end further consideration of my candidacy.
2. Persons listed as having knowledge of my past accomplishments on this
assessment questionnaire may be contacted for verification purposes.
3. An offer of employment is contingent on successful completion of the entire employment
selection process, including the receipt and review of references, satisfactory to the agency.
4. This verification may, but need not, begin prior to my receiving an offer.
5. I will be evaluated only on the information submitted.
Verication by documentation. Another applicant-driven verication strategy is to
request that applicants provide documentation of training and experience as part
of their application materials. Requested documentation may include, a resume,
certicates or professional license information, or documents from previous work
experience. Such documentation may be used to verify applicant claims on an
occupational questionnaire or other T&E assessment.
A Report by the U.S. Merit Systems Protection Board 45
Chapter 4 - How Can We Raise the Bar
Some judgment is necessary to decide how much evidence is provided by each
document. Documents from trusted sources, such as licenses or performance
appraisals, should carry greater weight than those created by the applicant. However,
even applicant-generated documents can be useful. For example, the contents of
an applicant’s resume could be compared with his or her claims on an occupational
questionnaire for consistency. Any inconsistency discovered could reect a mistake
by the applicant or some form of misrepresentation. Either case could prompt a
follow-up discussion with the applicant to reconcile the inconsistency, especially if
the applicant appears otherwise highly-qualied for the position.
Verication by elaboration. A third applicant-driven verication strategy requires
applicants to produce a written justication for their claim to have had experience
with a task or to possess an ability.
143
Research shows less exaggerated responding
when this type of verication is used. ere is less ination for the specic
questions that are veried and for the other questions that appeared with them on
the questionnaire.
144
Verication by elaboration works best when applicants do not
know beforehand which questions will require elaboration. is encourages them
to respond to all questions at a level which they can justify. Although this strategy
can be used for several T&E assessments, it is most often used for questionnaire-
based T&E assessments.
For example, after an applicant has completed self-ratings of their experience on
an occupational questionnaire, a subset of the questions is selected for elaboration.
Applicants are provided with a text box
145
to enter a brief justication of each task
or competency rating. ey are asked for additional details such as how often have
they done a task, how much time it typically took, how much help they had, and
what tools they used.
146
ese narratives are later reviewed by experts in the jobs
and tasks. Adjustments may be made to an applicant’s ratings to better match the
level of experience described and justied in the applicant’s narratives. Applicants
are told in advance that such adjustments may take place.
147
An example of such an
elaboration narrative appears in Figure4.
143
Current OPM Hiring Reform regulations preclude use of these and other assessments requiring written narratives as
part of the initial screen in Federal hiring.
144
Schmitt, N. & Kunce, C. (2002). The effects of required elaboration of answers to biodata questions. Personnel
Psycholog y, 55(3) 569–587; and Schmitt, N., Oswald, F. Kim, B. & Yoo, T. (2003). Impact of elaboration on socially desirable
responding and the validity of biodata measures. Journal of Applied Psycholog y, 88(6), 979-988.
145
This is easiest to implement in an online assessment system because the order in which applicants see information can
be controlled. But it can also be done in a pencil-and-paper environment.
146
IAG-CTEA, (2009). Also, Schmitt, N. & Kunce, C. (2002). The effects of requiring elaboration of answers to biodata
questions. Personnel Psycholog y, 55(3), 569-587.
147
See, for example, Boyce, A., Carter, L., Cober, A., Montanari, M. & Quinones, R. (2010). The practical implications
of a narrative review process for mitigating the impact of socially desirable responding on self-report training and experience
assessments. SIOP.
4646
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
Figure 4. Verification by Elaboration
Sample Narrative (Specific Item):
Using the text box below, please provide a sample accomplishment from your education
and/or experience that supports your response to the item above. Your sample
accomplishment should be brief (a few sentences), specific, and verifiable.
Sample Narrative (Content Area, Certain Responses):
Select the choice that best describes your experience in working in or leading work-related teams.
A. I have had no experience in working in or leading work-related teams.
B. I have participated as a team member, receiving task assignments
and project goals from my supervisor or team leader.
C. I have served as a team leader, communicating task assignments and
project goals to team members and monitoring their progress.
D. I have led several teams, ensuring that project goals and deadlines were met,
and I have provided feedback to team members on their performance.
If you chose ‘C’ or ‘D’ in the previous question, please give the name and email
address of someone who can verify your experience as a team leader.
Keeping applicants honest may not be the only benet of asking applicants to
elaborate on their claims—these elaborations may actually improve their ability
to self-evaluate. The process of elaboration encourages applicants to thoroughly
consider their degree of experience, knowledge, and ability while answering a
question. The greater effort required ensures a certain level of motivation on the
part of the applicants. It can make the assessment of experience seem fairer to the
applicants. It also makes it harder for applicants to out-and-out fabricate because of
the need to come up with many details and make sure they all t together believably.
Verication with bogus tasks. One agency-driven verication strategy involves
inserting descriptions of bogus tasks—tasks that do not actually exist—into a T&E
assessment to identify candidates who indicate they have performed them. is
technique is most eective if applicants are informed that these bogus items are
scattered throughout the questionnaire. It is best to use several such bogus items
and to only take action against applicants who demonstrate a strong pattern of
claiming experience with them.
148
Simply warning that bogus tasks are present also
reduces faking and exaggeration.
148
Fluckinger, C.D., McDaniel, M. & Whetzel, D. (2009). Review of faking in personnel selection. In Mandal, M. (Ed.)
(2009) Emerging Frameworks in Science and Technology Personnel Selection and Recruitment. McMillian: New Delhi.
A Report by the U.S. Merit Systems Protection Board 47
Chapter 4 - How Can We Raise the Bar
e bogus task strategy works well to identify unqualied and dishonest applicants
who rate themselves high on all tasks to obtain a good score. Studies have found
that in some cases as many as half of applicants have claimed to perform a bogus
task.
149
Yet, claiming to perform a bogus task does not always reect dishonesty.
Sometimes it happens from not knowing the proper name for a task or equipment,
or genuinely believing (mistakenly) that the bogus task refers to something that they
have actually done.
Despite the allure of the bogus task strategy, it is very challenging to implement well.
First, it is not easy to write an eective bogus task question. e question must
clearly signal its bogus nature to honest applicants while seeming like a real task to
dishonest applicants. Second, this strategy is not appropriate for all occupations or
jobs. It works best in subject areas with highly technical content, dense terminology,
and rapid change. Finally, it is not appropriate for all applicant pools. Some
applicants have a strong negative reaction when told that there are bogus tasks on
an occupational questionnaire. is may damage an agency’s recruitment eorts
and reputation.
Verication by later assessment. Another agency-driven verication strategy is to
use additional assessments to verify information provided by applicants on earlier
assessments. MSPB has recommended that agencies use multiple hurdles (sometimes
called sequential hurdles)
150
in the assessment process for positions. “Using
assessment tools in succession can make the assessment process even more eective
in managing the candidate pool and narrowing the eld of qualied candidates.”
151
Although the primary purpose for sequencing assessments is to narrow the applicant
pool eciently, it also provides the opportunity to use later assessments to verify the
results of earlier assessments, such as conrming an applicant’s assertions on an
occupational questionnaire.
For example, an initial T&E assessment, such as an occupational questionnaire, is
used to obtain an estimate of each applicant’s prociency. Lower scoring applicants
are removed from consideration. Higher scoring applicants receive an additional
assessment. In this later hurdle, an assessment is administered that directly
measures present ability level in the same area that was assessed by the occupational
questionnaire. Yet, unlike the occupational questionnaire, the second assessment is
proctored and measures applicant ability in real time, greatly diminishing the risk of
149
McGonigle, T. & Curnow, C. (2007). Measures of training and experience. In Applied Measurement: Industrial
Psychology in Human Resources Management. Lawrence Erlbaum Associates: Mahwah, NJ; See also Burns, G. &
Christiansen, N. (2011). Methods of measuring faking behavior. Human Performance, 24(4), 358-372; Doll, R. (1971). Item
susceptibility to attempted faking as related to item characteristic and adopted fake set. Journal of Psycholog y, 77, 9-16.
150
Gatewood, R., Felid, H. & Barrick, M. (2010). Human Resource Selection. Cengage Learning: Independence, KY.
151
MSPB (2006). Reforming Federal Hiring: Beyond Faster and Cheaper. MSPB: Washington, DC.
4848
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
applicant dishonesty. Applicants who have exaggerated their abilities to get through
the rst screen will be screened out by the second assessment.
Verication by reference check. A third agency-driven strategy veries applicants’
claims about their training and experience through consultation with people who
are familiar with applicants’ behavior in these past situations. Applicants are asked
to supply names and usable contact information for people who can corroborate
their claims.
152
In this type of verication, most often done for accomplishment
records, these references are called veriers. ese veriers are often supervisors,
coworkers, clients, or others who were present when the applicant was doing the
work that they claim responsibility for. As with other types of verication discussed
in this section, simply telling applicants that reference checking will occur reduces
falsication and exaggeration on T&E assessments. For additional information on
how to conduct sound reference checks, see MSPB’s report on reference checking.
153
In summary, these applicant-driven and agency-driven verication techniques, used
singly or in combination, can reduce the impact of applicant misrepresentation and
dishonesty on T&E assessments, thereby helping to improve the validity of T&E
assessments.
Improvement Strategy 3: Use Assessments that Focus on Present Proficiency
A straightforward, but often overlooked, solution to the challenges presented by
T&E assessments is to use alternatives. One such alternative is to use present-
oriented assessments. Instead of using the past as a proxy for an individual’s current
level of prociency in a particular area, present-oriented assessments examine the
abilities and prociencies that individuals currently possess. ey often take the
form of tests or other structured activities where the applicant’s performance is scored
against a standard. Further, these present-oriented assessments can provide insights
into what prociencies individuals currently have that give them the potential to
learn new skills and develop existing ones over time.
Present-oriented assessments can predict future job performance more accurately
than most T&E assessments which rely on the past as a proxy for current prociency.
If carefully and properly developed, present-oriented assessments are usually more
eective tools. Further, OPM has recommended that agencies consider using
alternatives to T&E assessment when the cost of a hiring error is high due to the
mission-critical nature of the position; when positions are nontechnical and do not
require specic education and expertise; or when there is a history of management
152
IAG-CTEA, (2009).
153
MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington, DC.
A Report by the U.S. Merit Systems Protection Board 49
Chapter 4 - How Can We Raise the Bar
dissatisfaction with previous hires or referrals for the position.
154
MSPB has also
recommended that Federal agencies avoid over-relying on T&E assessments for
hiring and promotion decisions.
155
Of course, there will be circumstances where
T&E assessment is the best practical alternative. Time, funding, and other resource
constraints often preclude developing or purchasing such preferable present-oriented
tools. Further, T&E assessments are often the best way to assess some technical and
specialized skills. And sometimes T&E assessments are expected by executives and
other applicants who prefer not to take a “test.” However, before choosing a T&E
assessment, it is wise to consider some alternatives like those described below.
Present-focused structured interviews. As discussed in Chapter 2, structured
interviews are easy to develop and highly valid assessments with a solid track record
in Federal hiring.
156
ey are in the “Very Useful” category for predicting future
job performance (see Table 1). Present-oriented structured interviews can be one
alternative to past-oriented structured interviews and other T&E assessments in
some situations. By presenting applicants with a hypothetical scenario or problem
and asking them how they would respond, present-oriented questions reduce
challenges of recall, lack of perspective, and dishonesty. e applicant’s performance
is in the present, where it can be observed and evaluated, rather than in the past.
Additionally, since the interviews are scored in real-time by trained raters (not
the applicant), structured interviews avoid challenges inherent in applicant self-
assessment.
Simulations. Simulations or current work samples are another present-focused
assessment with a predictive validity of .54—slightly better than structured
interviews. is puts simulations in the “Very Useful” category for predicting
future job performance (see Table 1). MSPB has studied how agencies can best use
such assessments.
157
Chapter 2 noted that past work samples are T&E assessments
similar to accomplishment records. A current work sample avoids uncertainties
about an applicant’s role in an accomplishment by requiring a work product to be
produced under controlled conditions. Although there are complex simulations,
such as aircraft ight simulators, simple and inexpensive simulations can also be
eective assessments. Often job-relevant abilities can be assessed with a writing
sample or in-basket exercise.
154
IAG-CTEA, (2009).
155
MSPB (2010). Making the Right Connections: Targeting the Best Competencies for Training. MSPB: Washington,
DC, p. 47.
156
MSPB (2003). The Federal Selection Interview: Unrealized Potential. MSPB: Washington, DC.
157
MSPB (2009). Job simulations: Trying out for a Federal job. MSPB: Washington, DC.
5050
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 4 - How Can We Raise the Bar?
A simulation may also serve as a realistic job preview that encourages some applicants
to opt out of the selection process after discovering that they do not enjoy certain
tasks required by the job. is benets both employer and applicant by reducing
the mutual problem of a poor person-job t. It may also encourage applicants to
drop out who may be prone to exaggerate or falsify information on subsequent
assessments.
ese two present-oriented strategies and others can be used instead of— or along
with—T&E assessments across a number of occupations and to evaluate a range of
competencies. Other present-oriented assessments may work better with particular
competencies or occupations. We reiterate that hiring managers must consider
advantages and disadvantages, and exercise wise judgment, when selecting an
assessment for their particular hiring needs.
Improvement Strategy 4: Evaluate Developments in T&E Assessment
e ability of T&E assessments to predict job performance has increased over time
as new types of assessments have been devised, and existing assessments have been
improved. is is not only progress, but a reminder that further innovations may
emerge. Work environments, policies, and practices may also change in ways that
aect T&E assessment.
ose who use T&E assessments should watch for changes that aect the validity
of these assessments. Here are a few of the changes that seem possible in the near
future:
• Social networking and access to online work history information
may change the way we do reference checking and other types of
verification;
• Transparency and widespread sharing of documents on the web
makes more written work products more readily accessible. This
may change the way an applicant’s previous accomplishments are
verified;
• Changes in patterns of employment—such as an increase in the
typical number of employers over the course of an employee’s
career— may make it more difficult to interpret applicant work
history information;
A Report by the U.S. Merit Systems Protection Board 51
Chapter 4 - How Can We Raise the Bar
• Changes in collaborative work may affect how prior work products are
evaluated. Video-conferencing, document sharing, and collaborative
software make it easier to work closely with others, perhaps making
the level of involvement of each participant more difficult to verify;
• New ways of using automated testing platforms may become
common practice, such as using applicant response patterns to
determine what questions require verification; and
• There likely will be new ways of assessing training and experience,
and new ways of determining the quality of T&E assessments.
Watch for well-designed validity studies which assess the relationship
between applicant scores on T&E assessments and their subsequent
job performance.
Some relevant changes may already be under way. One interesting proposal is the
use of Skills Transcripts which contain information about an individual’s skills and
accomplishments learned on the job and which could follow employees through
their careers.
158
e idea is similar to the competency scores in OPM’s USA Hire
battery which can be used to apply for dierent jobs at dierent agencies.
159
e
“skills transcript” idea has its challenges, including requiring great cooperation
between employers. However, this and similar trends may change the way T&E
information should be evaluated.
Training and experience assessments can bring value to a hiring or promotion process
if implemented appropriately. As the previous chapters have discussed, there are
advantages and disadvantages to using T&E assessments, and each hiring manager’s
threshold for the right balance will be dierent. Whenever possible, agencies should
augment the usefulness of T&E measures through adopting one or more of the
improvement strategies that are currently available. Further, agencies should keep a
lookout for future developments in T&E assessment. Such developments can play a
critical role in shaping the eectiveness of an agency’s overall assessment approach,
workforce composition, and ultimately mission success.
158
Masie, E. (2011). Skills transcripts for workers? Learning TRENDS, May 22, 2011.
159
www.opm.gov/services-for-agencies/assessment-evaluation/online-assessment/ . A similar single-score project is
under way to create a post-college certification exam that could avoid current problems caused by non-comparability of grading
practices across educational institutions. See http://t.nbcnews.com/business/not-enough-graduate-college-now-theres-exit-
exam-8C11006596.
5252
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 5 - Conclusions and
Recommendations
MSPB examined T&E assessments as part of its mission to uphold the Merit System
Principles and support effective Federal workforce management. Additionally, T&E
assessments are widely used, well regarded, and hiring decisions made using T&E
assessments can have a lasting impact on the Federal workforce. Further, given the
austere economic climate currently enveloping Federal agencies—and the resultant
budget and resource constraints and limited hiring opportunities—it is all the more
critical that hiring managers make wise decisions on which assessments they use
to bring in talent. Although T&E assessments can be useful in Federal hiring,
they also have disadvantages that require attention and mitigation. Together these
factors made T&E assessments a prime candidate for study.
Previous chapters discussed the value of training and experience assessments; reviewed
the main types of training and experience assessments; highlighted some challenges
common to all such assessments; and provided methods for increasing the accuracy
and usefulness of T&E assessments. is chapter summarizes this study’s findings
about training and experience assessment and presents recommendations intended
to increase the effectiveness of these measures and improve Federal hiring practices.
Conclusions
1. T&E assessments are worth thoughtful consideration by hiring managers,
HR professionals, and agency decision makers. T&E assessments fall
within the scope of Merit System Principles that address selection,
training, and effective management of the Federal workforce. T&E
assessments are generally regarded as fair measures of job-related
abilities and they are widely-used by Federal agencies. Some T&E
assessments are used early in multistage assessments where they may
have great impact on later hiring decisions. T&E assessments may
also give an unintended advantage to internal applicants who have
performed a job superficially similar to the one for which they are
applying.
2. ere are professional standards that inform the development and
evaluation of T&E assessments for selection and promotion. e
most important standard is high predictive validity: “e extent to
which the assessment method has been shown to accurately measure
a job-related competency and/or predict successful performance on
A Report by the U.S. Merit Systems Protection Board
53
54
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Chapter 5 - Conclusions and Recommendations
the job.”
160
T&E assessments have a range of validity, with some
being clearly superior to others. However, the validity of some T&E
assessments is likely to increase over time as improved techniques
emerge from research and practice.
3. Some T&E assessments are better predictors of future job performance
than others. e better predictors of job performance include: direct
questions, occupational questionnaires, accomplishment records, KSA
narratives, and reference checks. Structured interviews and biodata,
which can include T&E information, are also useful in gauging an
applicant’s likelihood of success in a job.
4. Some commonly-used T&E assessments do not predict future job
performance well. ey include years of job experience, number of
training classes attended, grade point average, and resumes (when
used without verication).
T&E assessments focus on the past. is seems like a strength, but has
disadvantages. It is dicult to determine what happened in the past;
people dier in their abilities to recall and accurately report the past;
and the requirements of the workplace change over time, sometimes
making past experiences irrelevant to current duties.
5. Applicants are not good at evaluating their own abilities. ey
lack perspective on the range of their abilities and how their skills
compare to others; they have diculty recalling their past experiences
accurately; and they often have not had the training to appropriately
rate their prociencies.
6. Some applicants exaggerate or deliberately misrepresent their
training and experience. T&E assessments rely on applicants to
supply information about their qualications for a particular job.
Unfortunately, some applicants are careless or dishonest in how they
represent their past experiences.
7. T&E assessments can be improved. e recommendations section
below contains several possible strategies to improve assessment of
T&E.
160
OPM’s Assessment Development Tool at apps.opm.gov/ADT/content.aspxapps.opm.gov/ADT/content.aspx.
A Report by the U.S. Merit Systems Protection Board 55
Chapter 5 - Conclusions and Recommendations
Recommendations
e conclusions above support the following recommendations:
1. Low-validity T&E assessments should not be used in hiring decisions.
ey include years of job experience, number of training classes
attended, grade point average, and resumes (when used without
verication). Any assessment which is not supported by validity
evidence should also be avoided.
2. Strengthen T&E assessments with techniques that improve their
accuracy. Successful techniques include grounding the assessment in
a careful and recent job analysis, writing clear questions that aid recall
of job-relevant events, crafting rating scales that ease the evaluation
task, providing training in self-rating, and using scoring procedures
that are professionally-developed.
3. OPM should develop brief self-rating training for applicants. Such
training would improve accuracy of occupational questionnaire
ratings by applicants for Federal jobs. Centrally funded, it would be
available to applicants to all agency jobs, improving the hiring process
Governmentwide.
4. Verify information supplied by applicants. Applicant information
on T&E assessments can be corroborated through agency-driven and
applicant-driven verication strategies. Agency-driven techniques
check the veracity of applicant supplied information, while applicant-
driven techniques encourage applicants to provide honest and accurate
information.
5. Consider present-oriented alternatives to T&E assessment. Before
deciding to use a T&E assessment, a hiring manager should consider
using present-oriented assessments with higher predictive validity,
such as present-oriented structured interviews or work samples.
6. Monitor developments. Researchers and practitioners will continue
to improve T&E assessment. Work environments, policies, and
practices may also change in ways that aect T&E assessment.
MSPB recommends that agency decision makers, hiring managers, and
HR specialists consult this report before using a T&E assessment. is
information, in combination with their best professional judgment, can
help improve the assessment procedures in their agencies and the hiring or
promotion decisions based upon them.
5656
Evaluating Job Applicants: The Role of Training and Experience in Hiring
A Report by the U.S. Merit Systems Protection Board 57
What is validity?
161
When people talk about assessments, such as an employment examination, they often ask
whether the test is valid or not. Validity is the extent to which a test (such as a psychological
test, an academic aptitude test, or an employment selection instrument) measures what it
claims to measure. In the context of hiring, validity is an estimate of the degree to which
the result of the assessment (such as an applicant’s score on an occupational questionnaire,
or the rating received by an applicant’s accomplishment) predicts future performance on
the job.
Why does validity matter?
Validity matters for both practical and principled reasons. First, a test must be valid for
its results to be accurately interpreted and applied. In hiring, an assessment must have
acceptable validity for it to help an agency screen job applicants, or help a manager select
among referred candidates. An assessment that has no validity (such as handwriting
analysis) or low validity (such as years of experience) does not help a manager determine
whether a candidate will perform successfully if hired, or identify the candidates who are
best-qualied and most likely to perform at a high level.
Second, the merit system principles, which envision “selection solely on the basis of relative
ability” implicitly require Federal agencies to use valid assessments to evaluate applicants
and make hiring decisions. Regulations such as 5 CFR 300 and guidelines make this
expectation explicit and prescribe measures to promote the use of job-related criteria and
valid assessments.
How is validity measured?
e validity of a particular assessment method (such as the T&E measures discussed in
this report) is measured through research that collects and analyzes data on (1) the scores
that applicants with dierent levels of ability receive on the assessment and (2) their level
of performance on the job.
e validity statistics presented in this report are based on research that synthesized results
from a large number of research studies to determine the expected validity of each type of
assessment—so long as that assessment is developed according to professional standards
and current best practices. is kind of information can be used along with cost and other
constraints to design an assessment approach for a particular type of job.
161
Some of the material in this Appendix was adapted from an online Q&A on validity available at psychology.about.com/
od/researchmethods/f/validity.htm.
APPENDIX A - Understanding Validity
5858
Evaluating Job Applicants: The Role of Training and Experience in Hiring
Appendix A
5858
What is a validity coecient?
Validity is not determined by a single statistic, but by a body of research that demonstrates
the relationship between an assessment and the behavior it is intended to measure. But a
single statistic, the validity coecient, is often used to summarize the validity evidence for a
given assessment. Validity coecients range between 0 and 1. Values closer to 1 indicate
that an assessment is a good predictor of job performance; values closer to 0 indicate a poor
predictor.
We cannot expect validity coecients to approach 1.0 for many reasons which have nothing
to do with assessment quality.
162
e results of past validity studies suggest how to set our
expectations. Assessments with validity coecients of .35 or above are considered very
useful in predicting future job performance. A coecient between .21 and .35 indicates
an assessment that is likely to be useful. Assessments with validity coecients between .11
and .20 may be useful in some circumstances. An assessment with a coecient of .11 or
below is unlikely to be useful.
163
Can an assessment method’s validity coecient change?
e ability of an assessment to predict job performance can improve over time. is occurs
as research and practice identify ways to improve the way they are developed, administered,
and scored. e high validity of structured employment interviews that MSPB reported
in a previous study
164
were achieved because such improvements now distinguish them
from unstructured interviews.
165
MSPB’s study of reference checking
166
noted that such
a distinction between structured and unstructured reference checks was not yet made in
validity studies. Subsequent validity studies found an improvement in reference checking
which adopted best practices, relative to less eective unstructured reference checking.
167
T&E assessments can also benet from this type of progress.
162
Schmidt, F., Caplan J., Bemis, S., Decuir, R., Dunn, L. & Antone, L. (1979). The behavioral consistency method
of unassembled examining (TM-79-21). Washington, DC: U.S. Office of Personnel Management, Personnel Research and
Development Center.
163
Sproule, C. (2009). Rationale and research evidence supporting the use of content validation in personnel assessment.
IPAC. Available at www.ipacweb.org.
164
MSPB (2003). The Federal Selection Interview: Unrealized Potential. MSPB: Washington, DC.
165
Huffcutt, A. & Culbertson, S. (2011). Interviews. In S. Zedeck (Ed.), APA Handbook of Industrial and Organizational
Psychology. APA: Washington, DC (Vol. 2, pp. 185-204); See also Huffcutt, A. & Arthur, W. (1994). Hunter & Hunter (1984)
revisited: Interview validity for entry-level jobs. Journal of Applied Psycholog y, 79, 184-190.
166
MSPB (2005). Reference Checking in Federal Hiring: Making the Call. MSPB: Washington, DC.
167
Taylor, P., Pajo, K., Cheung, G. & Stringfield, P. (2004). Dimensionality and validity of a structured telephone
reference check procedure. Personnel Psycholog y, 57(3), 745–772.
A Report by the U.S. Merit Systems Protection Board 59
APPENDIX B - Validity Coecients
e table below contains the predictive validity coecient for each type of
assessment discussed in this report.
How well do different types of assessment
predict future job performance?
Practical Value Assessment
167
Validity
Very Useful
Simulation .54
Structured Interview .51
Accomplishment Record
.45
Likely Useful
Biodata .35
Occupational Questionnaire
.15 to .28
Reference Checks
.26
KSA Narratives
—
Possibly Useful
Years of Experience .18
Transcript/GPA
.17
Resume (by itself)
.11
Not Useful Number of Training Classes —
Variable Direct Question —
167
Training and experience assessments listed in this column are in bold. Assessment names in italic
indicate assessments which contain T&E questions along with other types of questions.
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
6060
Evaluating Job Applicants: The Role of Training and Experience in Hiring
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S.
Merit
Systems
Protection
Board
Neil
A.
G.
McPhie,
Chairman
Mary
M.
Rose,
Vice
Chairman
Office
of
Policy
and
Evaluation
Director
John
Crum,
Ph.D.
Project
Manager
Cynthia
H.
Ferentinos,
Ph.D.
Assistant
Project
Manager
James
J.
Tsugawa
Project
Analysts
J.
Peter
Leeds,
Ph.D.
Allison
M.
Wiley
U.S. Merit Systems
Protection Board
1615 M Street, N.W.
Washington, DC 20419
Official Business
Penalty For Private Use $300
PRSRT STD
U.S. POSTAGE PAID
WASHINGTON DC
PER MIT NO. G-113
Evaluating Job Applicants:
e Role of Training and Experience in Hiring
