to Judge a Psychological Test
by Patrick Merlevede, MSc.
Understanding how you think and behave at work can have numerous
benefits on your life, such as being able to find your ideal job and having
increased confidence and contentment. You should know how to find a career
assessment that is worth your valuable time and money.
Most test developers and test users have good intentions,
yet due to inadequate training, there is some considerable misuse of test
data. This article has been written to warn about bad use of tests in
general and about "nonsense" claims that some commercially oriented
people may make about the tests they put on the market. Most of these
comments apply to any test, so anyone publishing one of these tests should
at least have a reasonable answer to these comments. Their answers should
explain what they do to address these specific issues.
Test design problems can be classified into two groups. The
first group has to do with the construction of the test itself
and the theoretical basis that underlies the test. A test without a solid
theoretical foundation cannot deliver good results. When a test has a
solid theoretical foundation, the psychometric analysis of the
test will confirm these foundations and will point out the problem areas
in the use of this test. Let us go a bit deeper into these two issues:
- Test Constructs - What theory underlies the test? How
are the test questions formulated? How do these questions relate to
the theory? Which were rejected during test development? Are these questions
context independent, or are they an exact match to the context for which
the test wants to predict performance? How do language and cultural
factors influence the test performance, even within one language area
or one geographical area? What other environmental factors play a role
and how are these issues addressed?
- Psychometric Quality - A task force of the American Psychological
Association examined honesty tests and found out that they couldn't
get access to research-based information, not even to basic properties
such as test reliability (e.g. internal consistency) and test validity
(correlations with various criteria), let alone test-retest reliability,
factor analysis, or other forms of internal structure validation. Test
designers should be able to provide information on these important topics.
There are numerous ways to misuse test date. In their book,
Eyde et. al. (1993) indicate 86 specific elements and 7 broad factors
that represent common problems in test use. This is a summary of the problems
- Comprehensive Assessment
- In part, this issue is about test follow up. "How do you assure
that the test score is an accurate description? With what other instruments
is this test combined that deliver extra predictor information?"
One must also consider the emotional state the tested person was in
when the test is filled out, as well as the test's research evidence
and test limitations.
- Proper Test Use - What training
did the persons using the test about this test? How is the quality of
test use controlled?
- Psychometric Knowledge -
Are correct basic statistical principles respected (concerning standard
error of measurement, reliability and validity)? Are these principles
applied for interpreting the test results? How does the tool limit the
number of false positives and false negatives (and the use of cut-offs)?
- Accuracy of Scoring
- Is the test filled out and tabulated by a person or by a computer?
This can make a major difference in the number of errors that are produced:
||Results tabulated by a
||Possible data entry error
and tabulation error
|Test taken via pen-and-paper
||Results tabulated by a computer
||Possible data entry error
|Test taken online
||Results tabulated by a computer
possible scoring problems
- Appropriate Use of Norms
- When using a standardized test, one can run into problems regarding
practical applications. Is this test measuring the important traits?
How do you objectively know if proactivity or teamwork is important
for success in this particular position? A good test will only use criteria
important to job performance, and test designers must be able to prove
the information directly relates to he job.
- Feedback to test-takers -
Test designers must provide correct interpretations, and be staffed
to do so.
Lack of Self-Knowledge - Test-takers
might not know enough about themselves to accurately answer the questions.
Overestimating or underestimating one's own abilities is a common challenge,
and tests must be designed to avoid this bias.
Falsification - Psychological
tests are often quite transparent, and it seems obvious to many observers
that job applicants would not willingly report undesirable behaviors that
would ruin their chances for employment (e.g. Goldberg et al. 1991). You
should always carefully examine a test before you use it for your organization.
Is there right and wrong answers? Will an educated candidate be able to
know what you want to hear? For example, some tests ask: At work, are
- Always late
- Sometimes late
- As punctual as the next guy
- Rarely late
Of course everyone will pick answer 4! Test designers should
be able to prove to you that they can eliminate falsification. This is
a major problem, because the number of "false positives" in
test results (this happens when people that pass the test, and then do
not perform as expected).
This article should raise some questions regarding responsible
and accurate test use. There are good tests out there, but they must answer
to these issues. The answers to these questions have to come from test
developers. When considering the purchase of a psychological assessment,
make sure it is a reliable choice with a solid research foundation.
To view our recommendation of a career test that complies
with all of these guidelines, click
L.D et al. (1993) Responsible Test Use - Case Studies for Assessing
Human Behavior, American Psychological Association, Washington DC
L.R. et al. (1991), Questionnaires used in the prediction of trustworthiness
in pre- employment selection decisions: An A.P.A. task force report.
S., Bradburn, N.M. (1982), Asking Questions - A Practical Guide to Questionnaire
Design; Jossey-Bass Publishers, San Francisco
S., Bradburn, N.M., Schwarz, N (1996), Thinking about Answers - The
Application of Cognitive Processes to Survey Methodology; Jossey-Bass
Publishers, San Francisco
Packages on Sale:
Everything a student needs to take the right steps
in his or her career. A perfect gift for any student you know!
Our flagship package, with two psychological tests
and a wealth of feedback, including unique 1-on-1 online training.
Ultimate Career Discovery
You won't find anything else like this. 3 tests
that cover every facet of your personality: attitudes, motivation, emotional
intelligence, and value systems.