Making sense of testing

We use career assessments in order to help our clients in identifying their unique characteristics. Each assessment is designed to measure different components, thus - with appropriate interpretation - assisting our clients to find career options which match their particular attributes, values, and skills (Osborn & Zunker, 2016).

While tests can assist client's decision making processes (Whitfield et al., 2009), to be effective, those tests need to be reliable and valid (Walsh & Betz, 2000). If a test is valid, it means that it actually measures what it says it measures: it does what it says on the tin (Heale & Twycross, 2015). There are three key types of validity: content validity (test accuracy); construct validity (does what it says on the tin - e.g. testing for job search skills might inadvertently be evaluating problem-solving skills); and criterion-related validity (where the same factor - or variable - is measured each time, through 'convergent' validity which is strongly correlated with similar tests; 'divergent' validity with poor correlation to different tests; and 'predictive' validity where the test is highly correlated to related factors - e.g. being task-oriented should lead to being a completer/finisher) (Heale & Twycross, 2015). 

Tests also need to have been normalised for the population group our client affiliates (awhis) to. That means that, when assessments are created, researchers have run a number of sample tests (usually around 300; Steve Evans, personal communication, 13 September 2021) on each population group, seeking normal distribution in the test results via cultural, ethnic, gender, political and socio-economic group factors (Hansen, 2003; Osborn & Zunker, 2016). We can see that normalising tests is going to be an expensive business, in giving 300 tests to measure each norm group.

We also need to have consistent test-retest rates: the same result needs to be achieved each time the test is run (Heale & Twycross, 2015). If our client does a test in March, we don't want to see that they obtain a completely different result when they repeat the test in July (one of the main bug-bears of MBTI; Mastrangelo, 2001). While it’s not possible to perfectly assess each career instrument, we can estimate their replicability (Heale & Twycross, 2015) through “internal [...] and test-retest reliability” (Osborn & Zunker, 2016, p. 37).

And, while we might have all reliability, validity and representative norm groups, we might still find that our client does not suit the test we propose. The client may complete the test and end up with results which make no sense. For example, each time I complete a RIASEC test, I get a different score. Over the years, I think I have seen a pattern: that in those of us with very generalist skills, the RIASEC test may lose it's test-retest reliability. I offer RIASEC here as one example: it is not the only one I have noticed. I have had clients who achieve poor results from HBDI, from MBTI, and from DiSC. All tests do not necessarily suit all people.

We must take all quantitative tests with a pinch of salt :-)

Sam

References:

Hansen, S. S. (2003). Career counselors as advocates and change agents for equality. The Career Development Quarterly, 52(1), 43-53. https://doi.org/10.1002/j.2161-0045.2003.tb00626.x

Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative studies. Evidence Based Nursing, 18(3), 66-67. https://doi.org/10.1136/eb-2015-102129

Herr, E. A. (2001). Chapter 2: Career Assessment: Perspectives on trends and issues. In J. T. Kapes, E. A. Whitfield (Eds.), A counselor's guide to career assessment instruments (4th ed., pp. 15-26). National Career Development Association.

Mastrangelo, P. M. (2001). [251] Myers-Briggs Type Indicator [Form M]. In B. S. Plake & J. C. Impara (Eds.), The fourteenth mental measurements yearbook (816-820). Buros Center for Testing.

Osborn, D. S., & Zunker, V. G. (2016). Using Assessment Results for Career Development (9th ed.). Cengage Learning.

Walsh, W. B., & Betz, N. E. (2000). Tests and Assessment (4th ed.). Prentice Hall.

Whitfield, E. A., Feller, R. W., & Wood, C. (Eds.). (2009). A counselor’s guide to career assessment instruments (5th ed., pp. 13–25). National Career Development Association.