Friday, 1 April 2022

Cross-group comparison where there is little data

When we are trying to evaluate something, such as an assessment, we need to know what the norms are for the base culture, to be able to make sense of the test results.

Norms are score standards developed from individual tests, analysed, checked for content and construct validity, and based on either the general population or on specific groups (Osborne & Zunker, 2016). Practitioners use test norms to determine whether their client's background and characteristics fit derived population norms (Osborne & Zunker, 2016). 

If we are testing a sub-culture within a national culture, then we would hope that the test had been normed for the culture we were applying the test to. There are significant cultural differences between Americans, and Pākehā and Māori population groups. US assessments being used in New Zealand appear to lack a local frame of reference for Aotearoa (Reid, 2010) - norm group - so results are likely to be less valid and less reliable. 

In the USA, obtaining sound statistical and population data for comparisons is simple, and run by the Buros Institute (Carlson et al., 2021). However, in New Zealand we have no Buros Institute: there is no national, independent, co-ordinating organisation with the resources to gather test data, to analyse that data, and to normalise the test results. 

Further, norming tests is expensive. If we consider that many of the tests cost between $300 and $500 per test, and most tests require around 350 norm group samples, the opportunity cost is immediately visible: $100,000 to $175,000. Ouch. Few tests in New Zealand have established norms, and those which exist are proprietary and not available for independent analysis and validation. 

On an international scale, Aotearoa is the size of a small city of 5m. We would have to do a LOT of paid tests to recoup the loss we made in creating a norm group. If we want to norm a Māori group, that is 16% of the overall population: 800,000 people. It is a small pool to be drawing from. It becomes difficult to reach enough participants to create a reliable norm group, and it becomes even more obvious that creating sub-culture norm groups may well be too expensive for the anticipated return. 

So how can we compare cultural markers and data from one country with the cultural marker from another country? By using a bridging measure. For example, if we had norm group data for a US test, we could use a cultural measure from the country where where the norm group data is based. We could compare the norm group culture to the culture of NZ, and bridge/not bridge the career assessment norm results (depending on the cultural alignment).

 A couple of possibilities for cultural bridging would be the Hofstede (1980, 1984) cultural continuum measures, or the work of House et al (2004; Chhokar et al., 2008). We attempt to work out the differences between the cultures using this intermediary marker. Then we compare the norm group from the US culture with the US cultural measure. We hope to see many similarities. Then we explore the differences between the cultural measure of Aotearoa and the cultural measure of the US. We then predict the likely norm group differences.

Although this is not ideal, the cross-cultural measurement bridge may help us here in New Zealand to better predict cultural fit of testing, and to identify measures that do not work here, such as - in my experience - the extraversion continuum in the Big 5 test. 


Sam

References:

  • Carlson, J. F., Geisinger, K. F. & Jonson, J. L. (Eds) (2021). The Twenty First Mental Measurements Yearbook. The Buros Institute of Mental Measurements. https://buros.org/mental-measurements-yearbook
  • Chhokar, J. S., Brodbeck, F. C. & House, R. J. (Eds.). Culture and Leadership Across the World: The GLOBE Book of In-Depth Studies of 25 Societies. Sage Publications, Inc.
  • Hofstede, G. H. (1980). Culture’s Consequences: International Differences in Work-Related Values (1984 Abridged Edition). SAGE Publications, Inc. 
  • House, R. J., Hanges, P. J., Javidan, M., Dorfman, P. W., & Gupta, V. (Eds.). (2004). Culture, leadership, and organizations: The GLOBE study of 62 societies. Sage Publications, Inc.
  • Osborne, D. S., &  Zunker, V. G. (2016). Using Assessment Results for Career Development (9th ed.). Cengage Learning.
  • Reid, L. A. (2010). Understanding how Cultural Values Influence Career Processes for Maori [Doctoral Thesis: AUT University]. http://openrepository.aut.ac.nz/handle/10292/1036

No comments :

Post a Comment

Thanks for your feedback. The elves will post it shortly.