Detecting Anomalies in Assessment Devices for Cross-Cultural Use: Fairness Concerns
Abstract
Assessment devices are developed with the undertaking of collecting data that will inform a valid and reliable crux of interest. Such information is mandatory as it will constitute a basis for responsible decision-making. In order to meet the projected function appropriately, any assessment device, let alone those for cross-cultural use, needs to be anomaly-free. While the use of assessment devices across nations has been ubiquitous worldwide, scrutinizing assessment devices for any existing covert defects is unavoidably imperative if fairness in the result interpretation is to be envisioned. This paper reviews briefly several concepts related with nuisances that potentially cause anomaly in assessment devices particularly for cross-cultural use.
Keywords: anomalies, assessment devices, cross-cultural studies
References
[1] F. A. Bachtiar, E. Cooper, G. H. Sulistyo, and K. Kamei, “Student assessment based on affective factors in English learning using fuzzy inference,” in International Journal of Affective Engineering, vol. 15, pp. 101–108, 2015.
[2] J. W. Berry, “Imposed Etics-Emics-Derived Etics: The Operationalization of a Compelling Idea,” International Journal of Psychology, vol. 24, no. 1, pp. 721–735, 1989.
[3] J. D. Brown, “What is construct validity?” JALT Testing & Evaluation SIG Newsletter, vol. 4, no. 2, p. 12, 2000.
[4] A. Brown, “Measurement invariance and differential item functioning. A short course in applied psychometrics,” Peterhouse College, vol. 10, p. 12, 2012.
[5] G. Camilli and A. L. Shepard, Methods for identifying biased test items, Sage, London, UK, 1994.
[6] E. Davidov, B. Meuleman, J. Cieciuch, P. Schmidt, and J. Billiet, “Measurement equivalence in cross-national research,” Annual Review of Sociology, vol. 40, pp. 55–75, 2014.
[7] R. C. Gardner, “Attitude/motivation test battery: International AMTB research project,” 2004, http://publish.uwo.ca/%7Egardner/docs/englishamtb.pdf.
[8] R. Hambleton and J. Rodgers, “Item bias review. Practical Assessment,” Research & Evaluation, vol. 4, no. 6, p. 1, 1995, http://pareonline.net/getvn.asp.
[9] J. He and de. F. Van Vijver, “Bias and equivalence in cross-cultural research,” Article, vol. 8, p. 19, 2012.
[10] J. L. Horn and J. J. McArdle, “A practical and theoretical guide to measurement invariance in aging research,” Experimental Aging Research, vol. 18, no. 3, pp. 117– 144, 1992.
[11] E. K. Horwitz, M. B. Horwitz, and A. J. Cope, “Foreign language classroom anxiety,” Modern Language Journal, vol. 70, no. 2, pp. 125–132, 1986.
[12] C. H. Hui and H. C. Triandis, “The instability of response sets,” Public Opinion Quarterly, vol. 49, no. 2, pp. 253–260, 1985.
[13] G. Iskifoglu, “Approaches to study culture, equivalency and bias: Triadic notions in developing research instruments for cross-cultural comparative studies,” in Proceedings of INTED2014 Conference 10th-12th, pp. 4653–4658, Valencia , Spain, 2014.
[14] D. N. Jackson and S. Messick, “Response styles on the MMPI: Comparison of clinical and normal samples,” Journal of Abnormal and Social Psychology, vol. 65, no. 5, pp. 285–299, 1962.
[15] A. R. Jensen, “An examination of culture bias in the wonderlic personnel test,” Intelligence, vol. 1, no. 1, pp. 51–64, 1977.
[16] T. P. Johnson, “Approaches to equivalence in cross-cultural and cross-national survey research,” ZUMA-Nachrichten Spezial, January 1998.
[17] J. L. Juodvalkis, B. A. Grefe, M. Hogue, D. J. Svyantek, and W. DeLamarter, “The effects of job stereotype, applicant gender, and communication style on ratings in screening interviews,” in and communication style on ratings in screening interviews. The International Journal of Organizational Analysis, vol. 11, pp. 67–84, 2003.
[18] E. S. Knowles and K. T. Nathan, “Acquiescent Responding in Self-Reports: Cognitive Style or Social Concern?” Journal of Research in Personality, vol. 31, no. 2, pp. 293– 301, 1997.
[19] A. J. Kunnan, “Fairness and justice for all. In Antony,” in Fairness and validation in language assessment: Selected papers from the 19th Language Testing Research Colloquium, John. Kunnan and John. Antony Kunnan, Eds., p. 14, University, Orlando, Florida, 2000.
[20] A. K. Liauw, Language anxiety in speaking among students in Khims English course (Unpublished Masters Thesis [Master, thesis], Nommensen University, Pematang Siantar, North Sumatera, 2012.
[21] L. Marotta, L. Tramonte, and J. Douglas Willms, “Equivalence of testing instruments in Canada: Studying item bias in a cross-cultural assessment for preschoolers,” Canadian Journal of Education, vol. 38, no. 3, 2015.
[22] L. Maycock and T. Green, “The effects on performance of computer familiarity and attitudes towards CB IELTS,” Research Notes, vol. 20, p. 3, 2005.
[23] C. D. Mccauley and J. Mendoza, “A Simulation Study of Item Bias Using a Two- Parameter Item Response Model,” Applied Psychological Measurement, vol. 9, no. 4, pp. 389–400, 1985.
[24] G. Meisenberg and A. Williams, “Are acquiescent and extreme response styles related to low intelligence and education?” Personality and Individual Differences, vol. 44, no. 7, pp. 1539–1550, 2008.
[25] S. Messick, (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd Ed.) (pp. 13-103). New York: Macmillan.
[26] R. E. Millsap, “Statistical approaches to measurement invariance,” Statistical Approaches to Measurement Invariance, pp. 1–355, 2012.
[27] com. Merriam-Webster. Merriam-Webster and n. d. Web, 2016.
[28] S. Osterlind, Test Item Bias, vol. , SAGE Publications, Inc., 2455 Teller Road, Newbury Park California 91320 United States of America, 1983.
[29] J. W. Popham, Assessment Bias: How to Banish It, Person, Boston, 2012.
[30] O. Rezaei, H. Barati, and M. Youhanaee, “The effect of content familiarity & test format on iranian efl test takers’ performance on test of reading comprehension,” International Journal of Applied Linguistics and English Literature, vol. 1, no. 4, pp. 1–14, 2012.
[31] M. Rodriguez and O. Abreu, “The stability of general foreign language classroom anxiety across English and French,” Modern Language Journal, vol. 87, no. 3, pp. 365–374, 2003.
[32] J. Salvia and E. J. Ysseldyke, Assessment, Houghton Mifllin Company, Boston, 8th edition, 2001.
[33] R. J. Sbordone, “Ecological validity: Some critical issues for neuropsychologists,” in Ecological validity of neuropsychological testing, R. J, C. J. Sbordone, and Long., Eds., pp. 15–42, St. Lucie Press, Boca Raton, 1998.
[34] M. A. Schmuckler, “What is Ecological Validity? A Dimensional Analysis,” Infancy, vol. 2, no. 4, pp. 419–436, 2001.
[35] J. Scheuneman, “Exploration of Causes of Bias in Test Items,” ETS Research Report Series, vol. 1985, no. 2, pp. i–73, 1985.
[36] A. Shantz and P. G. Latham, “The effect of stereotype threat on the interview performance of women,” Advancing Women in Leadership, vol. 32, no. 1, p. 29, 2012.
[37] L. A. Shepard, G. Camilli, and M. D. Williams, “Validity of approximation techniques for detecting item bias,” Journal of Educational Measurement Vol, vol. 22, no. 2, pp. 77–105, 1985.
[38] E. Shohamy, “Fairness in language testing,” in Fairness and Validation in Language Assessment: Selected papers from the 19th Language Testing Research Colloquium, A. J and Kunnan., Eds., pp. 15–19, University, Orlando, Florida, 2000.
[39] J.-B. E. M. Steenkamp and H. Baumgartner, “Assessing measurement invariance in cross-national consumer research,” Journal of Consumer Research, vol. 25, no. 1, pp. 78–90, 1998.
[40] G. H. Sulistyo, Reading for meaning, Pustaka Kaiswaran, Malang, 2011.
[41] G. H. Sulistyo, “EFL learning assessment at schools., Malang,” Bintang Sejahtera, 2015.
[42] C. Barker, The SAGE Dictionary of Cultural Studies, vol. , SAGE Publications Ltd, 1 Oliver’s Yard, 55 City Road, London EC1Y 1SP United Kingdom, 2004.
[43] R. J. Vandenberg and C. E. Lance, “A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research,” Organizational Research Methods, vol. 3, no. 1, pp. 4–69, 2000.
[44] K. Leung and F. J. R. Van de Vijver, Methods and data analysis for cross-cultural research, Sage, Newbury Park, CA, 1997.
[45] F. J. R. Van De Vijver and Y. H. Poortinga, “Towards an integrated analysis of bias in cross-cultural assessment,” European Journal of Psychological Assessment, vol. 13, no. 1, pp. 29–37, 1997.
[46] F. J. R. Van de Vijver, “Bias and equivalence: Cross-cultural perspectives. In,” in Crosscultural survey methods, J. A. Harkness F and P. Ph. Mohler, Eds., pp. 143–155, Wiley, New York, NY, 2003.
[47] F. J. R. Van de Vijver a and N. K. Tanzer, “Bias and equivalence in crosscultural assessment: an overview. Revue europeenne de psychologie appliquee: an overview. Revue europeenne de psychologie appliquee 54,” pp. 119–135, 2004.
[48] F. J. R. Van de Vijver and R. Fischer, “Improving methodological robustness in crosscultural organizational research,” R. S. Bhagat and R. M. Steers, Eds., pp. 491–517, Cambridge University Press, New York, 2009.
[49] M. Williams and R. Burden, Psychology for language teachers, Cambridge University Press, Cambridge, 1997.
[50] G. Yan, “The stability of general foreign language classroom anxiety across languages among Chinese undergraduate fforeign language learners,” Journal of Asia TEFL, vol. 7, no. 2, pp. 69–89, 2010