Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students
DOI:
https://doi.org/10.4995/wpom.v5i2.3220Keywords:
AHP, higher education, student assessment, peer assessment, self assessment, teacher assessment, multicriteria decision methodAbstract
The aim of this paper is to propose a critical example to assess student competencies using the paired comparison as multiple criteria decision making tool based on analytic hierarchy process (AHP). We apply it to innovation competency in a subject with few students (10 master's students, divided into two groups). With the proposed methodology it is possible to create an ordered list for each group, having a distance between students (in one dimension) that reflects the degree of learning (knowledge, skills and attitudes). We found that the scores given by an evaluator for a student are consistent with the rest of the evaluation scores given to the other students in the group. It has also been found that the scores obtained by various sources (teacher, student self-assessment and evaluation of student peers) are consistent with each other. In the future, we should check whether this model is viable with large groups and we should propose a variant that allows sorting, in a list, people from different groups or convert the relative distance between group members in an absolute measure of the degree of achieving the learning outcomes set for the course.
Downloads
References
Albayrak, E.; Erensal, Y. C. (2004). Using analytic hierarchy process (AHP) to improve human performance: An application of multiple criteria decision making problem. Journal of Intelligent Manufacturing, Vol. 15, nº. 4, pp. 491-503. http://dx.doi.org/10.1023/B:JIMS.0000034112.00652.4c
Andreu Andrés, M. A.; García-Casas, M. (2014). La evaluación de la participación en equipos de trabajo universitarios (Assessment of participation in higher education team working activities). WPOM-Working Papers on Operations Management, Vol. 5, nº. 1. http://dx.doi.org/10.4995/wpom.v5i1.1758
Chin, S.; Pun, F.; Xu, Y.; Chan, S. F. (2002). An AHP based study of critical factors for TQM implementation in Shanghai manufacturing industries. Technovation, Vol. 22, nº. 11, p. 707. http://dx.doi.org/10.1016/S0166-4972(01)00065-7
Comisión Europea (1995). Libro verde de la innovación. Comisión Europea (ES/13/95/55220800.P00).
Conlon, P.; Hecker, K.; Sabatini, S. (2012). What should we be selecting for? A systematic approach for determining which personal characteristics to assess for during admissions. BMC Medical Education, Vol. 12, nº. 1. http://dx.doi.org/10.1186/1472-6920-12-105
De Beuckelaer, A.; Toonen, S.; Davidov, E. (2013). On the optimal number of scale points in graded paired comparisons. Quality and Quantity, Vol. 47, nº. 5, pp. 2869-2882. http://dx.doi.org/10.1007/s11135-012-9695-2
European Commission (2008). The European Qualifications Framework for Lifelong Learning (EQF). Office for Official Publications of the European Communities.
Fernández March, A. (2010). La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria. Revista de Docencia Universitaria, Vol. 8, nº. 1, pp. 11-34.
Gee, S. (1981). Technology transfer, innovation & international cometitiveness. Wiley & Sons.
Goffin, K.; Mitchell, R. (2010). Innovation management. Palgrave-MacMillan.
González Pernía, J. L.; Pe-a-Legazkue, I. (2007). Determinantes de la capacidad de innovación de los negocios emprendedores en Espa-a. Economia Industrial nº. 363, pp. 129-147.
Goswami, S.; Mathew, M. (2005). Definition of innovation revisited: An empirical study on Indian information technology industry. International Journal of Innovation Management, Vol. 09, nº. 03, pp. 371-383. http://dx.doi.org/10.1142/S1363919605001307
Hatzinger, R.; Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, Vol. 48, nº. 10.
Ikehara, K.; Toyoda, H. (2012). A course evaluation model using paired comparisons to integrate importance of rating criteria and students' ratings of teaching: Comparison of students' and teachers' evaluations. Japanese Journal of Educational Psychology, Vol. 60, nº. 1, pp. 48-59. http://dx.doi.org/10.5926/jjep.60.48
Ingols, C.; Shapiro, M. (2014). Concrete Steps for Assessing the Soft Skills in an MBA Program. Journal of Management Education, Vol. 38, nº. 3, pp. 412-435. http://dx.doi.org/10.1177/1052562913489029
Kan Ma, H.; Min, C.; Neville, A.; Eva, K. (2013). How Good Is Good? Students and Assessors' Perceptions of Qualitative Markers of Performance. Teaching and Learning in Medicine, Vol. 25, nº. 1, pp. 15-23. http://dx.doi.org/10.1080/10401334.2012.741545
Kasirian, M. N.; Yusuff, R. M.; Ismail, M. Y. (2010). Application of AHP and ANP in supplier selection process-a case in an automotive company. International Journal of Management Science and Engineering Management, Vol. 5, nº. 2, pp. 125-135. http://dx.doi.org/10.1080/17509653.2010.10671100
Khalaf, M. A. & El Mokadem, M. Y. (2011). A Systematic approach for prioritizing lean practices using AHP, EurOMA.
Klippel, A. F.; Petter, C. O.; Antunes, J. (2008). Management Innovation, a way for mining companies to survive in a globalized world. Utilities Policy, Vol. 16, nº. 4, pp. 332-333. http://dx.doi.org/10.1016/j.jup.2007.09.001
Lasnier, F. (2000). Réussir la formation par compétences. Guérin.
Lawson, B.; Samson, D. (2001). Developing innovation capability in organizations: A dynamic capabilities approach. International Journal of Innovation Management, Vol. 05, nº. 03, pp. 377-400. http://dx.doi.org/10.1142/S1363919601000427
LeBreton, J. M.; Senter, J. L. (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organizational Research Methods, Vol. 11, nº. 4, pp. 815-852. http://dx.doi.org/10.1177/1094428106296642
Lehto, A.; Kairisto-Mertanene, L.; Penttilä, T. (2011). Towards innovation pedagogy. A new approach to teaching and learning for universities of applied sciences. Turku University of Apllied Sciences.
Li, Y., Wang, L., & Dong, X. (2009). An AHP-based study on the assessment of bilingual teaching in higher education institute, in 2009 2nd International Conference on Education Technology and Training, ETT 2009; Sanya; China; 13 December 2009 through 14 December 2009; Category numberP3936; Code 79728, pp. 231-234.
Lind, D. S.; Rekkas, S.; Bui, V.; Lam, T.; Beierle, E.; Copeland, I. I. I. (2002). Competency-Based Student Self-Assessment on a Surgery Rotation. Journal of Surgical Research, Vol. 105, nº. 1, pp. 31-34. http://dx.doi.org/10.1006/jsre.2002.6442
Ljungman, A. G.; Silén, C. (2008). Examination involving students as peer examiners. Assessment & Evaluation in Higher Education, Vol. 33, nº. 3, pp. 289-300. http://dx.doi.org/10.1080/02602930701293306
Lohmann, J. R.; Rollins, H. A.; Hoey, J. J. (2006). Defining, developing and assessing global competence in engineers. European Journal of Engineering Education, Vol. 31, nº. 1, pp. 119-131. http://dx.doi.org/10.1080/03043790500429906
Lyons, R. K.; Chatman, J. A.; Joyce, C. K. (2007). Innovation in services: Corporate culture and investment banking. California Management Review, Vol. 50, nº. 1, pp. 174-191. http://dx.doi.org/10.2307/41166422
Marin-Garcia, J. A. (2009). Los alumnos y los profesores como evaluadores. Aplicación a la calificación de presentaciones orales. Revista Espa-ola de Pedagogía, Vol. 67, nº. 242, pp. 79-97.
Marin-Garcia, J. A.; Aznar-Mas, L. E.; González-Ladrón-de-Gevara, F. (2011). Innovation types and talent managment for innovation. Working Papers on Operations Management, Vol. 2, nº. 2, pp. 25-31. http://dx.doi.org/10.4995/wpom.v2i2.926
Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Miralles, C.; Rodríguez Villalobos, A. (2008). Profile and competences of Spanish industrial engineers in the European Higher Education Area (EHEA). Journal of Industrial Engineering and Management, Vol. 1, nº. 2, pp. 269-284. http://dx.doi.org/10.3926/jiem.2008.v1n2.p269-284
Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Perello-Marin, M. R.; Canos-Daros, L. (2009a). Proposal of skills for the bachelor degree of Industrial Engineering in the context of the new curriculum. Intangible Capital, Vol. 5, nº. 4, pp. 387-406. http://dx.doi.org/10.3926/ic.2009.v5n4.p387-406
Marin-Garcia, J. A.; Garcia-Sabater, J. J.; Bonavia, T. (2009b). The impact of Kaizen Events on improving the performance of automotive components' first-tier suppliers. International Journal of Automotive Technology and Management, Vol. 9, nº. 4, pp. 362-376.
Marin-Garcia, J. A.; Perez-Pe-alver, M. J.; Watts, F. (2013). How to assess innovation competence in services: The case of university students. Dirección y Organización nº. 50, pp. 48-62.
Marrin, M. L.; McIntosh, K. A.; Keane, D.; Schmuck, M. L. (2004). Use of the paired-comparison technique to determine the most valued qualities of the McMaster medical programme admissions process. Advances in Health Sciences Education, Vol. 9, nº. 2, pp. 129-135. http://dx.doi.org/10.1023/B:AHSE.0000027439.18289.00
Mol, M. J.; Birkinshaw, J. (2009). The sources of management innovation: When firms introduce new management practices. Journal of Business Research, Vol. 62, nº. 12, pp. 1269-1280. http://dx.doi.org/10.1016/j.jbusres.2009.01.001
Morley, D. D. (2009). SPSS macros for assessing the reliability and agreement of student evaluations of teaching. Assessment & Evaluation in Higher Education, Vol. 34, nº. 6, pp. 659-671. http://dx.doi.org/10.1080/02602930802474151
Mula, J.; Días-Madro-ero, M.; Poler, R. (2012). Configuración del Grado en Ingeniería de Organización Industrial en las universidades espa-olas. Dirección y Organización, Vol. 47, pp. 5-20.
Penttilä, T. & Kairisto-Mertanene, L. (2012). Innovation competence barometer ICB - a tool for assessing students' innovation competences as learning outcomes in higher education, in INTED2012 Conference. 5th-7th March 2012, pp. 6347-6351.
Perrenoud, P. (2005). La universitat entre la transmissió de coneixements i el desenvolupament de competències. El debat sobre les competències a l'ensenyament universitari, ICE UB. Documents de Docència Universitària, núm. 5., pp. 8-25.
Pond, K. (2007). Student Experiences of Peer Review Marking of Team Projects. International Journal of Management Education, Vol. 6, nº. 1, pp. 30-43.
Saaty, T. L. (1980). The Analytic Hierarchy Process. McGraw-Hill.
Schumpeter, J. (1934). The Theory of Economic Development. Harvard University Press.
Terry, R. E.; Harb, J. N.; Hecker, W. C.; Wilding, W. V. (2002). Definition of student competencies and development of an educational plan to assess student mastery level. International Journal of Engineering Education, Vol. JAM-PDF, nº. 2, pp. 225-235.
Tonnessen, T. (2005). Continuous innovation through company wide employee participation. TQM Magazine, Vol. 17, nº. 2, pp. 195-207. http://dx.doi.org/10.1108/09544780510583254
Vaccaro, I. G.; Jansen, J. J. P.; Van Den Bosch, F. A. J.; Volberda, H. W. (2012). Management Innovation and Leadership: The Moderating Role of Organizational Size. Journal of Management Studies, Vol. 49, nº. 1, pp. 28-51. http://dx.doi.org/10.1111/j.1467-6486.2010.00976.x
Van Overveld, K. & Verhoeff, T. (2013). Self-consistent peer ranking for assessing student work: Dealing with large populations, in CSEDU 2013 - Proceedings of the 5th International Conference on Computer Supported Education, pp. 399-404.
Veronese Bentes, A.; Carneiro, J.; Ferreira da Silva, J.; Kimura, H. (2012). Multidimensional assessment of organizational performance: Integrating BSC and AHP. Journal of Business Research, Vol. 65, nº. 12, pp. 1790-1799. http://dx.doi.org/10.1016/j.jbusres.2011.10.039
Viladrich Segués, M. C.; Doval Dieguez, E. (2011). Medición: fiabilidad y validez. Laboratori d'Estadística Aplicada i de Modelització (UAB).
Villa Sánchez, A.; Poblete, M. (2007). Aprendizaje basado en competencias. Una propuesta para la evaluación de las competencias genéricas. Universidad de Deusto.
Watts, F.; Garcia-Carbonell, A.; Andreu Andrés, M. A. (2013). Innovation competencies development: INCODE barometer and use guide. Turku University od Applied Sciences.
Watts, F.; Marin-Garcia, J. A.; Garcia-Carbonell, A.; Aznar-Mas, L. E. (2012). Validation of a rubric to assess innovation competence. Working Papers on Operations Management, Vol. 3, nº. 1, pp. 61-70. http://dx.doi.org/10.4995/wpom.v3i1.1159
Yao, J. (2010). Feature AHP method used for excellent students evaluation, in 2010 International Conference on Optics, Photonics and Energy Engineering, OPEE 2010; Wuhan; China; 10 May 2010 through 11 May 2010; Category numberCFP1033I-PRT; Code 81252, pp. 422-425.
Yiu, E.; Chan, K.; Mok, R. (2007). Reliability and confidence in using a paired comparison paradigm in perceptual voice quality evaluation. Clinical Linguistics and Phonetics, Vol. 21, nº. 2, pp. 129-145 http://dx.doi.org/10.1080/02699200600756355
Downloads
Published
How to Cite
Issue
Section
