Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students

Juan A. Marin-Garcia, Pablo Aragonés Beltrán, Mónica García Melón


The aim of this paper is to propose a critical example to assess student competencies using the paired comparison as multiple criteria decision making tool based on analytic hierarchy process (AHP). We apply it to innovation competency in a subject with few students (10 master's students, divided into two groups). With the proposed methodology it is possible to create an ordered list for each group, having a distance between students (in one dimension) that reflects the degree of learning (knowledge, skills and attitudes). We found that the scores given by an evaluator for a student are consistent with the rest of the evaluation scores given to the other students in the group. It has also been found that the scores obtained by various sources (teacher, student self-assessment and evaluation of student peers) are consistent with each other. In the future, we should check whether this model is viable with large groups and we should propose a variant that allows sorting, in a list, people from different groups or convert the relative distance between group members in an absolute measure of the degree of achieving the learning outcomes set for the course.


AHP; higher education; student assessment; peer assessment; self assessment; teacher assessment; multicriteria decision method

Full Text:



Albayrak, E.; Erensal, Y. C. (2004). Using analytic hierarchy process (AHP) to improve human performance: An application of multiple criteria decision making problem. Journal of Intelligent Manufacturing, Vol. 15, nº. 4, pp. 491-503.

Andreu Andrés, M. A.; García-Casas, M. (2014). La evaluación de la participación en equipos de trabajo universitarios (Assessment of participation in higher education team working activities). WPOM-Working Papers on Operations Management, Vol. 5, nº. 1.

Chin, S.; Pun, F.; Xu, Y.; Chan, S. F. (2002). An AHP based study of critical factors for TQM implementation in Shanghai manufacturing industries. Technovation, Vol. 22, nº. 11, p. 707.

Comisión Europea (1995). Libro verde de la innovación. Comisión Europea (ES/13/95/55220800.P00).

Conlon, P.; Hecker, K.; Sabatini, S. (2012). What should we be selecting for? A systematic approach for determining which personal characteristics to assess for during admissions. BMC Medical Education, Vol. 12, nº. 1.

De Beuckelaer, A.; Toonen, S.; Davidov, E. (2013). On the optimal number of scale points in graded paired comparisons. Quality and Quantity, Vol. 47, nº. 5, pp. 2869-2882.

European Commission (2008). The European Qualifications Framework for Lifelong Learning (EQF). Office for Official Publications of the European Communities.

Fernández March, A. (2010). La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria. Revista de Docencia Universitaria, Vol. 8, nº. 1, pp. 11-34.

Gee, S. (1981). Technology transfer, innovation & international cometitiveness. Wiley & Sons.

Goffin, K.; Mitchell, R. (2010). Innovation management. Palgrave-MacMillan.

González Pernía, J. L.; Pe-a-Legazkue, I. (2007). Determinantes de la capacidad de innovación de los negocios emprendedores en Espa-a. Economia Industrial nº. 363, pp. 129-147.

Goswami, S.; Mathew, M. (2005). Definition of innovation revisited: An empirical study on Indian information technology industry. International Journal of Innovation Management, Vol. 09, nº. 03, pp. 371-383.

Hatzinger, R.; Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, Vol. 48, nº. 10.

Ikehara, K.; Toyoda, H. (2012). A course evaluation model using paired comparisons to integrate importance of rating criteria and students' ratings of teaching: Comparison of students' and teachers' evaluations. Japanese Journal of Educational Psychology, Vol. 60, nº. 1, pp. 48-59.

Ingols, C.; Shapiro, M. (2014). Concrete Steps for Assessing the Soft Skills in an MBA Program. Journal of Management Education, Vol. 38, nº. 3, pp. 412-435.

Kan Ma, H.; Min, C.; Neville, A.; Eva, K. (2013). How Good Is Good? Students and Assessors' Perceptions of Qualitative Markers of Performance. Teaching and Learning in Medicine, Vol. 25, nº. 1, pp. 15-23.

Kasirian, M. N.; Yusuff, R. M.; Ismail, M. Y. (2010). Application of AHP and ANP in supplier selection process-a case in an automotive company. International Journal of Management Science and Engineering Management, Vol. 5, nº. 2, pp. 125-135.

Khalaf, M. A. & El Mokadem, M. Y. (2011). A Systematic approach for prioritizing lean practices using AHP, EurOMA.

Klippel, A. F.; Petter, C. O.; Antunes, J. (2008). Management Innovation, a way for mining companies to survive in a globalized world. Utilities Policy, Vol. 16, nº. 4, pp. 332-333.

Lasnier, F. (2000). Réussir la formation par compétences. Guérin.

Lawson, B.; Samson, D. (2001). Developing innovation capability in organizations: A dynamic capabilities approach. International Journal of Innovation Management, Vol. 05, nº. 03, pp. 377-400.

LeBreton, J. M.; Senter, J. L. (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organizational Research Methods, Vol. 11, nº. 4, pp. 815-852.

Lehto, A.; Kairisto-Mertanene, L.; Penttilä, T. (2011). Towards innovation pedagogy. A new approach to teaching and learning for universities of applied sciences. Turku University of Apllied Sciences.

Li, Y., Wang, L., & Dong, X. (2009). An AHP-based study on the assessment of bilingual teaching in higher education institute, in 2009 2nd International Conference on Education Technology and Training, ETT 2009; Sanya; China; 13 December 2009 through 14 December 2009; Category numberP3936; Code 79728, pp. 231-234.

Lind, D. S.; Rekkas, S.; Bui, V.; Lam, T.; Beierle, E.; Copeland, I. I. I. (2002). Competency-Based Student Self-Assessment on a Surgery Rotation. Journal of Surgical Research, Vol. 105, nº. 1, pp. 31-34.

Ljungman, A. G.; Silén, C. (2008). Examination involving students as peer examiners. Assessment & Evaluation in Higher Education, Vol. 33, nº. 3, pp. 289-300.

Lohmann, J. R.; Rollins, H. A.; Hoey, J. J. (2006). Defining, developing and assessing global competence in engineers. European Journal of Engineering Education, Vol. 31, nº. 1, pp. 119-131.

Lyons, R. K.; Chatman, J. A.; Joyce, C. K. (2007). Innovation in services: Corporate culture and investment banking. California Management Review, Vol. 50, nº. 1, pp. 174-191.

Marin-Garcia, J. A. (2009). Los alumnos y los profesores como evaluadores. Aplicación a la calificación de presentaciones orales. Revista Espa-ola de Pedagogía, Vol. 67, nº. 242, pp. 79-97.

Marin-Garcia, J. A.; Aznar-Mas, L. E.; González-Ladrón-de-Gevara, F. (2011). Innovation types and talent managment for innovation. Working Papers on Operations Management, Vol. 2, nº. 2, pp. 25-31.

Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Miralles, C.; Rodríguez Villalobos, A. (2008). Profile and competences of Spanish industrial engineers in the European Higher Education Area (EHEA). Journal of Industrial Engineering and Management, Vol. 1, nº. 2, pp. 269-284.

Marin-Garcia, J. A.; Garcia-Sabater, J. P.; Perello-Marin, M. R.; Canos-Daros, L. (2009a). Proposal of skills for the bachelor degree of Industrial Engineering in the context of the new curriculum. Intangible Capital, Vol. 5, nº. 4, pp. 387-406.

Marin-Garcia, J. A.; Garcia-Sabater, J. J.; Bonavia, T. (2009b). The impact of Kaizen Events on improving the performance of automotive components' first-tier suppliers. International Journal of Automotive Technology and Management, Vol. 9, nº. 4, pp. 362-376.

Marin-Garcia, J. A.; Perez-Pe-alver, M. J.; Watts, F. (2013). How to assess innovation competence in services: The case of university students. Dirección y Organización nº. 50, pp. 48-62.

Marrin, M. L.; McIntosh, K. A.; Keane, D.; Schmuck, M. L. (2004). Use of the paired-comparison technique to determine the most valued qualities of the McMaster medical programme admissions process. Advances in Health Sciences Education, Vol. 9, nº. 2, pp. 129-135.

Mol, M. J.; Birkinshaw, J. (2009). The sources of management innovation: When firms introduce new management practices. Journal of Business Research, Vol. 62, nº. 12, pp. 1269-1280.

Morley, D. D. (2009). SPSS macros for assessing the reliability and agreement of student evaluations of teaching. Assessment & Evaluation in Higher Education, Vol. 34, nº. 6, pp. 659-671.

Mula, J.; Días-Madro-ero, M.; Poler, R. (2012). Configuración del Grado en Ingeniería de Organización Industrial en las universidades espa-olas. Dirección y Organización, Vol. 47, pp. 5-20.

Penttilä, T. & Kairisto-Mertanene, L. (2012). Innovation competence barometer ICB - a tool for assessing students' innovation competences as learning outcomes in higher education, in INTED2012 Conference. 5th-7th March 2012, pp. 6347-6351.

Perrenoud, P. (2005). La universitat entre la transmissió de coneixements i el desenvolupament de competències. El debat sobre les competències a l'ensenyament universitari, ICE UB. Documents de Docència Universitària, núm. 5., pp. 8-25.

Pond, K. (2007). Student Experiences of Peer Review Marking of Team Projects. International Journal of Management Education, Vol. 6, nº. 1, pp. 30-43.

Saaty, T. L. (1980). The Analytic Hierarchy Process. McGraw-Hill.

Schumpeter, J. (1934). The Theory of Economic Development. Harvard University Press.

Terry, R. E.; Harb, J. N.; Hecker, W. C.; Wilding, W. V. (2002). Definition of student competencies and development of an educational plan to assess student mastery level. International Journal of Engineering Education, Vol. JAM-PDF, nº. 2, pp. 225-235.

Tonnessen, T. (2005). Continuous innovation through company wide employee participation. TQM Magazine, Vol. 17, nº. 2, pp. 195-207.

Vaccaro, I. G.; Jansen, J. J. P.; Van Den Bosch, F. A. J.; Volberda, H. W. (2012). Management Innovation and Leadership: The Moderating Role of Organizational Size. Journal of Management Studies, Vol. 49, nº. 1, pp. 28-51.

Van Overveld, K. & Verhoeff, T. (2013). Self-consistent peer ranking for assessing student work: Dealing with large populations, in CSEDU 2013 - Proceedings of the 5th International Conference on Computer Supported Education, pp. 399-404.

Veronese Bentes, A.; Carneiro, J.; Ferreira da Silva, J.; Kimura, H. (2012). Multidimensional assessment of organizational performance: Integrating BSC and AHP. Journal of Business Research, Vol. 65, nº. 12, pp. 1790-1799.

Viladrich Segués, M. C.; Doval Dieguez, E. (2011). Medición: fiabilidad y validez. Laboratori d'Estadística Aplicada i de Modelització (UAB).

Villa Sánchez, A.; Poblete, M. (2007). Aprendizaje basado en competencias. Una propuesta para la evaluación de las competencias genéricas. Universidad de Deusto.

Watts, F.; Garcia-Carbonell, A.; Andreu Andrés, M. A. (2013). Innovation competencies development: INCODE barometer and use guide. Turku University od Applied Sciences.

Watts, F.; Marin-Garcia, J. A.; Garcia-Carbonell, A.; Aznar-Mas, L. E. (2012). Validation of a rubric to assess innovation competence. Working Papers on Operations Management, Vol. 3, nº. 1, pp. 61-70.

Yao, J. (2010). Feature AHP method used for excellent students evaluation, in 2010 International Conference on Optics, Photonics and Energy Engineering, OPEE 2010; Wuhan; China; 10 May 2010 through 11 May 2010; Category numberCFP1033I-PRT; Code 81252, pp. 422-425.

Yiu, E.; Chan, K.; Mok, R. (2007). Reliability and confidence in using a paired comparison paradigm in perceptual voice quality evaluation. Clinical Linguistics and Phonetics, Vol. 21, nº. 2, pp. 129-145

Abstract Views

Metrics Loading ...

Metrics powered by PLOS ALM


  • There are currently no refbacks.


Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Protocol: Is there agreement or disagreement between the absolute and relative impact indices obtained from the Web of Science and Scopus data?
Juan A. Marin-Garcia, Rafaela Alfalla-Luque
WPOM-Working Papers on Operations Management  vol: 9  issue: 1  first page: 53  year: 2018  
doi: 10.4995/wpom.v9i1.8989

This journal is licensed under a Creative Commons Attribution 4.0 International License.

Universitat Politècnica de València

e-ISSN: 1989-9068