Protocol: Comparing advantages and disadvantages of Rating Scales, Behavior Observation Scales and Paired Comparison Scales for behavior assessment of competencies in workers. A systematic literature review


  • Juan A. Marin-Garcia Universitat Politècnica de València
  • Lucia Ramirez Bayarri Department d'Ensenyament de la Generalitat de Catalunya
  • Lorena Atares-Huerta Universitat Politècnica de València



Protocol, Systematic literature review, Rating Scales, Behavior Observation Scales, Paired Comparison Scales, behavior assessment


This is the protocol for a review and there is no abstract. The objectives are as follows: Identify the characteristics of each of the types of scale and how they differ from each other. Estimate the extent to which they are used by organizations in the assessment of skills of their employees. Summarize the advantages and disadvantages of each. Propose which of them would be more appropriate for assessing the competence of innovation in people (performance evaluation, promotion of workers, recruitment, etc.) and the mode of administration (self, peer, expert assessment).



Download data is not yet available.

Author Biographies

Juan A. Marin-Garcia, Universitat Politècnica de València

Lorena Atares-Huerta, Universitat Politècnica de València

Departamento Tecnología de Alimentos


Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32(2), 153-170.

Boyatzis, R. E. (2008). Competencies in the 21st century. Journal of management development, 27(1), 5-12.


Bryman, A., & Bell, E. (2011). Business Research Methods. USA: Oxford University Press.

Charmaz, K. (2006). Constructing grounded theory. A practical guide through qualitative analysis. London: SAGE.

Dolan, S. L., Valle Cabrera, R., Jackson, S. E., & Schuler, R. S. (2007). La gestión de los recursos humanos. Cómo atraer, retener y desarrollar con éxito el capital humano en tiempos de transformación. Madrid: Mcgraw-Hill.

Dowdy, E., Twyford, J., & Sharkey, J. D. (2013). Methods of Assessing Behavior: Observations and Rating Scales. In D. H. Saklofske, V. L. Schwean, & C. R. Reynolds (Eds.), The Oxford Handbook of Child Psychological Assessment (pp. 623-650): Oxford University Press.

DuVernet, A. M., Dierdorff, E. C., & Wilson, M. A. (2015). Exploring Factors That Influence Work Analysis Data: A Meta-Analysis of Design Choices, Purposes, and Organizational Context. Journal of Applied Psychology.

Friese, S. (2012). Qualitative Data Analysis with ATLAS.ti. London: SAGE Publications Ltd.

Groth-Marnat, G. (2003). Handbook of psichological assessment. Hoboken, New Jersey: John Wiley & Sons, Inc.

Hartig, J. (2008). Psychometric models for the assessment of competencies. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 69-90). Guttingen: Hogrefe & Huber Publishers.

Hatzinger, R., & Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, 48(10). Retrieved from

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89.

Heidemeier, H., & Moser, K. (2009). Self-Other Agreement in Job Performance Ratings: A Meta-Analytic Test of a Process Model. Journal of Applied Psychology, 94(2), 353-370.

Hoyt, W. T., & Kerns, M. D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4(4), 403-424.

Lenburg, C. (1999). The Framework, Concepts and Methods of the Competency Outcomes and Performance Assessment (COPA) Model Online Journal of Issues in Nursing, 4(2).

Lohmann, A., & Prumper, J. (2006). Questionnaire for direct participation in the office (FdP-B) - results concerning its reliability and validity. Zeitschrift fur Arbeits-und Organisationspsychologie, 50(3), 119-134. Retrieved from ://000239034600001

Marin-Garcia, J. A., Aragonés Belgran, P., & Melón, G. (2014). Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students. Working Papers on Operations Management, 5(2), 24-46.

Marin-Garcia, J. A., Garcia-Sabater, J. J., Maheut, J., Valero-Herrero, M., & Andres-Romano, C. (2012). Gestión de recursos humanos para ingenieros de la rama industrial. Harlow: Pearson Education.

Marin-Garcia, J. A., Perez-Peñalver, M. J., & Watts, F. (2013). How to assess innovation competence in services: The case of university students. Direccion y Organizacion(50), 48-62. Retrieved from

Marin-Garcia, J. A., Ramirez Bayarri, L., & Andreu Andres, M. A. (2015). Comparación de los métodos de escalas y frecuencia de comportamiento para valorar la competencia de innovación. El punto de vista de alumnos y profesor en el caso de una asignatura de máster. Paper presented at the Congreso In-Red 2015-Universitat Politècnica de València.

Marin-Garcia, J. A., & Santandreu-Mascarell, C. (2015). What do we know about rubrics used in higher education? Intangible Capital, 11(1), 118-145.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The, P. G. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7), e1000097.

Moore, D. R., Cheng, M. I., & Dainty, A. R. J. (2002). Competence, competency and competencies: performance assessment in organisations. Work Study, 51(6), 314-319.

Rowe, C. (1995). Clarifying the use of competence and competency models in recruitment, assessment and staff development. Industrial and Commercial Training, 27(11), 12-17.

Saaty, T. L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill.

Saaty, T. L. (1996). Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, PA.: RWS Publication.

Tziner, A., & Kopelman, R. E. (2002). Is there a preferred performance rating format? A non-psychometric perspective. Applied Psychology, 51(3), 479-503.

Voskuijl, O. F., & Van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: A meta-analysis. European Journal of Psychological Assessment, 18(1), 52-62.




How to Cite

Marin-Garcia, J. A., Ramirez Bayarri, L., & Atares-Huerta, L. (2015). Protocol: Comparing advantages and disadvantages of Rating Scales, Behavior Observation Scales and Paired Comparison Scales for behavior assessment of competencies in workers. A systematic literature review. WPOM-Working Papers on Operations Management, 6(2), 49–63.



SLR protocol

Most read articles by the same author(s)

1 2 3 4 5 > >>