Protocol: Comparing advantages and disadvantages of Rating Scales, Behavior Observation Scales and Paired Comparison Scales for behavior assessment of competencies in workers. A systematic literature review

Juan A. Marin-Garcia, Lucia Ramirez Bayarri, Lorena Atares-Huerta


This is the protocol for a review and there is no abstract. The objectives are as follows: Identify the characteristics of each of the types of scale and how they differ from each other. Estimate the extent to which they are used by organizations in the assessment of skills of their employees. Summarize the advantages and disadvantages of each. Propose which of them would be more appropriate for assessing the competence of innovation in people (performance evaluation, promotion of workers, recruitment, etc.) and the mode of administration (self, peer, expert assessment).



Protocol; Systematic literature review; Rating Scales; Behavior Observation Scales; Paired Comparison Scales; behavior assessment

Full Text:



Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32(2), 153-170.

Boyatzis, R. E. (2008). Competencies in the 21st century. Journal of management development, 27(1), 5-12.


Bryman, A., & Bell, E. (2011). Business Research Methods. USA: Oxford University Press.

Charmaz, K. (2006). Constructing grounded theory. A practical guide through qualitative analysis. London: SAGE.

Dolan, S. L., Valle Cabrera, R., Jackson, S. E., & Schuler, R. S. (2007). La gestión de los recursos humanos. Cómo atraer, retener y desarrollar con éxito el capital humano en tiempos de transformación. Madrid: Mcgraw-Hill.

Dowdy, E., Twyford, J., & Sharkey, J. D. (2013). Methods of Assessing Behavior: Observations and Rating Scales. In D. H. Saklofske, V. L. Schwean, & C. R. Reynolds (Eds.), The Oxford Handbook of Child Psychological Assessment (pp. 623-650): Oxford University Press.

DuVernet, A. M., Dierdorff, E. C., & Wilson, M. A. (2015). Exploring Factors That Influence Work Analysis Data: A Meta-Analysis of Design Choices, Purposes, and Organizational Context. Journal of Applied Psychology.

Friese, S. (2012). Qualitative Data Analysis with ATLAS.ti. London: SAGE Publications Ltd.

Groth-Marnat, G. (2003). Handbook of psichological assessment. Hoboken, New Jersey: John Wiley & Sons, Inc.

Hartig, J. (2008). Psychometric models for the assessment of competencies. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 69-90). Guttingen: Hogrefe & Huber Publishers.

Hatzinger, R., & Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, 48(10). Retrieved from

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89.

Heidemeier, H., & Moser, K. (2009). Self-Other Agreement in Job Performance Ratings: A Meta-Analytic Test of a Process Model. Journal of Applied Psychology, 94(2), 353-370.

Hoyt, W. T., & Kerns, M. D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4(4), 403-424.

Lenburg, C. (1999). The Framework, Concepts and Methods of the Competency Outcomes and Performance Assessment (COPA) Model Online Journal of Issues in Nursing, 4(2).

Lohmann, A., & Prumper, J. (2006). Questionnaire for direct participation in the office (FdP-B) - results concerning its reliability and validity. Zeitschrift fur Arbeits-und Organisationspsychologie, 50(3), 119-134. Retrieved from ://000239034600001

Marin-Garcia, J. A., Aragonés Belgran, P., & Melón, G. (2014). Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students. Working Papers on Operations Management, 5(2), 24-46.

Marin-Garcia, J. A., Garcia-Sabater, J. J., Maheut, J., Valero-Herrero, M., & Andres-Romano, C. (2012). Gestión de recursos humanos para ingenieros de la rama industrial. Harlow: Pearson Education.

Marin-Garcia, J. A., Perez-Peñalver, M. J., & Watts, F. (2013). How to assess innovation competence in services: The case of university students. Direccion y Organizacion(50), 48-62. Retrieved from

Marin-Garcia, J. A., Ramirez Bayarri, L., & Andreu Andres, M. A. (2015). Comparación de los métodos de escalas y frecuencia de comportamiento para valorar la competencia de innovación. El punto de vista de alumnos y profesor en el caso de una asignatura de máster. Paper presented at the Congreso In-Red 2015-Universitat Politècnica de València.

Marin-Garcia, J. A., & Santandreu-Mascarell, C. (2015). What do we know about rubrics used in higher education? Intangible Capital, 11(1), 118-145.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The, P. G. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7), e1000097.

Moore, D. R., Cheng, M. I., & Dainty, A. R. J. (2002). Competence, competency and competencies: performance assessment in organisations. Work Study, 51(6), 314-319.

Rowe, C. (1995). Clarifying the use of competence and competency models in recruitment, assessment and staff development. Industrial and Commercial Training, 27(11), 12-17.

Saaty, T. L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill.

Saaty, T. L. (1996). Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, PA.: RWS Publication.

Tziner, A., & Kopelman, R. E. (2002). Is there a preferred performance rating format? A non-psychometric perspective. Applied Psychology, 51(3), 479-503.

Voskuijl, O. F., & Van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: A meta-analysis. European Journal of Psychological Assessment, 18(1), 52-62.

Abstract Views

Metrics Loading ...

Metrics powered by PLOS ALM


  • There are currently no refbacks.


Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Escala para Evaluar Artículos Científicos en Ciencias Sociales y Humanas- EACSH
Emerson López-López, Sergio Tobón, Luis Gibrán Juárez-Hernández
REICE. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación  vol: 17  issue: 4  first page: 111  year: 2019  
doi: 10.15366/reice2019.17.4.006

This journal is licensed under a Creative Commons Attribution 4.0 International License.

Universitat Politècnica de València

e-ISSN: 1989-9068