Protocol: Comparing advantages and disadvantages of Rating Scales, Behavior Observation Scales and Paired Comparison Scales for behavior assessment of competencies in workers. A systematic literature review

Juan A. Marin-Garcia; Lucia Ramirez Bayarri; Lorena Atares-Huerta

doi:10.4995/wpom.v6i2.4032

Authors

Juan A. Marin-Garcia Universitat Politècnica de València https://orcid.org/0000-0001-5416-3938
Lucia Ramirez Bayarri Department d'Ensenyament de la Generalitat de Catalunya
Lorena Atares-Huerta Universitat Politècnica de València

DOI:

https://doi.org/10.4995/wpom.v6i2.4032

Keywords:

Protocol, Systematic literature review, Rating Scales, Behavior Observation Scales, Paired Comparison Scales, behavior assessment

Abstract

This is the protocol for a review and there is no abstract. The objectives are as follows: Identify the characteristics of each of the types of scale and how they differ from each other. Estimate the extent to which they are used by organizations in the assessment of skills of their employees. Summarize the advantages and disadvantages of each. Propose which of them would be more appropriate for assessing the competence of innovation in people (performance evaluation, promotion of workers, recruitment, etc.) and the mode of administration (self, peer, expert assessment).

Downloads

Download data is not yet available.

Author Biographies

Juan A. Marin-Garcia, Universitat Politècnica de València

ResearcherID A-4069-2011 Scopus Author ID 14024595300

Lorena Atares-Huerta, Universitat Politècnica de València

Departamento Tecnología de Alimentos

References

Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32(2), 153-170. https://doi.org/10.1016/j.stueduc.2006.04.006

Boyatzis, R. E. (2008). Competencies in the 21st century. Journal of management development, 27(1), 5-12. https://doi.org/10.1108/02621710810840730

Bretz, R. D., Milkovich, G. T., & Read, W. (1992). THE CURRENT STATE OF PERFORMANCE-APPRAISAL RESEARCH AND PRACTICE - CONCERNS, DIRECTIONS, AND IMPLICATIONS. Journal of Management, 18(2), 321-352. https://doi.org/10.1177/014920639201800206

Bryman, A., & Bell, E. (2011). Business Research Methods. USA: Oxford University Press.

Charmaz, K. (2006). Constructing grounded theory. A practical guide through qualitative analysis. London: SAGE.

Dolan, S. L., Valle Cabrera, R., Jackson, S. E., & Schuler, R. S. (2007). La gestión de los recursos humanos. Cómo atraer, retener y desarrollar con éxito el capital humano en tiempos de transformación. Madrid: Mcgraw-Hill.

Dowdy, E., Twyford, J., & Sharkey, J. D. (2013). Methods of Assessing Behavior: Observations and Rating Scales. In D. H. Saklofske, V. L. Schwean, & C. R. Reynolds (Eds.), The Oxford Handbook of Child Psychological Assessment (pp. 623-650): Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199796304.013.0026

DuVernet, A. M., Dierdorff, E. C., & Wilson, M. A. (2015). Exploring Factors That Influence Work Analysis Data: A Meta-Analysis of Design Choices, Purposes, and Organizational Context. Journal of Applied Psychology. https://doi.org/10.1037/a0039084

Friese, S. (2012). Qualitative Data Analysis with ATLAS.ti. London: SAGE Publications Ltd.

Groth-Marnat, G. (2003). Handbook of psichological assessment. Hoboken, New Jersey: John Wiley & Sons, Inc.

Hartig, J. (2008). Psychometric models for the assessment of competencies. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 69-90). Guttingen: Hogrefe & Huber Publishers.

Hatzinger, R., & Dittrich, R. (2012). Prefmod: An R package for modeling preferences based on paired comparisons, rankings, or ratings. Journal of Statistical Software, 48(10). Retrieved from http://www.scopus.com/inward/record.url?eid=2-s2.0-84863314424&partnerID=40&md5=fe41f891ec6e3f17584078fe2d73b5ef

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89. https://doi.org/10.1080/19312450709336664

Heidemeier, H., & Moser, K. (2009). Self-Other Agreement in Job Performance Ratings: A Meta-Analytic Test of a Process Model. Journal of Applied Psychology, 94(2), 353-370. https://doi.org/10.1037/0021-9010.94.2.353

Hoyt, W. T., & Kerns, M. D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4(4), 403-424. https://doi.org/10.1037//1082-989X.4.4.403

Lenburg, C. (1999). The Framework, Concepts and Methods of the Competency Outcomes and Performance Assessment (COPA) Model Online Journal of Issues in Nursing, 4(2).

Lohmann, A., & Prumper, J. (2006). Questionnaire for direct participation in the office (FdP-B) - results concerning its reliability and validity. Zeitschrift fur Arbeits-und Organisationspsychologie, 50(3), 119-134. Retrieved from ://000239034600001 https://doi.org/10.1026/0932-4089.50.3.119

Marin-Garcia, J. A., Aragonés Belgran, P., & Melón, G. (2014). Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students. Working Papers on Operations Management, 5(2), 24-46. https://doi.org/10.4995/wpom.v5i2.3220

Marin-Garcia, J. A., Garcia-Sabater, J. J., Maheut, J., Valero-Herrero, M., & Andres-Romano, C. (2012). Gestión de recursos humanos para ingenieros de la rama industrial. Harlow: Pearson Education.

Marin-Garcia, J. A., Perez-Peñalver, M. J., & Watts, F. (2013). How to assess innovation competence in services: The case of university students. Direccion y Organizacion(50), 48-62. Retrieved from http://www.revistadyo.com/index.php/dyo/article/viewFile/431/451

Marin-Garcia, J. A., Ramirez Bayarri, L., & Andreu Andres, M. A. (2015). Comparación de los métodos de escalas y frecuencia de comportamiento para valorar la competencia de innovación. El punto de vista de alumnos y profesor en el caso de una asignatura de máster. Paper presented at the Congreso In-Red 2015-Universitat Politècnica de València. https://doi.org/10.4995/INRED2015.2015.1556

Marin-Garcia, J. A., & Santandreu-Mascarell, C. (2015). What do we know about rubrics used in higher education? Intangible Capital, 11(1), 118-145.

https://doi.org/10.3926/ic.538

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The, P. G. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

Moore, D. R., Cheng, M. I., & Dainty, A. R. J. (2002). Competence, competency and competencies: performance assessment in organisations. Work Study, 51(6), 314-319. https://doi.org/10.1108/00438020210441876

Rowe, C. (1995). Clarifying the use of competence and competency models in recruitment, assessment and staff development. Industrial and Commercial Training, 27(11), 12-17. https://doi.org/10.1108/00197859510100257

Saaty, T. L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill. https://doi.org/10.21236/ADA214804

Saaty, T. L. (1996). Decision Making with Dependence and Feedback: The Analytic Network Process. Pittsburgh, PA.: RWS Publication.

Tziner, A., & Kopelman, R. E. (2002). Is there a preferred performance rating format? A non-psychometric perspective. Applied Psychology, 51(3), 479-503. https://doi.org/10.1111/1464-0597.00104

Voskuijl, O. F., & Van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: A meta-analysis. European Journal of Psychological Assessment, 18(1), 52-62. https://doi.org/10.1027//1015-5759.18.1.52