Protocol: Is there agreement or disagreement between the absolute and relative impact indices obtained from the Web of Science and Scopus data?

Juan A. Marin-Garcia, Rafaela Alfalla-Luque


This protocol justifies the relevance of the research questions that will be posed in future research that will address whether there are differences between the classifications of scientific journals depending on the impact factor indicators that are used: WoS (JCR ), and Scopus (CiteScore and SJR) and, if any, to what may be due. At the same time, the process to be followed to capture and analyze the data that will allow us to respond to the stated objectives is described in detail. Finally, the results of a pilot analysis focused on the 21 top journals of International Human Resources Management according to Caligiuri (1999) are presented. In it, we can verify that the correlation between JCR and CiteScore is practically perfect and that the quartiles of Scopus represent in a much more appropriate way than those of JCR the classification of these 21 as top journals of the category. It is still pending to carry out the complete investigation to verify if these results are generalizable to the field of economic and business sciences in global or to other sub-areas of this field.


Protocol; journal; ranking; citation analysis

Full Text:



Aghaei Chadegani, A., Salehi, H., Md Yunus, M., Farhadi, H., Fooladi, M., Farhadi, M., & Ale Ebrahim, N. (2013). A comparison between two main academic literature collections: Web of science and scopus databases. Asian Social Science, 9(5), 19-26.

Akbulut, M., & Akca, S. (2017) Popularity or prestige?: The case of jasist rank. Journal of Scientometric Research, 6(3), 141-150.

Ameen, E. C., & Guffey, D. M. (2017). A citation analysis and review of research issues and methodologies in advances in accounting education: Teaching and curriculum innovations Advances in Accounting Education: Teaching and Curriculum Innovations (Vol. 21, pp. 1-25)

ANECA. (Noviembre 2017). Méritos evaluables para la acreditación nacional para el acceso a los cuerpos docentes universitarios: (accedido 29 enero 2017)

Balatsky, E. V., & Ekimova, N. A. (2017). Competition of economic journals in russia: Results of the three waves of ranking. Ekonomicheskaya Politika, 12(6), 178-201.

Bar-Ilan, J. (2008). Which h-index? - a comparison of wos, scopus and google scholar. Scientometrics, 74(2), 257-271.

Bar-Ilan, J., Levene, M., & Lin, A. (2007). Some measures for comparing citation databases. Journal of Informetrics,

Barker, K. (2007). The uk research assessment exercise: The evolution of a national research evaluation system. Research Evaluation, 16(1), 3-12.

Barman, S., Hanna, M. D., & LaForge, R. L. (2001). Perceived relevance and quality of pom journals: A decade later. Journal of Operations Management, 19(3), 367-385.

Bartol, T., Budimir, G., Dekleva-Smrekar, D., Pusnik, M., & Juznic, P. (2014). Assessment of research fields in scopus and web of science in the view of national research evaluation in slovenia. Scientometrics, 98(2), 1491-1504.

Basu, A., Banshal, S. K., Singhal, K., & Singh, V. K. (2016). Designing a composite index for research performance evaluation at the national or regional level: Ranking central universities in india. Scientometrics, 107(3), 1171-1193.

Batiz-Lazo, B., Eskandari, R., & Goddard, J. (2015). Online publishing and citation success in the accounting, business and economic history of spain, 1997-2011. Investigaciones De Historia Economica, 11(3), 153-163.

Baumgartner, H., & Pieters, R. (2003). The structural influence of marketing journals: A citation analysis of the discipline and its subareas over time. Journal of marketing, 67(2), 123-139.

Beed, C., & Beed, C. (1996). Measuring the quality of academic journals: The case of economics. Journal of Post Keynesian Economics, 18(3), 369-396.

Beets, S. D., Lewis, B. R., & Brower, H. H. (2016). The quality of business ethics journals: An assessment based on application. Business & Society, 55(2), 188-213.

Bettencourt, L. A., & Houston, M. B. (2001). The impact of article method type and subject area on article citations and reference diversity in jm, jmr, and jcr. Marketing Letters, 12(4), 327-340.

Bonner, S. E., Hesford, J. W., Van der Stede, W. A., & Young, S. M. (2006). The most influential journals in academic accounting. Accounting, Organizations and Society, 31(7), 663-685.

Bontis, N., & Serenko, A. (2009). A follow-up ranking of academic journals. Journal of Knowledge Management, 13(1), 16-26.

Brown, L. D., & Gardner, J. C. (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research (car). Journal of Accounting Research, 23(1), 84-109.

Caligiuri, P. M. (1999). The ranking of scholarly journals in international human resource management. International Journal of Human Resource Management, 10(3), 515-519.

Cantin, M., Munoz, M., & Roa, I. (2015). Comparison between impact factor, eigenfactor score, and scimago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183-1188.

Carreras, J. (2011). Evaluation of teaching and faculty promotion (viii). University spanish legislation (f): Quality of teaching and complementary retributions (1st part). Educacion Medica, 14(4), 199- 205.

Chan, K. C., Chan, K. C., Seow, G. S., & Tam, K. (2009). Ranking accounting journals using dissertation citation analysis: A research note. Accounting, Organizations and Society, 34(6-7), 875-885.

Chan, K. C., Fung, H. G., & Yau, J. (2013). Predominant sources and contributors of influential business ethics research: Evidence and implications from a threshold citation analysis. Business Ethics-a European Review, 22(3), 263-276.

Chan, K. C., Lai, P., & Liano, K. (2012). A threshold citation analysis in marketing research. European Journal of Marketing, 46(1), 134-156.

Chan, K. C., & Liano, K. (2009). Threshold citation analysis of influential articles, journals, institutions and researchers in accounting. Accounting and Finance, 49(1), 59-74.

Chapman, K., & Ellinger, A. E. (2009). Constructing impact factors to measure the influence of supply chain management and logistics journals. Journal of Business Logistics, 30(2), 197-+.

Chatha, K. A., Butt, I., & Tariq, A. (2015). Research methodologies and publication trends in manufacturing strategy a content analysis based literature review. International Journal of Operations & Production Management, 35(4), 487-546.

Chow, C. W., Haddad, K., Singh, G., & Wu, A. (2008). A citation analysis of the top-ranked management and financial accounting journals. International Journal of Managerial and Financial Accounting, 1(1), 29-44.

Delgado, F. J., & Fernández-Llera, R. (2012). Sobre la evaluación del profesorado universitario (especial referencia a ciencias económicas y jurídicas). 2012, 35(2), 15.

Di Vaio, G., & Weisdorf, J. L. (2010). Ranking economic history journals: A citation-based impact-adjusted analysis. Cliometrica, 4(1), 1-17.

Ding, Y., & Cronin, B. (2011). Popular and/or prestigious? Measures of scholarly esteem. Information Processing & Management, 47(1), 80-96.

DuBois, F. L., & Reeb, D. (2000). Ranking the international business journals. Journal of International Business Studies, 31(4), 689-704.

Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of scimago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623-2628.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.

Fu, S. C., & Chan, K. M. (2014). Ranking of orthopaedic journals: A challenge to the citation-based metrics. Journal of Orthopaedic Translation, 2(3), 131-138.

Galán, A., & Zych, I. (2011). Analysis of the criteria of the national commission for the evaluation of research activity (cneai) for the concession of the research incentives in education. Bordon, 63(2), 117-139.

García-Lillo, F., Úbeda-García, M., & Marco-Lajara, B. (2015). The intellectual structure of human resource management research: A bibliometric study of the international journal of human resourcemanagement, 2000-2012. Revista Europea de Direccion y Economia de la Empresa, 24(3), 149-161.

Garcia-Pachon, E., & Arencibia-Jorge, R. (2014). A comparison of the impact factor and the scimago journal rank index in respiratory system journals. Archivos De Bronconeumologia, 50(7), 308-309.

Haddad, K., Singh, G., Sciglimpaglia, D., & Chan, H. (2014). To what extent do articles published in other than "top journals" have impact on marketing? European Journal of Marketing, 48(1-2), 271-287.

Harris, C. (2008). Ranking the management journals. Journal of Scholarly Publishing, 39(4), 373-409.

Harzing, A.-W. (2008). On becoming a high impact jour al in international business and management. European Journal of International Management, 2(2), 115-118.

Harzing, A.-W., & van der Wal, R. (2009). A google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41-46.

Harzing, A. W., & Alakangas, S. (2016). Google scholar, scopus and the web of science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787-804.

Hole, A. R. (2017). Ranking economics journals using data from a national research evaluation exercise. Oxford Bulletin of Economics and Statistics, 79(5), 621-636.

Hoobler, J. M., & Johnson, N. B. (2004). An analysis of current human resource management publications. Personnel Review, 33(6), 665-676.

Jiménez-Contreras, E., Robinson-García, N., & Cabezas-Clavijo, A. (2011). Productivity and impact of spanish researchers: Reference thresholds within scientific areas. Revista Espanola De Documentacion Cientifica, 34(4), 505-525.

Jobber, D., & Simpson, P. (1 88). A citation analysis of selected marketing journals. International Journal of Research in Marketing, 5(2), 137-142.

Kastrinos, N., & Katsoulacos, Y. (1995). Towards a national system of research evaluation in greece. Research Evaluation, 5(1), 63-68.

Kianifar, H., Sadeghi, R., & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and scimago journal rank indicator of pediatric neurology journals. Acta informatica medica: AIM: journal of the Society for Medical Informatics of Bosnia & Herzegovina: casopis Drustva za medicinsku informatiku BiH, 22(2), 103-106.

Law, R., & Van Der Veen, R. (2008). The popularity of prestigious hospitality journals: A google scholar approach. International Journal of Contemporary Hospitality Management, 20(2), 113-125.

Leydesdorff, L., & Opthof, T. (2010). Scopus's source normalized impact per paper (snip) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61(11), 2365-2369.

Losilla, J. M., Navarro, J. B., Palmer, A., Rodrigo, M. F., & Ato, M. (2005). Análisis de datos. Del contraste de hipótesis al modelado estadístico. Barcelona: Edicions a Petició.

Lowe, A., & Locke, J. (2005). Perceptions of journal quality and research paradigm: Results of a web- based survey of british accounting academics. Accounting Organizations and Society, 30(1), 81-98.

Marin-Garcia, J. A., Aragonés Beltran, P., & Melón, G. (2014). Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students. WPOM-Working Papers on Operations Management, 5(2), 24-46.

Markoulli, M., Lee, C. I. S. G., Byington, E., & Felps, W. A. (2017). Mapping human resource management: Reviewing the field and charting future directions. Human Resource Management Review, 27(3), 367-396.

Marler, J. H., & Boudreau, J. W. (2017). An evidence-based review of hr analytics. International Journal of Human Resource Management, 28(1), 3-26.

Medina-López, C., Alfalla-Luque, R., & Marin-Garcia, J. A. (2011). Research in operations management teaching: Trends and challenges. Intangible Capital, 7(2), 507-548.

Medina-López, C., Marin-Garcia, J. A., & Alfalla-Luque, R. (2012). Una sistematización de los instrumentos de medida de la calidad de la investigación (a systematization of instruments to measure the quality of research). WPOM-Working Papers on Operations Management, 3(1), 28- 45.

Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of lis faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105-2125.

Merigo, J. M., Rocafort, A., & Aznar-Alarcon, J. P. (2016). Bibliometric overview of business & economics research. Journal of Business Economics and Management, 17(3), 397-413.

Mingers, J., & Leydesdorff, L. (2015). Identifying research fields within business and management: A journal cross-citation analysis. Journal of the Operational Research Society, 66(8), 1370-1384.

Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of web of science and scopus: A comparative analysis. Scientometrics, 106(1), 213-228.

Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the google scholar-based hg-index. Journal of Informetrics, 4(1), 107-117.

O'Leary, D. (2008). On the relationship between citations and appearances on “top 25” download lists in the international journal of accounting information systems. International Journal of Accounting Information Systems, 9(1), 61-75.

Olson, J. E. (2005). Top-25-business-school professors rate journals in operations management and related fields. Interfaces, 35(4), 323-338.

Petersen, C. G., Aase, G. R., & Heiser, D. R. (2011). Journal ranking analyses of operations management research. International Journal of Operations & Production Management, 31(4), 405-422.

Pindek, S., Kessler, S. R., & Spector, P. E. (2017). A quantitative and qualitative review of what meta-analyses have contributed to our understanding of human resource management. Human Resource Management Review, 27(1), 26-38.

Pislyakov, V. (2009). Comparing two “thermometers”: Impact factors of 20 leading economic journals according to journal citation reports and scopus. Scientometrics, 79(3), 541-550.

Quintas-Froufe, N. (2016). Quality indicators for scientific publications in the social sciences in spain: A comparative analysis between evaluation agencies. Revista de Investigación Educativa, 34(1), 259-272.

Ramin, S., & Sarraf Shirazi, A. (2012). Comparison between impact factor, scimago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear medicine review. Central& Eastern Europe, 15( ), 132-136.

Rao, S., Iyengar, D., & Goldsby, T. J. (2013). On the measurement and benchmarking of research impact among active logistics scholars. International Journal of Physical Distribution & Logistics Management, 43(10), 814-832.

Rosenstreich, D., & Wooliscroft, B. (2009). Measuring the impact of accounting journals using google scholar and the g-index. British Accounting Review, 41(4), 227-239.

Rosenstreich, D., & Wooliscroft, B. (2012). Assessing international journal impact: The case of marketing. European Business Review, 24(1), 58- 7

Rosenstreich, D., & Wooliscroft, B. (2013). An improved method for assessing the impact of management journals. Asian Academy of Management Journal, 18(2), 37-54.

Ruiz-Pérez, R., López-Cózar, E. D., & Jiménez-Contreras E. (2010). Principles and criteria used by the national evaluation committee of research activity (cneai-spain) for the assessment of scientific publications: 1989-2009. Psicothema, 22(4), 898-908.

Russ-Eft, D. (2008). Ssci, isi, jcr, jif, if, and journal quality. Human Resource Development Quarterly, 19(3), 185-189.

Salgado, J. F., & Páez, D. (2007). Scientific productivity and hirsch's h index of Spanish social psychology: Convergence between productivity indexes and comparison with other areas. Psicothema, 19(2), 179-189.

Salvador-Oliván, J. A., & Agustín-Lacruz, C. (2015). Correlation between bibliometric indicators in web of science y scopus journals. Revista General de Informacion y Documentacion, 25(2), 341-359.

Schöpfel, J., & Prost, H. (2009). Comparison of scimago journal rank indicator (sjr) with jcr journal impact factor (if) for french journals. Psychologie Francaise, 54(4), 287-305.

Segalla, M. (2009). Will you cite me? The emerging strategy of academic publishing. European Journal of International Management, 3(4), 411-418.

Serenko, A., & Bontis, N. (2009). Global ranking of knowledge management and intellectual capital academic journals. Journal of Knowledge Management, 13(1), 4-15.

Serenko, A., & Bontis, N. (2013). Global ranking of knowledge management and intellectual capital academic journals: 2013 update. Journal of Knowledge Management, 17(2), 307-326.

Serenko, A., & Bontis, N. (2017). Global ranking of knowledge management and intellectual capital academic journals: 2017 update. Journal of Knowledge Management, 21(3), 675-692.

Shugan, S. M. (2003). Editorial: Journal rankings: Save the outlets for your research. Marketing Science, 22(4), 437-441.

Sicilia, M. A., Sanchez-Alonso, S., & Garcia-Barriocanal, E. (2011). Comparing impact factors from two different citation databases: The case of computer science. Journal of Informetrics, 5, 698-704.

Siebelt, M., Siebelt, T., Pilot, P., Bloem, R. M., Bhandari, M., & Poolman, R. W. (2010). Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for impact factor and scimago. BMC Muscul skeletal Disorders, 11.

Stonebraker, J. S., Gil, E., Kirkwood, C. W., & Handfield, R. B. (2012). Impact factor as a metric to assess journals where om research is published. Journal of Operations Management, 30(1-2), 24-43.

Svensson, G. (2006). The paradoxnoia of top journal(s) of top in marketing. European Journal of Marketing, 40(11-12), 1153-1168.

Tadajewski, M. (2016). Academic labour, journal ranking lists and the politics of knowledge production in marketing. Journal of Marketing Management, 32(1-2), 1-18.

Theussl, S., Reutterer, T., & Hornik, K. (2014). How to derive consensus among various marketing journal rankings? Journal of Business Research, 67(5), 998-1006.

Tsay, M. Y. (2012). An overview of the journal impact indicators. Journal of Educational Media and Library Science, 49(2).

Viladrich Segués, M. C., & Doval Dieguez, E. (2011). Medición: Fiabilidad y validez. Bellaterra: Laboratori d'Estadística Aplicada i de Modelització (UAB).

Vokurka, R. J. (1996). The relative importance of journals used in operations management research: A citation analysis. Journal of Operations Management, 14(4), 345-355.

Walters, W. H. (2017). Citation-based journal rankings: Key questions, metrics, and data sources. IEEE Access, 5, 22036-22053.

Zarifmahmoudi, L., Jamali, J., & Sadeghi, R. (2015). Google scholar journal metrics: Comparison with impact factor and scimago journal rank indicator for nuclear medicine journals. Iranian Journal of Nuclear Medicine, 23(1), 8-14.

Zhou, Y. B., Lü, L., & Li, M. (2012). Quantifying the influence of scientists and their publications: Distinguishing between prestige and popularity. New Journal of Physics, 14. -

Zinkhan, G. M., & Leigh, T. W. (1999). Assessing the quality ranking of the journal of advertising, 1986- 1997. Journal of Advertising, 28(2), 51-70.

Zupic, I., & Cater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18(3), 429-472.

Abstract Views

Metrics Loading ...

Metrics powered by PLOS ALM


  • There are currently no refbacks.


Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Avances de los investigadores españoles de Dirección de Operaciones en el año 2019: el caso de los miembros de ACEDEDOT
Lidia Sanchez-Ruiz, Elsa Diez-Busto
WPOM-Working Papers on Operations Management  vol: 11  issue: 1  first page: 54  year: 2020  
doi: 10.4995/wpom.v11i1.13637

This journal is licensed under a Creative Commons Attribution 4.0 International License.

Universitat Politècnica de València

e-ISSN: 1989-9068