Protocol: Is there agreement or disagreement between the absolute and relative impact indices obtained from the Web of Science and Scopus data?

Juan A. Marin-Garcia, Rafaela Alfalla-Luque

Abstract

This protocol justifies the relevance of the research questions that will be posed in future research that will address whether there are differences between the classifications of scientific journals depending on the impact factor indicators that are used: WoS (JCR ), and Scopus (CiteScore and SJR) and, if any, to what may be due. At the same time, the process to be followed to capture and analyze the data that will allow us to respond to the stated objectives is described in detail. Finally, the results of a pilot analysis focused on the 21 top journals of International Human Resources Management according to Caligiuri (1999) are presented. In it, we can verify that the correlation between JCR and CiteScore is practically perfect and that the quartiles of Scopus represent in a much more appropriate way than those of JCR the classification of these 21 as top journals of the category. It is still pending to carry out the complete investigation to verify if these results are generalizable to the field of economic and business sciences in global or to other sub-areas of this field.


Keywords

Protocol; journal; ranking; citation analysis

Full Text:

PDF

References

Aghaei Chadegani, A., Salehi, H., Md Yunus, M., Farhadi, H., Fooladi, M., Farhadi, M., & Ale Ebrahim, N. (2013). A comparison between two main academic literature collections: Web of science and scopus databases. Asian Social Science, 9(5), 19-26. https://doi.org/10.5539/ass.v9n5p18

Akbulut, M., & Akca, S. (2017) Popularity or prestige?: The case of jasist rank. Journal of Scientometric Research, 6(3), 141-150. https://doi.org/10.5530/jscires.6.3.21

Ameen, E. C., & Guffey, D. M. (2017). A citation analysis and review of research issues and methodologies in advances in accounting education: Teaching and curriculum innovations Advances in Accounting Education: Teaching and Curriculum Innovations (Vol. 21, pp. 1-25) https://doi.org/10.1108/S1085-462220170000021002

ANECA. (Noviembre 2017). Méritos evaluables para la acreditación nacional para el acceso a los cuerpos docentes universitarios: https://www.mecd.gob.es/servicios-al-ciudadano-mecd/dms/mecd/servicios-al-ciudadano-mecd/catalogo/general/educacion/academia/ficha/academia/criterios-sociales-juridicas.pdf (accedido 29 enero 2017)

Balatsky, E. V., & Ekimova, N. A. (2017). Competition of economic journals in russia: Results of the three waves of ranking. Ekonomicheskaya Politika, 12(6), 178-201. https://doi.org/10.18288/1994-5124-2017-6-09

Bar-Ilan, J. (2008). Which h-index? - a comparison of wos, scopus and google scholar. Scientometrics, 74(2), 257-271. https://doi.org/10.1007/s11192-008-0216-y

Bar-Ilan, J., Levene, M., & Lin, A. (2007). Some measures for comparing citation databases. Journal of Informetrics, https://doi.org/10.1016/j.joi.2006.08.001

Barker, K. (2007). The uk research assessment exercise: The evolution of a national research evaluation system. Research Evaluation, 16(1), 3-12. https://doi.org/10.3152/095820207X190674

Barman, S., Hanna, M. D., & LaForge, R. L. (2001). Perceived relevance and quality of pom journals: A decade later. Journal of Operations Management, 19(3), 367-385. https://doi.org/10.1016/S0272-6963(00)00060-7

Bartol, T., Budimir, G., Dekleva-Smrekar, D., Pusnik, M., & Juznic, P. (2014). Assessment of research fields in scopus and web of science in the view of national research evaluation in slovenia. Scientometrics, 98(2), 1491-1504. https://doi.org/10.1007/s11192-013-1148-8

Basu, A., Banshal, S. K., Singhal, K., & Singh, V. K. (2016). Designing a composite index for research performance evaluation at the national or regional level: Ranking central universities in india. Scientometrics, 107(3), 1171-1193. https://doi.org/10.1007/s11192-016-1935-0

Batiz-Lazo, B., Eskandari, R., & Goddard, J. (2015). Online publishing and citation success in the accounting, business and economic history of spain, 1997-2011. Investigaciones De Historia Economica, 11(3), 153-163. https://doi.org/10.1016/j.ihe.2014.07.010

Baumgartner, H., & Pieters, R. (2003). The structural influence of marketing journals: A citation analysis of the discipline and its subareas over time. Journal of marketing, 67(2), 123-139. https://doi.org/10.1509/jmkg.67.2.123.18610

Beed, C., & Beed, C. (1996). Measuring the quality of academic journals: The case of economics. Journal of Post Keynesian Economics, 18(3), 369-396. https://doi.org/10.1080/01603477.1996.11490078

Beets, S. D., Lewis, B. R., & Brower, H. H. (2016). The quality of business ethics journals: An assessment based on application. Business & Society, 55(2), 188-213. https://doi.org/10.1177/0007650313478974

Bettencourt, L. A., & Houston, M. B. (2001). The impact of article method type and subject area on article citations and reference diversity in jm, jmr, and jcr. Marketing Letters, 12(4), 327-340. https://doi.org/10.1023/A:1012272305777

Bonner, S. E., Hesford, J. W., Van der Stede, W. A., & Young, S. M. (2006). The most influential journals in academic accounting. Accounting, Organizations and Society, 31(7), 663-685. https://doi.org/10.1016/j.aos.2005.06.003

Bontis, N., & Serenko, A. (2009). A follow-up ranking of academic journals. Journal of Knowledge Management, 13(1), 16-26. https://doi.org/10.1108/13673270910931134

Brown, L. D., & Gardner, J. C. (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research (car). Journal of Accounting Research, 23(1), 84-109. https://doi.org/10.2307/2490908

Caligiuri, P. M. (1999). The ranking of scholarly journals in international human resource management. International Journal of Human Resource Management, 10(3), 515-519. https://doi.org/10.1080/095851999340468

Cantin, M., Munoz, M., & Roa, I. (2015). Comparison between impact factor, eigenfactor score, and scimago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183-1188. https://doi.org/10.4067/S0717-95022015000300060

Carreras, J. (2011). Evaluation of teaching and faculty promotion (viii). University spanish legislation (f): Quality of teaching and complementary retributions (1st part). Educacion Medica, 14(4), 199- 205. https://doi.org/10.4321/S157-18132011000400002

Chan, K. C., Chan, K. C., Seow, G. S., & Tam, K. (2009). Ranking accounting journals using dissertation citation analysis: A research note. Accounting, Organizations and Society, 34(6-7), 875-885. https://doi.org/10.1016/j.aos.2008.12.002

Chan, K. C., Fung, H. G., & Yau, J. (2013). Predominant sources and contributors of influential business ethics research: Evidence and implications from a threshold citation analysis. Business Ethics-a European Review, 22(3), 263-276. https://doi.org/10.1111/beer.12024

Chan, K. C., Lai, P., & Liano, K. (2012). A threshold citation analysis in marketing research. European Journal of Marketing, 46(1), 134-156. https://doi.org/10.1108/03090561211189211

Chan, K. C., & Liano, K. (2009). Threshold citation analysis of influential articles, journals, institutions and researchers in accounting. Accounting and Finance, 49(1), 59-74. https://doi.org/10.1111/j.1467-629X.2007.00254.x

Chapman, K., & Ellinger, A. E. (2009). Constructing impact factors to measure the influence of supply chain management and logistics journals. Journal of Business Logistics, 30(2), 197-+. https://doi.org/10.1002/j.2158-1592.2009.tb00119.x

Chatha, K. A., Butt, I., & Tariq, A. (2015). Research methodologies and publication trends in manufacturing strategy a content analysis based literature review. International Journal of Operations & Production Management, 35(4), 487-546. https://doi.org/10.1108/IJOPM-07-2012-0285

Chow, C. W., Haddad, K., Singh, G., & Wu, A. (2008). A citation analysis of the top-ranked management and financial accounting journals. International Journal of Managerial and Financial Accounting, 1(1), 29-44. https://doi.org/10.1504/IJMFA.2008.020460

Delgado, F. J., & Fernández-Llera, R. (2012). Sobre la evaluación del profesorado universitario (especial referencia a ciencias económicas y jurídicas). 2012, 35(2), 15. https://doi.org/10.3989/redc.2012.2.861

Di Vaio, G., & Weisdorf, J. L. (2010). Ranking economic history journals: A citation-based impact-adjusted analysis. Cliometrica, 4(1), 1-17. https://doi.org/10.1007/s11698-009-0039-y

Ding, Y., & Cronin, B. (2011). Popular and/or prestigious? Measures of scholarly esteem. Information Processing & Management, 47(1), 80-96. https://doi.org/10.1016/j.ipm.2010.01.002

DuBois, F. L., & Reeb, D. (2000). Ranking the international business journals. Journal of International Business Studies, 31(4), 689-704. https://doi.org/10.1057/palgrave.jibs.8490929

Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of scimago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623-2628. https://doi.org/10.1096/fj.08-107938

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/h0031619

Fu, S. C., & Chan, K. M. (2014). Ranking of orthopaedic journals: A challenge to the citation-based metrics. Journal of Orthopaedic Translation, 2(3), 131-138. https://doi.org/10.1016/j.jot.2014.03.005

Galán, A., & Zych, I. (2011). Analysis of the criteria of the national commission for the evaluation of research activity (cneai) for the concession of the research incentives in education. Bordon, 63(2), 117-139.

García-Lillo, F., Úbeda-García, M., & Marco-Lajara, B. (2015). The intellectual structure of human resource management research: A bibliometric study of the international journal of human resourcemanagement, 2000-2012. Revista Europea de Direccion y Economia de la Empresa, 24(3), 149-161. https://doi.org/10.1016/j.redee.2015.07.001

Garcia-Pachon, E., & Arencibia-Jorge, R. (2014). A comparison of the impact factor and the scimago journal rank index in respiratory system journals. Archivos De Bronconeumologia, 50(7), 308-309. https://doi.org/10.1016/j.arbres.2013.10.006

Haddad, K., Singh, G., Sciglimpaglia, D., & Chan, H. (2014). To what extent do articles published in other than "top journals" have impact on marketing? European Journal of Marketing, 48(1-2), 271-287. https://doi.org/10.1108/EJM-11-2010-0592

Harris, C. (2008). Ranking the management journals. Journal of Scholarly Publishing, 39(4), 373-409. https://doi.org/10.3138/jsp.39.4.373

Harzing, A.-W. (2008). On becoming a high impact jour al in international business and management. European Journal of International Management, 2(2), 115-118. https://doi.org/10.1504/EJIM.2008.017763

Harzing, A.-W., & van der Wal, R. (2009). A google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41-46. https://doi.org/10.1002/asi.20953

Harzing, A. W., & Alakangas, S. (2016). Google scholar, scopus and the web of science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787-804. https://doi.org/10.1007/s11192-015-1798-9

Hole, A. R. (2017). Ranking economics journals using data from a national research evaluation exercise. Oxford Bulletin of Economics and Statistics, 79(5), 621-636. https://doi.org/10.1111/obes.12185

Hoobler, J. M., & Johnson, N. B. (2004). An analysis of current human resource management publications. Personnel Review, 33(6), 665-676. https://doi.org/10.1108/00483480410561547

Jiménez-Contreras, E., Robinson-García, N., & Cabezas-Clavijo, A. (2011). Productivity and impact of spanish researchers: Reference thresholds within scientific areas. Revista Espanola De Documentacion Cientifica, 34(4), 505-525. https://doi.org/10.3989/redc.2011.4.828

Jobber, D., & Simpson, P. (1 88). A citation analysis of selected marketing journals. International Journal of Research in Marketing, 5(2), 137-142. https://doi.org/10.1016/0167-8116(88)90065-1

Kastrinos, N., & Katsoulacos, Y. (1995). Towards a national system of research evaluation in greece. Research Evaluation, 5(1), 63-68. https://doi.org/10.1093/rev/5.1.63

Kianifar, H., Sadeghi, R., & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and scimago journal rank indicator of pediatric neurology journals. Acta informatica medica: AIM: journal of the Society for Medical Informatics of Bosnia & Herzegovina: casopis Drustva za medicinsku informatiku BiH, 22(2), 103-106. https://doi.org/10.5455/aim.2014.22.103-106

Law, R., & Van Der Veen, R. (2008). The popularity of prestigious hospitality journals: A google scholar approach. International Journal of Contemporary Hospitality Management, 20(2), 113-125. https://doi.org/10.1108/09596110810852113

Leydesdorff, L., & Opthof, T. (2010). Scopus's source normalized impact per paper (snip) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61(11), 2365-2369. https://doi.org/10.1002/asi.21371

Losilla, J. M., Navarro, J. B., Palmer, A., Rodrigo, M. F., & Ato, M. (2005). Análisis de datos. Del contraste de hipótesis al modelado estadístico. Barcelona: Edicions a Petició.

Lowe, A., & Locke, J. (2005). Perceptions of journal quality and research paradigm: Results of a web- based survey of british accounting academics. Accounting Organizations and Society, 30(1), 81-98. https://doi.org/10.1016/j.aos.2004.05.002

Marin-Garcia, J. A., Aragonés Beltran, P., & Melón, G. (2014). Intra-rater and inter-rater consistency of pair wise comparison in evaluating the innovation competency for university students. WPOM-Working Papers on Operations Management, 5(2), 24-46. https://doi.org/10.4995/wpom.v5i2.3220

Markoulli, M., Lee, C. I. S. G., Byington, E., & Felps, W. A. (2017). Mapping human resource management: Reviewing the field and charting future directions. Human Resource Management Review, 27(3), 367-396. https://doi.org/10.1016/j.hrmr.2016.10.001

Marler, J. H., & Boudreau, J. W. (2017). An evidence-based review of hr analytics. International Journal of Human Resource Management, 28(1), 3-26. https://doi.org/10.1080/09585192.2016.1244699

Medina-López, C., Alfalla-Luque, R., & Marin-Garcia, J. A. (2011). Research in operations management teaching: Trends and challenges. Intangible Capital, 7(2), 507-548. https://doi.org/10.3926/ic.2011.v7n2.p507-548

Medina-López, C., Marin-Garcia, J. A., & Alfalla-Luque, R. (2012). Una sistematización de los instrumentos de medida de la calidad de la investigación (a systematization of instruments to measure the quality of research). WPOM-Working Papers on Operations Management, 3(1), 28- 45. https://doi.org/10.4995/wpom.v3i1.1104

Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of lis faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105-2125. https://doi.org/10.1002/asi.20677

Merigo, J. M., Rocafort, A., & Aznar-Alarcon, J. P. (2016). Bibliometric overview of business & economics research. Journal of Business Economics and Management, 17(3), 397-413. https://doi.org/10.3846/16111699.2013.807868

Mingers, J., & Leydesdorff, L. (2015). Identifying research fields within business and management: A journal cross-citation analysis. Journal of the Operational Research Society, 66(8), 1370-1384. https://doi.org/10.1057/jors.2014.113

Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of web of science and scopus: A comparative analysis. Scientometrics, 106(1), 213-228. https://doi.org/10.1007/s11192-015-1765-5

Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the google scholar-based hg-index. Journal of Informetrics, 4(1), 107-117. https://doi.org/10.1016/j.joi.2009.10.001

O'Leary, D. (2008). On the relationship between citations and appearances on “top 25” download lists in the international journal of accounting information systems. International Journal of Accounting Information Systems, 9(1), 61-75. https://doi.org/10.1016/j.accinf.2008.02.001

Olson, J. E. (2005). Top-25-business-school professors rate journals in operations management and related fields. Interfaces, 35(4), 323-338. https://doi.org/10.1287/inte.1050.0149

Petersen, C. G., Aase, G. R., & Heiser, D. R. (2011). Journal ranking analyses of operations management research. International Journal of Operations & Production Management, 31(4), 405-422. https://doi.org/10.1108/01443571111119533

Pindek, S., Kessler, S. R., & Spector, P. E. (2017). A quantitative and qualitative review of what meta-analyses have contributed to our understanding of human resource management. Human Resource Management Review, 27(1), 26-38. https://doi.org/10.1016/j.hrmr.2016.09.003

Pislyakov, V. (2009). Comparing two “thermometers”: Impact factors of 20 leading economic journals according to journal citation reports and scopus. Scientometrics, 79(3), 541-550. https://doi.org/10.1007/s11192-007-2016-1

Quintas-Froufe, N. (2016). Quality indicators for scientific publications in the social sciences in spain: A comparative analysis between evaluation agencies. Revista de Investigación Educativa, 34(1), 259-272. http://dx.doi.org/10.6018/rie.34.1.210191

Ramin, S., & Sarraf Shirazi, A. (2012). Comparison between impact factor, scimago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear medicine review. Central& Eastern Europe, 15( ), 132-136.

Rao, S., Iyengar, D., & Goldsby, T. J. (2013). On the measurement and benchmarking of research impact among active logistics scholars. International Journal of Physical Distribution & Logistics Management, 43(10), 814-832. https://doi.org/10.1108/IJPDLM-07-2012-0207

Rosenstreich, D., & Wooliscroft, B. (2009). Measuring the impact of accounting journals using google scholar and the g-index. British Accounting Review, 41(4), 227-239. https://doi.org/10.1016/j.bar.2009.10.002

Rosenstreich, D., & Wooliscroft, B. (2012). Assessing international journal impact: The case of marketing. European Business Review, 24(1), 58- 7 https://doi.org/10.1108/09555341211191553

Rosenstreich, D., & Wooliscroft, B. (2013). An improved method for assessing the impact of management journals. Asian Academy of Management Journal, 18(2), 37-54.

Ruiz-Pérez, R., López-Cózar, E. D., & Jiménez-Contreras E. (2010). Principles and criteria used by the national evaluation committee of research activity (cneai-spain) for the assessment of scientific publications: 1989-2009. Psicothema, 22(4), 898-908.

Russ-Eft, D. (2008). Ssci, isi, jcr, jif, if, and journal quality. Human Resource Development Quarterly, 19(3), 185-189. https://doi.org/10.1002/hrdq.1235

Salgado, J. F., & Páez, D. (2007). Scientific productivity and hirsch's h index of Spanish social psychology: Convergence between productivity indexes and comparison with other areas. Psicothema, 19(2), 179-189.

Salvador-Oliván, J. A., & Agustín-Lacruz, C. (2015). Correlation between bibliometric indicators in web of science y scopus journals. Revista General de Informacion y Documentacion, 25(2), 341-359. https://doi.org/10.5209/rev_RGID.2015.v25.n2.51241

Schöpfel, J., & Prost, H. (2009). Comparison of scimago journal rank indicator (sjr) with jcr journal impact factor (if) for french journals. Psychologie Francaise, 54(4), 287-305. https://doi.org/10.1016/j.psfr.2009.07.002

Segalla, M. (2009). Will you cite me? The emerging strategy of academic publishing. European Journal of International Management, 3(4), 411-418. https://doi.org/10.1504/EJIM.2009.028847

Serenko, A., & Bontis, N. (2009). Global ranking of knowledge management and intellectual capital academic journals. Journal of Knowledge Management, 13(1), 4-15. https://doi.org/10.1108/13673270910931125

Serenko, A., & Bontis, N. (2013). Global ranking of knowledge management and intellectual capital academic journals: 2013 update. Journal of Knowledge Management, 17(2), 307-326. https://doi.org/10.1108/13673271311315231

Serenko, A., & Bontis, N. (2017). Global ranking of knowledge management and intellectual capital academic journals: 2017 update. Journal of Knowledge Management, 21(3), 675-692. https://doi.org/10.1108/JKM-11-2016-0490

Shugan, S. M. (2003). Editorial: Journal rankings: Save the outlets for your research. Marketing Science, 22(4), 437-441. https://doi.org/10.1287/mksc.22.4.437.24904

Sicilia, M. A., Sanchez-Alonso, S., & Garcia-Barriocanal, E. (2011). Comparing impact factors from two different citation databases: The case of computer science. Journal of Informetrics, 5, 698-704. https://doi.org/10.1016/j.joi.2011.01.007

Siebelt, M., Siebelt, T., Pilot, P., Bloem, R. M., Bhandari, M., & Poolman, R. W. (2010). Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for impact factor and scimago. BMC Muscul skeletal Disorders, 11. https://doi.org/10.1186/1471-2474-11-4

Stonebraker, J. S., Gil, E., Kirkwood, C. W., & Handfield, R. B. (2012). Impact factor as a metric to assess journals where om research is published. Journal of Operations Management, 30(1-2), 24-43. https://doi.org/10.1016/j.jom.2011.05.002

Svensson, G. (2006). The paradoxnoia of top journal(s) of top in marketing. European Journal of Marketing, 40(11-12), 1153-1168. https://doi.org/10.1108/03090560610702740

Tadajewski, M. (2016). Academic labour, journal ranking lists and the politics of knowledge production in marketing. Journal of Marketing Management, 32(1-2), 1-18. https://doi.org/10.1080/0267257X.2015.1120508

Theussl, S., Reutterer, T., & Hornik, K. (2014). How to derive consensus among various marketing journal rankings? Journal of Business Research, 67(5), 998-1006. https://doi.org/10.1016/j.jbusres.2013.08.006

Tsay, M. Y. (2012). An overview of the journal impact indicators. Journal of Educational Media and Library Science, 49(2).

Viladrich Segués, M. C., & Doval Dieguez, E. (2011). Medición: Fiabilidad y validez. Bellaterra: Laboratori d'Estadística Aplicada i de Modelització (UAB).

Vokurka, R. J. (1996). The relative importance of journals used in operations management research: A citation analysis. Journal of Operations Management, 14(4), 345-355. https://doi.org/10.1016/S0272-6963(96)00092-7

Walters, W. H. (2017). Citation-based journal rankings: Key questions, metrics, and data sources. IEEE Access, 5, 22036-22053. https://doi.org/10.1109/ACCESS.2017.2761400

Zarifmahmoudi, L., Jamali, J., & Sadeghi, R. (2015). Google scholar journal metrics: Comparison with impact factor and scimago journal rank indicator for nuclear medicine journals. Iranian Journal of Nuclear Medicine, 23(1), 8-14.

Zhou, Y. B., Lü, L., & Li, M. (2012). Quantifying the influence of scientists and their publications: Distinguishing between prestige and popularity. New Journal of Physics, 14. - https://doi.org/10.1088/1367-2630/14/3/033033

Zinkhan, G. M., & Leigh, T. W. (1999). Assessing the quality ranking of the journal of advertising, 1986- 1997. Journal of Advertising, 28(2), 51-70. https://doi.org/10.1080/00913367.1999.10673583

Zupic, I., & Cater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18(3), 429-472. https://doi.org/10.1177/1094428114562629

Abstract Views

3307
Metrics Loading ...

Metrics powered by PLOS ALM

Refbacks

  • There are currently no refbacks.


 

Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Avances de los investigadores españoles de Dirección de Operaciones en el año 2019: el caso de los miembros de ACEDEDOT
Lidia Sanchez-Ruiz, Elsa Diez-Busto
WPOM-Working Papers on Operations Management  vol: 11  issue: 1  first page: 54  year: 2020  
doi: 10.4995/wpom.v11i1.13637



This journal is licensed under a Creative Commons Attribution 4.0 International License.

Universitat Politècnica de València

e-ISSN: 1989-9068   https://doi.org/10.4995/wpom