GRAPE-MARS: A digital tool for enhanced multimodal discourse analysis in video-based research
Submitted: 2025-09-14
|Accepted: 2026-06-02
|Published: 2026-07-24
Copyright (c) 2026 Revista de Lingüística y Lenguas Aplicadas
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Downloads
Keywords:
multimodal discourse analysis, GRAPE-MARS, multimodal annotation, multimodal data coding, research methods, video analysis
Supporting agencies:
This study has been funded by the Spanish Ministry of Science and Innovation and by Universitat Jaume I
Abstract:
Multimodal analyses play a crucial role in understanding meaning-making processes. However, their complexity and labor-intensive nature pose significant challenges. This paper introduces GRAPE-MARS (Multimodal Analysis Research Software), a web-based, open-source tool developed to enhance multimodal research by streamlining annotation, data retrieval, and visualization. GRAPE-MARS integrates AI-driven automatic transcription, user-friendly annotation, and advanced data analysis. In this paper, we present the design and functionalities of the software. Furthermore, we apply it in a case study to analyse engagement strategies in a YouTube science dissemination video. Using GRAPE-MARS, we quantify the distribution of semiotic modes and examine how engagement strategies are realized through distinct modal configurations. These results highlight the potential of the software to facilitate empirical multimodal analyses by providing structured, quantitative insights into semiotic interactions. This research contributes to the advancement of multimodal discourse analysis methodologies and underscores the value of digital tools in optimizing multimodal data coding and analysis.
References:
ATLAS.ti Scientific Software Development GmbH. (2023). ATLAS.ti Mac (version 23.2.1) [Qualitative data analysis software]. https://atlasti.com
Bateman, J. A., Thiele, L., & Akin H. (2021) Explanation videos unravelled: Breaking the waves. Journal of Pragmatics,175, 112-128.https://doi.org/10.1016/j.pragma.2020.12.009
Bernad-Mechó, E. & Girón-García, C. (2023). A Multimodal Analysis of Humour as an Engagement Strategy in YouTube Research Dissemination Videos. The European Journal of Humour Research, 11(1), 46-66. https://doi.org/10.7592/EJHR2023.11.1.760
Bernad-Mechó, E. & Valeiras-Jurado, J. (2023b) Engaging the audience in online YouTube science dissemination videos: A look at the uptake of multimodal engagement strategies. ESP Today, 11(2), 302-327. https://doi.org/10.18485/esptoday.2023.11.2.6
Bernad-Mechó, E. (2018). A multimodal approach to metadiscourse as an organizational tool in lectures (Unpublished doctoral dissertation). Universitat Jaume I, Castelló de la Plana, Spain. http://doi.org/10.6035/14110.2018.189457
Bernad-Mechó, E., & Valeiras-Jurado, J. (2023a). Multimodal engagement strategies in science dissemination: A case study of TED talks and YouTube science videos. Discourse Studies, 25(6), 733-754. https://doi.org/10.1177/14614456231161755
Bezemer, J., & Mavers, D. (2011) Multimodal transcription as academic practice: a social semiotic perspective. International Journal of Social Research Methodology, 14(3), 191–206. https://doi.org/10.1080/13645579.2011.563616
Boersma P (2001) Praat, a system for doing phonetics by computer. Glot International, 5(9): 341–345.
Boggio-López E and Author 3 (2024) [Invited speakers] ICLHE webinar series. January 23, 2024.
Boggio-López, E., & Costa, F. (2025). Exploring language-related episodes (LREs) in English-medium instruction (EMI) from a translanguaging and multimodal perspective. Journal of Multilingual and Multicultural Development, 1–21. https://doi.org/10.1080/01434632.2024.2449063
Bonsignori V, Crarwford-Camiciottoli B and Filmer D (2022) Analyzing Multimodality in Specialized Discourse Settings. Innovative Research Methods and Applications. Vernon Press.
Cvetkovic A and Author 4 (2024) [Presentation] 41st AESLA Conference. Valencia, April 17-19, 2024.
Duncan S, Rohlfing K and Loehr D (2013) Multimodal annotation tools. In Müller C, Cienki A, Fricke E, Ladewig S, McNeill D and Tessendorf S (eds) Body - Language - Communication. Volume 1 De Gruyter Mouton, pp. 1015-1022 https://doi.org/10.1515/9783110261318.1015
Escobar-Lluch N and Author 3 (2024) Multimodal approach to translanguaging strategies: from translanguaging to trans-semiotising in an EMI Business course. [Presentation] 41st AESLA Conference. Valencia, April 17-19, 2024.
Fortanet-Gómez, I., Ruiz-Madrid,N., Bernad-Mechó, E., Valeiras-Jurado, J. (2025). GRAPE -MARS: una nueva herramienta para el análisis multimodal en la investigación sobre segundas lenguas. TEISEL. Tecnologías para la investigación en segundas lenguas, 4, 1-26. https://doi.org/10.1344/teisel.v4.46861
Girón-García C and Author 4 (2023). English for Specific Purposes 70: 164-176 https://doi.org/10.1016/j.esp.2022.12.005
Guillén Galve I and Vela Tafalla M A (2023) (eds.). Digital Genres and Multimodality. ESP Today 11(2). https://doi.org/10.18485/esptoday.2023.11.2.1
Jewitt C (2013) Multimodal methods for researching digital technologies. In Price S, Jewitt C and Brown B (eds) The Sage handbook of digital technology research . London: Sage, pp.250-265
Kipp M (2001) ANVIL A generic annotation tool for multimodal dialogue. Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367-1370.
Kress G (2006) Review of literacy in the new media age. Routledge.
Lim F V and Tan-Chia L (2022) Designing Learning for Multimodal Literacy: Teaching Viewing and Representing (1st ed.). Routledge. https://doi.org/10.4324/9781003258513
Liu Y and Xiang Y (2024) Computer-assisted multimodal analysis research: A technical review of GRAPE-MARS. Multimodality & Society 4(2): 239-246. https://doi.org/10.1177/26349795241251623
Mena-Alba A (2024) New pedagogical artefacts for new learning needs: Instagram reels and multimodal literacy in pre-tertiary education. Unpublished Master Thesis. Universitat Jaume I: Castelló de la Plana, Spain.
Mestre-Segarra, M. Á. (2023). A multimodal rhetorical analysis of video resumes. ESP Today, 11(2), 349-370.
https://doi.org/10.18485/esptoday.2023.11.2.8
Muñoz Morcillo J, Czurda K and Robertson-von Trotha C Y (2016) Typologies of the popular science web video. Journal of Science Communication 15(4): 1–32. https://doi.org/10.22323/2.15040202
Norris, S. (2004) Analyzing Multimodal Interaction. A methodological framework. Routledge.https://doi.org/10.4324/9780203379493
O’Halloran, K. L., Pal, G., & Jin, M. (2021) Multimodal approach to analysing big social and news media data.Discourse, Context & Media, 40, 100467. https://doi.org/10.1016/j.dcm.2021.100467
O’Halloran, K. L., Podlasov, A., Chua, A., & Marissa, K. L. E. (2012) Interactive software for multimodal analysis. Visual Communication, 11(3), 363–381. https://doi.org/10.1177/1470357212446414
O’Halloran, K. L., Tan, S., & Marissa, K. L. E. (2015) Multimodal analysis for critical thinking. Learning, Media and Technology,42(2), 147–170. https://doi.org/10.1080/17439884.2016.1101003
Querol-Julián M (2011) Evaluation in discussion sessions of conference paper presentations: a multimodal approach. LAP Lambert Academic Publishing GmbH & Co. KG.
Ruiz-Madrid, M. N. (2021). A multimodal discourse approach to Research Pitches. Journal of English for Academic Purposes,15.https://doi.org/10.1016/j.jeap.2021.101003
Ruiz-Madrid, N., Fortanet-Gómez, I., Bernad-Mechó, E., & Valeiras-Jurado, J. (2023). GRAPE-MARS (Multimodal Analysis Research Software). Castelló de la Plana: Universitat Jaume I. https://mars.grape.uji.es
Ruiz-Madrid, N., Valeiras-Jurado, J. (2023). Reconceptualisation of Genre(s) in Scholarly and Scientific Digital Practices: A Look at Multimodal Online Genres for the Dissemination of Science. In Plo-Alastrué, R., Corona, I. (eds) Digital Scientific Communication. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-031-38207-9_9
Serrano-Hernández, E. (2025). A pedagogical proposal to foster multimodal literacy in Spanish Official School of Languages. In Bernad-Mechó, E. (Ed.) Transformacions socials i culturals: Nous horitzons en la recerca de ciències humanes i socials. Publicacions de la Universitat Jaume I.
Serrano-Hernández, E., & Bernad-Mechó (2026). Science Dissemination in Instagram reels: A multimodal overview Círculo de Lingüística Aplicada a la Comunicación, 105, 293.
Valeiras Jurado, J., Ruiz Madrid, M. N., & Jacobs, G. (2018). Revisiting persuasion in oral academic and professional genres: Towards a methodological framework for Multimodal Discourse Analysis of research dissemination talks . Ibérica, (35), 93–118.
Valeiras-Jurado, J. & Bernad-Mechó, E. (2022). Modal Density and Coherence in Science Dissemination: Orchestrating Multimodal Ensembles in Online TED Talks and YouTube Science Videos. Journal of English for Academic Purposes, 58, 101118. https://doi.org/10.1016/j.jeap.2022.101118
Valeiras-Jurado, J., & Ruiz-Madrid, N. (2023). The influence of discipline, medium and target audience in multimodal recontextualization practices: The case of popular science online videos. Ibérica, (46), 237–269 https://doi.org/10.17398/2340-2784.46.237
Vivas-Peraza, A. C. (2026). Multimodality as a framework for the study of verbal and non-verbal resources in webbased science communication. The case of science crowdfunding videos (SCVs). Círculo de Lingüística Aplicada a la Comunicación. https://doi.org/10.5209/clac.94692
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006) ELAN: a professional framework for multimodality research. In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation, pp. 1556–1559. https://doi.org/10.63317/5pwa5zpssv4z
Xia, S. A., & Hafner, C. A. (2021) Engaging the online audience in the digital era: A multimodal analysis of engagement strategies in TED talk videos. Ibérica, 42, 33-58. https://doi.org/10.17398/2340-2784.42.33
Yuan, C., He, Y., & Liu, Y. (2021) Rule by law versus rule of law: A multimodal analysis of persuasion and legal ideologies in anti-corruption discourse in China. Multimodality & Society, 1(4), 429-454.https://doi.org/10.1177/26349795211060751
