Constructing an evidence-base for future CALL design with ‘engineering power’: The need for more basic research and instrumental replication

Zöe Handley


This paper argues that the goal of Computer-Assisted Language Learning (CALL) research should be to construct a reliable evidence-base with ‘engineering power’ and generality upon which the design of future CALL software and activities can be based. In order to establish such an evidence base for future CALL design, it suggests that CALL research needs to move away from CALL versus non-CALL comparisons, and focus on investigating the differential impact of individual attributes and affordances, that is, specific features of a technology which might have an impact on learning. Further, in order to help researchers find possible explanations for the success or failure of CALL interventions and make appropriate adjustments to their design, it argues that these studies should be conducted within the framework of Second Language Acquisition (SLA) theory and research. Despite this, a recent review of research examining the effectiveness of CALL in primary and secondary English as a Foreign Language (EFL) found that CALL vs. non-CALL comparisons are still common and studies focusing on individual coding elements are rare. Further, few studies make links with SLA and few measure linguistic outcomes using measures developed in the field of SLA.  One reason for this may be poor reporting of methods and difficulty in obtaining the instruments used in SLA research. Reporting guidelines and the use of the IRIS database ( are introduced as possible solutions to these problems.


Research methods; basic research; second language acquisition; replication; instruments

Full Text:



Bannan-Ritland, B. (2003). The role of design in research: The integrative learning design framework. Educational Researcher, 32(1), 21-24.

Barab, S. & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14.

Borg, S. (2003). Teacher cognition in language teaching: A review of research on what language teachers think, know, believe and do. Language Teaching, 36(2), 81-109.

Burkhardt, H. & Schoenfeld, A. H. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(3), 3-14.

Burston, J (2006). Working towards effective assessment of CALL. In Donaldson, P. R. & Haggstrom, M. A. (eds.). Changing Language Education Through CALL. London: Routledge, 249-270.

Campbell, M., Fitzpatrick, R., Haines, A., Kinmouth, A. L., Sandercock, P., Spiegelhalter, D. & Tyrer, P. (2000). Framework for design and evaluation of complex interventions to improve health. British Medical Journal, 321, 694-696.

Chapelle, C. (2010). The spread of computer-assisted language learning. Language Teaching, 43(1), 66-74.

Chen, C.-M. & Li, Y. L. (2010). Personalised context-aware ubiquitous learning system for supporting effective English as a second language. TESOL Quarterly, 20(1), 27-46.

Chun, D. (2012). Review article: Replication studies in CALL research. CALICO Journal, 29(4), 591-600.

Colpaert, J. (2006). Pedagogy-driven design for online language teaching and learning. CALICO Journal, 23(3), 477-497.

Colpaert, J. (2010). Elicitation of language learners' personal goals as design concepts. Innovation in language learning and teaching, 4(3) 259-274.

Dalton, B., Proctor, C. P., Uccelli, P., Mo, E. & Snow, C. E. (2011). Designing for diversity: The role of reading strategies and interactive vocabulary in a digital reading environment for fifth-grade monolingual English and bilingual students. Journal of Literacy Research, 43, 68-100

Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8

García Mayo, M. (2005). Interactional strategies for interlanguage communication: Do they provide evidence for attention to form? In A. Housen & M. Pierrard (Eds.), Investigations in instructed second language acquisition (Studies on Language Acquisition Series). Mouton de Gruyter.

Garrett, N. (1991). Technology in the service of language learning: Trends and issues. Modern Language Journal, 75, 74-101

Gorard, S., Roberts, K. & Taylor, C. (2004). What kind of creature is a design experiment? British Educational Research Journal, 30(4), 577-590.

Gruber-Miller, J. & Benton, C. (2001). How do you say 'MOO' in Latin? Assessing student learning and motivation in beginning Latin. CALICO Journal, 18(2), 305-38.

Gutiérrez-Colon, M.; Gibert, M.I.; Triana, I.; Gimeno, A.; Appel, C. & Hopkins, J. (2003). Improving learners' reading skills through instant short messages: a sample study using WhatsApp. WorldCALL 2013 – CALL: Sustainability and Computer-Assisted Language Learning Conference Proceedings. University of Ulster, pp.80-84.

Hiebert, J., Gallimore, R. & Stigler, J. (2002). A knowledge base for the teaching profession: What would it look like and how can we get one? Educational Researcher, 31(3), 3-15.

Higher Education Research Funding Council (1999). Guidance on submissions research assessment exercise, Paragraph 1.12. London: Higher Education Funding Council for England and Wales 1999.

Hwang, W. Y. & Chen, H. S. L. (2013). Users' familiar situational context facilitate the practice of EFL in elementary schools with mobile devices. Computer-Assisted Language Learning, 26(2), 101-125

Kim, D.; Gilman, D. A. (2008). Effects of Text, Audio, and Graphic Aids in Multimedia Instruction for Vocabulary Learning. Educational Technology & Society. 11 (3): 114-126.

Kumaravadivelu, B. (1994). The post-method condition: (E)merging strategies for Second/Foreign language teaching. TESOL Quarterly, 28(1), 27-48.

Levy, M. & Stockwell, G. (2006). CALL Dimensions: Options and Issues in Computer-Assisted Language Learning. London: Lawrence Erlbaum.

Lu, M. (2008). Effectiveness of vocabulary learning via mobile phones. Journal of Computer Assisted Learning, 24(6), 515-525.

Luckin, R. & Clark, W. (2011). More than a game: The participatory Design of contextualised technology-rich learning experiences with the ecology of resources. Journal of e-Learning Knowledge Society, 7(3), 33-50.

Lund, A. (2008). Wikis: A collective approach to language production. ReCALL, 20(1), 35-54.

Macaro, E., Handley, Z. L., & Walter, C. (2012). A systematic review of CALL in English as a second language: Focus on primary and secondary education. Language Teaching, 45(1), 1-43

Marsden, E. & King, J. (2013). The Instruments for Research into Second Languages (IRIS) digital repository. The Language Teacher, 37(2), 35-38.

Marsden, E. J., & Mackey, A. (2014). IRIS: a new resource for second language research. Linguistic Approaches to Bilingualism, 4(1), 125-130.

Mifka Profozic, N. (2012). Oral corrective feedback, individual differences and L2 acquisition of French past tenses. (Unpublished doctoral dissertation). University of Auckland.

Moher, D., Jones, A. & Lepage, L. (2001). Use of the CONSORT Statement and quality of reports of randomized trials. A comparative before-and-after evaluation. The Journal of the American Medical Association, 285, 1992-5.

Moher, D., Schulz, K. ., Altman, D. (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. The Journal of the American Medical Association, 285, 1987-91.

O'Hara, & Pritchard, (2008). Hypermedia authoring as a vehicle for vocabulary development in middle school English as a second language classrooms. Clearing House: A Journal of Educational Strategies, Issues, and Ideas, 82(2), 60-65.

Oswald, F. L. & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85-110.

Pardo-Ballester, C. & Rodriguez, J. C. (2009). Using design-based research to guide the development of online instructional materials. In Chapelle, C. A., Jun, H. G., & Katz, I. (Eds.). Developing and evaluating language learning materials (pp. 86-102). Ames, IA: Iowa State University.

Pardo-Ballester, C. & Rodriguez, J. C. (2010). Developing Spanish online readings using design-based research. CALICO Journal, 27(3), 540-553.

Pederson, K (1987). Research on CALL. In Smith, W. F. (ed.). Modern media in foreign language education: Theory and implementation. Lincolnwood, Illinois: National Textbook Company, pp. 99-131.

Polio, C. (2012). Replication in published applied linguistics research: A historical perspective. In Porte, G. (ed.). Replication research in applied linguistics. Cambridge: Cambridge University Press, pp. 47-91.

Polio, C. & Gass, S. (1997). Replication and reporting. Studies in Second Language Acquisition, 19, 499-508.

Porte, G. (2013). Who needs replication? CALICO Journal, 30(1), 10-15

Porte, G. & Richards, K. (2012). Focus article: Replication in second language writing research. Journal of Second Language Writing, 21(2012), 284-193.

Proctor, C. P., Dalton, B. & Grisham, D. L. (2007). Scaffolding English language learners and struggling readers in a universal literacy environment with embedded strategy instruction and vocabulary support. Journal of Literacy Research, 39(1), 71-93.

Révész, A. (2011). Task complexity, focus on L2 constructions, and individual differences: A classroom-based study. The Modern Language Journal, 95(4).

Sasaki, A. & Takeuchi, O. (2010). EFL students' vocabulary learning in NS-NNS e-mail interactions: Do they learn new words by imitation? ReCALL, 22(1): 70-82.

Slavin, R. E. (1995). Best evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology, 48(1), 9-18.

Stockwell, G. (2012). Diversity in research and practice. In Stockwell, G. (ed.). Computer-Assisted Language Learning: Diversity in Research and Practice. Cambridge: Cambridge University Press, pp. 147-163.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus-groups. International Journal for Quality in Health Care, 19(6), 349-357.

van Lier, L. (2004). The Ecology and Semiotics of Language Learning. A Sociocultural Perspective. Boston: Kluwer Academic

Yildiz, R. & Atkins, M. (1993). Evaluating multimedia applications. Computers in Education, 21(1/2), 133-139.

Yutdhana, S. (2008). Design-based research in CALL. In Egbert, J. L. & Petrie, G. M. (eds.). CALL research perspectives. Mahwah, NJ: Lawrence Erlbaum, pp. 169-178.

Abstract Views

Metrics Loading ...

Metrics powered by PLOS ALM


  • There are currently no refbacks.


Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Replication research in computer-assisted language learning: Replication of Neri et al. (2008) and Satar & Ӧzdener (2008)
Zöe Handley
Language Teaching  vol: 51  issue: 3  first page: 417  year: 2018  
doi: 10.1017/S0261444817000040

Licencia Creative Commons

This journal is licensed under a  Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Universitat Politècnica de València

e-ISSN: 1695-2618