An Evaluation Of A Linguistically Motivated Conversational Software Agent Framework

Kulvinder Panesar


This paper presents a critical evaluation framework for a linguistically motivated conversational software agent (CSA). The CSA prototype investigates the integration, intersection and interface of the language, knowledge, and speech act constructions (SAC) based on a grammatical object, and the sub-model of belief, desires and intention (BDI) and dialogue management (DM) for natural language processing (NLP). A long-standing issue within NLP CSA systems is refining the accuracy of interpretation to provide realistic dialogue to support human-to-computer communication. This prototype constitutes three phase models: (1) a linguistic model based on a functional linguistic theory – Role and Reference Grammar (RRG), (2) an Agent Cognitive Model with two inner models: (a) a knowledge representation model, (b) a planning model underpinned by BDI concepts, intentionality and rational interaction, and (3) a dialogue model. The evaluation strategy for this Java-based prototype is multi-approach driven by grammatical testing (English language utterances), software engineering and agent practice. A set of evaluation criteria are grouped per phase model, and the testing framework aims to test the interface, intersection and integration of all phase models. The empirical evaluations demonstrate that the CSA is a proof-of-concept, demonstrating RRG’s fitness for purpose for describing, and explaining phenomena, language processing and knowledge, and computational adequacy. Contrastingly, evaluations identify the complexity of lower level computational mappings of NL – agent to ontology with semantic gaps, and further addressed by a lexical bridging solution.


conversational software agents; natural language processing; speech act construction; knowledge representation; belief-desire and intentions; functional linguistics

Full Text:



Abran, A., A. Khelifi, W. Suryn, and A. Seffah. 2003. "Consolidating the ISO usability models. ." In proceedings of 11th International Software Quality Management Conference 2003:pp. 23-25.

Bimson, Kent D., and Richard D. Hull. 2016. "Unnatural Language Processing: Characterizing the Challenges in Translating Natural Language Semantics into Ontology Semantics." In Semantic Web: Implications for Technologies and Business Practices, edited by Michael Workman, 119-135. Cham: Springer International Publishing.

Bimson, Kent D., Richard D. Hull, and Daniel Nieten. 2016. "The Lexical Bridge: A Methodology for Bridging the Semantic Gaps between a Natural Language and an Ontology." In Semantic Web: Implications for Technologies and Business Practices, edited by Michael Workman, 137-151. Cham: Springer International Publishing.

Bratman, Michael. 1987. Intention, plans, and practical reason. Cambridge: HUP.

Cohen, Philip R, and Hector J Levesque. 1988. Rational interaction as the basis for communication. DTIC Document.

Cohen, Philip R, and Hector J Levesque. 1990. "Intention is choice with commitment." Artificial Intelligence 42 (2):213-261.

Dijkstra, Edsger W. 1959. "A note on two problems in connexion with graphs." Numerische mathematik 1 (1):269-271.

Francois, Jacques. 2014. "Functionalist views on the genesis and development of grammar " 25th European Systemic Functional Linguistics Conference Paris, July 11th 2014.

GraphIK. 2012. "CoGui (Conceptual Graphs Graphical User Interface) User Guide." accessed 12 March 2014.

Horrocks, Ian, Peter F Patel-Schneider, Sean Bechhofer, and Dmitry Tsarkov. 2005. "OWL rules: A proposal and prototype implementation." Web Semantics: Science, Services and Agents on the World Wide Web 3 (1):23-40.

Kamp, Hans, Josef Van Genabith, and Uwe Reyle. 2011. "Discourse representation theory." In Handbook of philosophical logic, 125-394. Springer.

Lester, James, Karl Branting, and Bradford Mott. 2004. "Conversational agents." The Practical Handbook of Internet Computing, CRC Press, UK.

Mao, Xuetao, Jean-Paul Sansonnet, and Lin Li. 2012. "Textual Conversation Agent for Enhancing Attraction in E-Learning." Proceedings of International Conference on Computer Science and Information Technology.

Nolan, Brian. 2014. "Constructions as grammatical objects : A case study of prepositional ditransitive construction in Modern Irish." In Linking Constructions into Functional Linguistics: The role of constructions in grammar, edited by Brian Nolan and Elke Diedrichsen, 143-178. Amsterdam/Philadelphia: John Benjamins Publishing Company.

Panesar, Kulvinder. 2017. "A linguistically centred text-based conversational software agent." Unpublished PhD thesis, School of Computing, Creative Technologies and Engineering, Leeds Beckett University.

Perez-Marin, Diana, Ismael Pascual-Nieto, and IGI Global. 2011. Conversational Agents and Natural Language Interaction: Techniques and Effective Practices: Information Science Reference.

Radziwill, Nicole M, and Morgan C Benton. 2017. "Evaluating Quality of Chatbots and Intelligent Conversational Agents." arXiv preprint arXiv:1704.04579.

Russell, S., and P. Norvig. 2016. Artifical Intelligence: A Modern Approach - Global Edition. Sixth ed: Pearson.

Searle, John R. 1969. Speech acts: An essay in the philosophy of language. Vol. 626. Cambridge: Cambridge University Press.

Searle, John R. 1980. "Minds, brains, and programs." Behavioral and Brain Sciences 3 (3): 417-424.

Searle, John R. 1983. Intentionality: An essay in the philosophy of mind: Cambridge University Press.

Stalnaker, Robert. 2002. "Common ground." Linguistics and philosophy 25 (5):701-721.

Turing, Alan M. 1950. "Computing machinery and intelligence." Mind, Volume LIX, Issue 236: 433-460.

Van Valin Jr, Robert D. 2005. Exploring the syntax-semantics interface: Cambridge University Press.

Weizenbaum, Joseph. 1966. "ELIZA-a computer program for the study of natural language communication between man and machine." Communications of the ACM 9 (1):36-45.

Wooldridge, Michael. 2013. "Intelligent Agents." In Multiagent Systems, edited by Gerhard Weiss, 3-50. USA: Massachusetts Institute of Technology.

Worswick, Steve. 2018. "Mitsuku - Loebner Prize 2018.", accessed 12th September 2018.

Yao, Haining, and Letha Etzkorn. 2006. "Automated conversion between different knowledge representation formats." Knowledge-Based Systems 19 (6):404-412.

Abstract Views

Metrics Loading ...

Metrics powered by PLOS ALM


Cited-By (articles included in Crossref)

This journal is a Crossref Cited-by Linking member. This list shows the references that citing the article automatically, if there are. For more information about the system please visit Crossref site

1. Conversational artificial intelligence - demystifying statistical vs linguistic NLP solutions
Kulvinder Panesar
Journal of Computer-Assisted Linguistic Research  vol: 4  issue: 1  first page: 47  year: 2020  
doi: 10.4995/jclr.2020.12932

Creative Commons License

This journal is licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Universitat Politècnica de València

e-ISSN: 2530-9455