Evaluating the Impact of AI Tools on Grammar Mastery: A Comparative Study of Learning Outcomes
DOI:
https://doi.org/10.29408/veles.v8i3.27856Keywords:
Artificial intelligence, education, grammar proficiency, learning media, learning outcomeAbstract
Artificial Intelligence (AI) tools are increasingly integrated into education, offering innovative solutions to enhance learning, particularly in language acquisition. In grammar instruction, tools like Grammarly and ChatGPT provide real-time feedback, error correction, and personalized learning experiences. This study investigates the impact of these AI-assisted learning tools on grammar proficiency among students from diverse academic programs in two higher education institutions in Indonesia. A quasi-experimental design was employed, comparing pre-test and post-test grammar scores to evaluate the effectiveness of these tools. A total of 150 students participated, representing programs such as Nursing, Pharmacy, Health Administration, Economics Education, and English Education. The findings revealed significant improvements in grammar scores, particularly among students in non-language-focused programs, with an average increase of 15%. Students in the English Education program showed smaller gains (5%), attributed to their higher baseline proficiency. While the results demonstrate the potential of AI tools in enhancing grammar accuracy, concerns were raised about over-reliance on technology and the need for transparent integration in educational contexts. The absence of qualitative insights and long-term retention data are acknowledged as limitations. This study highlights the importance of using AI as a supplementary resource to support meaningful grammar learning, especially for students with limited exposure to formal language instruction.
References
Abulibdeh, A., Zaidan, E., & Abulibdeh, R. (2024). Navigating the confluence of artificial intelligence and education for sustainable development in the era of industry 4.0: Challenges, opportunities, and ethical dimensions. Journal of Cleaner Production, 437, 140527. https://doi.org/10.1016/j.jclepro.2023.140527
Agarwal, P., Swami, S., & Malhotra, S. K. (2022). Artificial intelligence adoption in the post covid-19 new-normal and role of smart technologies in transforming business: A review. Journal of Science and Technology Policy Management, 15(3), 506–529. https://doi.org/10.1108/jstpm-08-2021-0122
Ahmed, I., Jeon, G., & Piccialli, F. (2022). From artificial intelligence to explainable artificial intelligence in industry 4.0: A survey on what, how, and where. IEEE Transactions on Industrial Informatics, 18(8), 5031–5042. https://doi.org/10.1109/tii.2022.3146552
Almusharraf, N., & Alotaibi, H. (2022). An error-analysis study from an EFL writing context: Human and automated essay scoring approaches. Technology Knowledge and Learning, 28(3), 1015–1031. https://doi.org/10.1007/s10758-022-09592-z
Alshumaimeri, Y. A., & Alshememry, A. K. (2023). The extent of AI applications in EFL learning and teaching. IEEE Transactions on Learning Technologies, 17, 653–663. https://doi.org/10.1109/tlt.2023.3322128
Amankwah-Amoah, J., Khan, Z., Wood, G., & Knight, G. (2021). COVID-19 and digitalization: The great acceleration. Journal of Business Research, 136, 602–611. https://doi.org/10.1016/j.jbusres.2021.08.011
Bai̇doo-Anu, D., & Ansah, L. O. (2023). Education in the era of generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–62. https://doi.org/10.61969/jai.1337500
Barrot, J. S. (2021). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(4), 584–607. https://doi.org/10.1080/09588221.2021.1936071
Calma, A., Cotronei-Baird, V., & Chia, A. (2022). Grammarly: An instructional intervention for writing enhancement in management education. The International Journal of Management Education, 20(3), 100704. https://doi.org/10.1016/j.ijme.2022.100704
Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., Bywaters, D., & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652–661. https://doi.org/10.1177/1744987120927206
Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00408-3
Chang, T., Li, Y., Huang, H., & Whitfield, B. (2021). Exploring EFL students’ writing performance and their acceptance of AI-based automated writing feedback. ICEDS ’21: Proceedings of the 2021 2nd International Conference on Education Development and Studies. https://doi.org/10.1145/3459043.3459065
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264–75278. https://doi.org/10.1109/access.2020.2988510
Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge. https://doi.org/10.4324/9781315456539
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications.
Dazeley, R., Vamplew, P., Foale, C., Young, C., Aryal, S., & Cruz, F. (2021). Levels of explainable artificial intelligence for human-aligned conversational explanations. Artificial Intelligence, 299, 103525. https://doi.org/10.1016/j.artint.2021.103525
De Andrade Pereira, F., Paul, L., Pritoni, M., Casillas, A., Prakash, A., Huang, W., Shaw, C., Martin-Toral, S., Finn, D., & Donnell, J. O. (2024). Enabling portable demand flexibility control applications in virtual and real buildings. Journal of Building Engineering, 86, 108645. https://doi.org/10.1016/j.jobe.2024.108645
Dutta, S., Ranjan, S., Mishra, S., Sharma, V., Hewage, P., & Iwendi, C. (2024). Enhancing educational adaptability: A review and analysis of AI-Driven adaptive learning platforms. 2024 4th International Conference on Innovative Practices in Technology and Management (ICIPTM) (pp. 1–5). https://doi.org/10.1109/iciptm59628.2024.10563448
Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2023). ChatGPT effects on cognitive skills of undergraduate students: Receiving instant responses from AI-based conversational large language models (LLMs). Computers and Education Artificial Intelligence, 6, 100198. https://doi.org/10.1016/j.caeai.2023.100198
Fitria, T. N. (2022). Identifying grammatical and mechanical errors of students’ writing: Using “Grammarly” as an online assessment. Lingua Didaktika Jurnal Bahasa Dan Pembelajaran Bahasa, 16(2), 169. https://doi.org/10.24036/ld.v16i2.116824
Getchell, K. M., Carradini, S., Cardon, P. W., Fleischmann, C., Ma, H., Aritz, J., & Stapp, J. (2022). Artificial intelligence in business communication: The changing landscape of research and teaching. Business and Professional Communication Quarterly, 85(1), 7–33. https://doi.org/10.1177/23294906221074311
Gopalan, M., Rosinger, K., & Ahn, J. B. (2020). Use of quasi-experimental research designs in education research: Growth, promise, and challenges. Review of Research in Education, 44(1), 218–243. https://doi.org/10.3102/0091732x2090330
Habib, S., Vogel, T., Anli, X., & Thorne, E. (2023). How does generative artificial intelligence impact student creativity? Journal of Creativity, 34(1), 100072. https://doi.org/10.1016/j.yjoc.2023.100072
Hwang, G., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of Artificial Intelligence in Education. Computers and Education Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001
Jeon, J. (2022). Exploring AI chatbot affordances in the EFL classroom: young learners’ experiences and perspectives. Computer Assisted Language Learning, 37(1–2), 1–26. https://doi.org/10.1080/09588221.2021.2021241
Jiang, R. (2022). How does artificial intelligence empower EFL teaching and learning nowadays? A review on artificial intelligence in the EFL context. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.1049401
Kabudi, T., Pappas, I., & Olsen, D. H. (2021). AI-enabled adaptive learning systems: A systematic mapping of the literature. Computers and Education Artificial Intelligence, 2, 100017. https://doi.org/10.1016/j.caeai.2021.100017
Kamalov, F., Calonge, D. S., & Gurrib, I. (2023). New era of artificial intelligence in education: Towards a sustainable multifaceted revolution. Sustainability, 15(16), 12451. https://doi.org/10.3390/su151612451
Kessler, M., Loewen, S., & Gönülal, T. (2023). Mobile-assisted language learning with Babbel and Duolingo: Comparing L2 learning gains and user experience. Computer Assisted Language Learning, 1–25. https://doi.org/10.1080/09588221.2023.2215294
Kholis, A. (2021). Elsa speak app: Automatic speech recognition (ASR) for supplementing English pronunciation skills. Pedagogy Journal of English Language Teaching, 9(1), 01. https://doi.org/10.32332/joelt.v9i1.2723
Kim, N. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37–46. https://doi.org/10.14400/jdc.2019.17.8.037
Koltovskaia, S. (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44, 100450. https://doi.org/10.1016/j.asw.2020.100450
Lameras, P., & Arnab, S. (2021). Power to the teachers: An exploratory review on artificial intelligence in education. Information, 13(1), 14. https://doi.org/10.3390/info13010014
Licardo, J. T., Domjan, M., & Orehovački, T. (2024). Intelligent robotics—A systematic review of emerging technologies and trends. Electronics, 13(3), 542. https://doi.org/10.3390/electronics13030542
Lin, C., Huang, A. Y. Q., & Lu, O. H. T. (2023). Artificial intelligence in intelligent tutoring systems toward sustainable education: A systematic review. Smart Learning Environments, 10(1). https://doi.org/10.1186/s40561-023-00260-y
Liu, Y., Saleh, S., & Huang, J. (2021). Artificial intelligence in promoting teaching and learning transformation in schools. International Journal of Innovation, Creativity and Change, 891–902. https://doi.org/10.53333/ijicc2013/15369
Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students’ writing: EFL teachers’ perspective. Cogent Education, 10(2). https://doi.org/10.1080/2331186x.2023.2236469
Ni, A., & Cheung, A. (2022). Understanding secondary students’ continuance intention to adopt AI-powered intelligent tutoring system for English learning. Education and Information Technologies, 28(3), 3191–3216. https://doi.org/10.1007/s10639-022-11305-z
Pinheiro, C. M. S. (2022). The use of alternative methods of assessment in higher education: a study of university teachers and students.
Popenici, S. a. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1). https://doi.org/10.1186/s41039-017-0062-8
Qu, K., & Wu, X. (2024). ChatGPT as a CALL tool in language education: A study of hedonic motivation adoption models in English learning environments. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12598-y
Rashid, A. B., & Kausik, M. a. K. (2024). AI revolutionizing industries worldwide: A Comprehensive overview of its diverse applications. Hybrid Advances, 7, 100277. https://doi.org/10.1016/j.hybadv.2024.100277
Riedl, M. O. (2019). Human‐centered artificial intelligence and machine learning. Human Behavior and Emerging Technologies, 1(1), 33–36. https://doi.org/10.1002/hbe2.117
Sanders, S. (2019). A brief guide to selecting and using Pre-Post assessments. The National Technical Assistance Center for the Education of Neglected or Delinquent Children and Youth.
Sarker, I. H. (2022). AI-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems. SN Computer Science, 3(2). https://doi.org/10.1007/s42979-022-01043-x
Schmidt-Fajlik, R. (2023). ChatGPT as a grammar checker for Japanese English language learners: A comparison with Grammarly and ProWritingAid. AsiaCALL Online Journal, 14(1), 105–119. https://doi.org/10.54855/acoj.231417
Selwyn, N. (2022). The future of AI and education: Some cautionary notes. European Journal of Education, 57(4), 620-631. https://doi.org/10.1111/ejed.12532
Shloul, T. A., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024). Role of activity-based learning and ChatGPT on students’ performance in education. Computers and Education Artificial Intelligence, 6, 100219. https://doi.org/10.1016/j.caeai.2024.100219
Shrestha, Y. R., Ben-Menahem, S. M., & Von Krogh, G. (2019). Organizational decision-making structures in the age of artificial intelligence. California Management Review, 61(4), 66–83. https://doi.org/10.1177/0008125619862257
Sun, W. (2023). The impact of automatic speech recognition technology on second language pronunciation and speaking skills of EFL learners: a mixed methods investigation. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1210187
Tien, J. M. (2017). Internet of things, real-time decision making, and artificial intelligence. Annals of Data Science, 4(2), 149–178. https://doi.org/10.1007/s40745-017-0112-5
Urlaub, P., & Dessein, E. (2022). Machine translation and foreign language education. Frontiers in Artificial Intelligence, 5. https://doi.org/10.3389/frai.2022.936111
Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems With Applications, 252, 124167. https://doi.org/10.1016/j.eswa.2024.124167
Wei, L. (2023). Artificial intelligence in language instruction: Impact on English learning achievement, L2 motivation, and self-regulated learning. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1261955
Xiao, Y., & Zhi, Y. (2023). An exploratory study of EFL learners’ use of ChatGPT for language learning tasks: Experience and perceptions. Languages, 8(3), 212. https://doi.org/10.3390/languages8030212
Yesilyurt, Y. E. (2023). AI-enabled assessment and feedback mechanisms for language learning. In Kartal, G. (Eds.). Transforming the language teaching experience in the age of AI. (pp. 25–43). IGI Global. https://doi.org/10.4018/978-1-6684-9893-4.ch002
Zainuddin, Z., Chu, S. K. W., Shujahat, M., & Perera, C. J. (2020). The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 30, 100326. https://doi.org/10.1016/j.edurev.2020.100326
Zebua, J. a. Z., & Katemba, C. V. (2024). Students’ perceptions of using the OpenAI ChatGPT application in improving writing skills. Journal of Language and Literature Studies, 4(1), 110–123. https://doi.org/10.36312/jolls.v4i1.1805
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., ... & Li, Y. (2021). A Review of Artificial Intelligence (AI) in Education from 2010 to 2020. Complexity, 2021(1), 8812542. https://doi.org/10.1155/2021/8812542
Zhang, C., & Lu, Y. (2021). Study on artificial intelligence: The state of the art and future prospects. Journal of Industrial Information Integration, 23, 100224. https://doi.org/10.1016/j.jii.2021.100224
Zhao, D. (2024). The impact of AI-enhanced natural language processing tools on writing proficiency: an analysis of language precision, content summarization, and creative writing facilitation. Education and Information Technologies, 1-32. https://doi.org/10.1007/s10639-024-13145-5
Zinkevich, N. A., & Ledeneva, T. V. (2021). Using grammarly to enhance students’ academic writing skills. Professional Discourse & Communication, 3(4), 51–63. https://doi.org/10.24833/2687-0126-2021-3-4-51-63
Zou, B., Wang, C., Yan, Y., Du, X., & Ji, Y. (2024). Exploring English as a foreign language learners’ adoption and utilisation of ChatGPT for Speaking practice through an extended technology acceptance model. International Journal of Applied Linguistics. https://doi.org/10.1111/ijal.12658
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 James Edward Lalira, Yopie A. T. Pangemanan, Jane E. Scipio, Sjerly Lumi, Theo Ch. Merentek, Vivi Nansy Tumuju
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with the VELES Journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).
- Authors are able to enter into separate, additional contractual arrangements for the distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
VELES Journal is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
